I understand what you are saying, this has happened with me a lot of time.
First of all never move the slider to the extreme left.
When you open the video page, the video starts buffering automatically, now when you change the video quality from 360p to 240p (say), the content already loaded is lost and new buffering starts at 240p from the time 00:01 (sec) (or may be even a fraction of a second difference).
Saying that, if you move the slider to a portion of the video that has not been downloaded at all (00:00 is not downloaded at all - it was downloaded at 360p but then lost), all the downloaded content is again lost and the video starts buffering again from the point you have set the slider in.
What is it when it is not extreme left?
You have started viewing a video, the progress bar shows at to up-to which extent the video has been downloaded and you move the slider to a point within that range, but still you loose all the content and buffering starts again - WHY?
This is because there are a lot of problems with the YouTube's progress bar. The progress is always actually a few seconds behind what is shown in the progress bar. And also sometimes, the progress level is completely lost and the player shows that the video is downloaded till the end, but it is not - And as I have already said above, moving the slider to a point where the video has not been downloaded makes you loose all the content and starts buffering from that point.