The quickest way to lose your passion for something is to make a career out of it. That's my opinion anyway.
I used to mess around with HTML and CSS from time to time and I used to genuinely enjoy it. The thing is, that was the occasional 3 hour session at most. When it comes down to "You WILL do this between 9am and 5:30pm!" it is a totally different story.
I think that is why I wouldn't go into game design full time. I usually map for a couple of hours at a time and I enjoy it. If I was told to build levels for 9 hours straight, I would rapidly get fed up with the processes involved and the quality of the work would suffer. Job security is another issue of course. If I make a bad website I spend a few hours fixing it up. If you make a bad game, you stop making games.
Another factor about doing design as a hobby is that if you're not sure how to approach a problem or if you simply get tired of working on a project, you can have a break, maybe leave it for a day or 2 and come back to it. At work, I have the option to say "I don't know what to do with this" but all that happens is the site moves down the list 1 place and I am given ANOTHER site to work on. It's draining. In the end of course, you HAVE to come back to the project within a reasonable amount of time or the client gets pissed off.
Sorry for the wall of text.