My final thoughts and I'll let it go. I think my main displeasure at the phrase "It's just a game" is that it's typically used to marginalize something that's important to other people. Bullies use that phrase a lot, "it's just a game," to justify their actions against their victims as "merely" an unimportant aspect of their lives. However, to the victims, it's anything but a game, and trying to treat it like a game is running away from the fact that people are misusing privileges granted to them.
The fact that games are important to me can be equated to how religion is important to other people, or cars, or pets, or fantasy football. Other people might not get it, but it doesn't mean it's any less important to my life, just because someone says so. I'm not a religious person, and I don't own any pets, but I don't say "It's just religion" or "It's just a cat" when I feel that people are being too zealous about it.
EDIT: I should clarify that I don't think there's anything *wrong* with someone's personal feelings that something else is more important than the game. However, when using that specific phrase "it's just a game," you're imposing your own values onto someone else and making them feel that they are wrong or mistaken for valuing games highly, in the same way that you might be offended if I said "it's just religion" or "it's just a cat."