There is a definite issue of being judged by the worst of your "group", no matter how loosely that group association is. My partner is vegan, has been vegan for 10 years. He's a very live and let live kind of guy, just does what he does for himself. Recently he got into a heated argument with another vegan over who is "really" vegan, because my partner bought a meatball sub for a homeless person who asked. Just buying meat for someone else was seen as this horrible sin, for lack of a better word, by another vegan. There have been several times that just saying "No thanks, I'm vegan" has been met with "Oh, so you think you're better than me" style reactions. Because no matter how calm he is, there are people in his "group" that are jerks and people who have dealt with enough jerks to see the entire group this way.