The 'racist crowd' only invade subreddits that affect reddit's larger image. They're using it as a platform to push their agenda. They also use some interesting strategy in coaxing racism and racism-oriented discussion from the community, by posting things that I believe started with near-homeopathic doses of racism. This, by virtue of how likely it is that some people in a large group will share a "mildly" racist opinion, ends up showcasing how there are still a surprisingly large amount of people who feel, for instance, that they should be more careful around black people because they're prone to robbing peopley. This is obviously still racist by definition, but it's pretty far from Stormfront-level racism, and it's not hard to see how someone could delude themselves into thinking it isn't racist (provided you understand the capacity humans have for deceiving themselves).
Now, adding to that, people who hold that racist opinion will feel persecuted. Because people are going to tell them they're racist (which is true enough), but also do silly things like compare them to the KKK. Which they will see as a reason to be louder, and bring up the same issue again and again, and participate in every thread to show that they're not crazy or ignorant, other people believe this too! Eventually the people who aren't racist just stop caring and accept reddit as it has come to be.
The net effect this has had... I want to say it's like a shit tornado. This was a website where we'd discuss the news, the latest xkcd, random shit from the internet and shitty programming jokes. I guess that wasn't really reddit's own culture/community, just the one that happened to occupy it in the beginning.
sidenote: talking about reddit here still feels awkward to me