Casual Racism is more along the lines of using racial slurs/ideas/stereotypes in a light hearted matter in an everyday setting such as an office or at dinner and just expecting that everyone is going to be cool with it. I mean your friends might not have issues with all the race jokes but supposed someone from outside your group hears them? They might have a problem with it because some jokes hurt a lot more than you think to some people. Everyone is different in what offends them, so it's kinda tough to determine where ones draw the line in this sort of thing. You just have to be careful though because you have no idea how someone's going to something you might see as light hearted.
Interesting view, and I half-agree with you. In an unfamiliar/professional setting it's definitely important to remain sensitive to these things, because, as you said, you don't know what people might take offense at. If someone from outside my friend group overhears us and finds it offensive... Then they find it offensive. They can have a problem with it all they like. However, I'd be hesitant to label it as 'casual racism'; doing so is subtly calling the speaker a racist. A person isn't a racist unless they believe that one race is inferior to the other. Unless the speaker genuinely believes that and their words reflect that belief, it shouldn't be labeled as 'racism'. Stripping away the intention and focusing on the words will always lead to misinterpretation. For example, by focusing solely on the words, something as innocuous as "I love steak" can be interpreted as promoting animal cruelty. However, the speaker could really just be enjoying his dinner. It's impossible to make sure that nobody is offended by what I say, so I shouldn't be branded a racist just because someone took offense. If anyone is offended at anything, they should think "Why am I offended?" and "What was that person's intention?" We shouldn't be so quick to jump the gun and call them racists. I mean your friends might not have issues with all the race jokes but supposed someone from outside your group hears them? They might have a problem with it because some jokes hurt a lot more than you think to some people.