A fair point. I would probably be more upset that a group of men is beating on a woman. The bone and muscle structure of men tends to lead to them being stronger on average and therefore they could do longer lasting damage in a shorter time to a woman. I think that hate crime gets thrown around too easily. The Ku Klux Klan lynching a black person would be a hate crime. I don't think that a seemingly random attack between two parties of differing races should be classified as a hate crime. It's tough to accurately explore a person's motivations, but even if the situation was reversed, I don't think there is enough information given to determine a racial or gender (unsure if this is the correct form of the word) motivation.

Edit - I also think that I'm being influenced by the way my parents raised me. "Don't hit girls" and whatnot. I think that your hypothetical offends my ingrained sensibilities more than it logically makes a difference. I hesitate to use absolutes, but no group of people should be beating on anyone. Both situations involve repugnant acts and should be met with swift and decisive justice.