Having a bunch of friends in or near Quidditch teams (my roommate went to the European Cup twice), I have also noticed that. There is also lots of drama / toxicity, which is even more puzzling.
1) it gives you that "sports" kick without having to participate in any of the sports the people who don't like you participate in. 2) The "fairness" of most sports is core to their playability and comprises the majority of arguments over rules. The "fairness" of Quiddich is taken to be nonexistent - that whole "find the snitch and win" thing is inherently unfair as it makes all other efforts pointless and there isn't a match in the whole of the wizarding world that has any fairness to it. So. It's going to attract people who want to play sports, but not with people who play sports, and who are attracted to unfairness. Nothing puzzling about it.