The problem here is you're arguing the abstract, from your personal experience and credo, and I'm arguing the concrete.
Yes. Absolutely. There are aspects of my past that chip away at the "privileged" label. Yes. Absolutely. There are aspects of the underprivileged that chip away at their "underprivileged" label.
BUT IT DOESN'T FUCKING MATTER.
I've spent 20 years of my life apologizing for my whiteness. I'm "allowed" to speak freely about Israel because I'm jewish enough for the Right of Return. I'm "allowed" to have an opinion about racism as one of the few white people in the world to experience racism at the hands of another ethnicity. But I'm not jewish. And the prejudice I experienced is a tiny corner case of what the overwhelming majority of non-white people go through routinely.
The bullshit part is that in the kabuki of privilege warfare, these meaningless gambits are enough to have my arguments listened to as a white man. They don't make my arguments more rational. They don't buttress my facts. All they do is allow me to wave my hands in front of my opponent and say
"I'm less of a straight white protestant male than that imaginary boogeyman you think you're debating, therefore you're obligated to at least pay lip service to my arguments on their merits."
_wage used to do that shit all the time: "Your argument is invalid because my childhood was tougher than yours." The fuck does that have to do with GDP? The whole privilege debate should be logos rhetoric: "here are the facts, here's what they mean." But the whole privilege debate is pathos rhetoric: "here are the facts, your interpretation is invalid because you're privileged." There is NOBODY - No. One. - who does not leverage their diminished privilege in order to argue that someone else has too much. And when you're a white male trying to talk about this shit, arguing that you understand someone else's viewpoint is the rhetorical equivalent of arguing that you have lots of black friends.
This is why there's such a lower-class white backlash against "privilege" - it is assumed by default that all people with white skin and normative heterosexual preferences have more of it than every person without white skin or without heterosexual preferences. Us white guys aren't even allowed to argue that we have, at some point in the past, experienced hardship because it hasn't been institutionalized hardship.
Yeah, the people giving me the stinkeye in the restroom are idiots. but their idiocy is sanctioned. At a baseline, being closed-minded is allowed if you're in the minority position - after all, you're oppressed.
Which means, at a baseline, if you're in the majority you're an oppressor.
End of line.
And you and I can talk in the concrete about what "you" and "I" feel and do but at the end of the day, it'll be a straight white male you don't know hating on your bathroom choices without knowing the first fucking thing about you and at the end of the day, it'll be someone other than a straight white male hating on me for, I dunno, resenting trigger warnings on ancient Greek plays and at the end of the day, you have it harder and I have it easier but this debate?
It rarely fosters understanding.