I hear what you're saying and, like everyone, I can't claim to be an authority on race. But since you shared your background with me, maybe sharing my background with you is also in order.
I grew up in an average family. My parents prided themselves on that fact. They were once farmers who escaped the lands my family had lived on for generations to find something better. They moved around a little bit but ultimately wound up in a mid sized city, occupying a mid sized home, living a middle class life, and I was comfortable.
That life afforded me opportunities. Opportunities for advancement and opportunities for ignorance. Through the former I got into good schools and I got nice work, though the latter I was protected from some of the harsher truths of life.
Down the street was a black family who was much like our own. The parents came from modest means and the children lived in modest comfort. In our little neighborhood, all things seemed equal.
And in that neighborhood they were. There was no malice, no one preventing children from being with one another, no prejudice (that we could see) and so it goes for most. But to me something seemed off, and it was. The crux of the problem, though, is that what had made the world as it was (the world I was first hand to, anyway) happened before I was born.
I learned years later what that really was. There were social and economic forces set into motion, long before I was even an idea, that still reverberate through our culture. It should be noted that I don't mean to attribute connotation to that statement (or, hopefully, any of these statements) I simply offer them as facts of another time.
In 1924 the National Housing Act began a practice called redlining where banks would draw a litteral red line on maps to mark areas in which they would not invest. These areas were primarily black ghettos, which had the consequence of ensuring economically depressed areas remained economically depressed.
The primarily black ghettos became primarily black because it's not like slavery ended and then everything became amazing. When slavery ended most blacks were only allowed to live in certain parts of the city.
So you have these primarily black neighborhoods that can't get capital investment to improve. Then you get blockbusting during the suburban flight. As blacks start realizing their situation is bad they try to get out and they start to move to the white suburbs. Once a house in a white suburb is purchased by a black family, the banks would call or visit other houses in that neighborhood and talk about how the whole place was going to pot. They would offer to buy white house for less than it was worth, get the whole block on their books, and sell them back to mobile blacks at far more than they were worth, making economically depressed and largely black areas. And because the loans were so high, owners would often default and have to move back to the ghettos.
This created an american mentality of "blackspace" and "whitespace". Ghettos are blackspace, suburbs are whitespace. This has social effects which are well ingrained in our culture, generally to the detriment of blacks and to the benefit of whites.
Most of that was quashed by the courts during the civil rights movements of the 1960's and 70's. So the language shifted from racially charged to economically charged. The Obama Administration's recent stimulus package, for example, targeted pre-existing assets, which the past 140 years of racially motivated legislation insured fell into the hands of predominantly white individuals. 1.5% of the beneficiaries of this package were black (I have the source on paper but not immediately available).
So, imagine playing a game of monopoly where everyone else gets 10 turns before you can put your piece on the board. You're at a marked disadvantage. This, generally, is what people mean when they speak of the socio-economic disadvantage to a certain group of people. It might not exist currently or in the forms that have been so well documented in the past, but it does in some way exist.
Now, this isn't to say that whites can't also be affected by these same forces. The scope of these programs is so high and so wide that, while they primarily affected minorities, there were also whites who were equally affected. Further, it seems that the current wealth disparity has far fewer of these race based caveats and are targeting, in mass, anyone who is simply not rich. This would seem to be a turning point in the constant power struggle in our world, but it doesn't erase the social, cultural, and economic disadvantages that were placed on many black families.
I'm still learning about all of this and I will be the first to say that there are likely gaps in my knowledge and my arguments. I am always researching and always seeking new information and if you have any questions I'm happy to share anything I can find.
These are deep and touchy subjects in American culture and they generally have to be navigated delicately. I've never been one for yelling matches and boasting, but I can get on a soapbox from time to time. If I did so above it was not my intent and I hope something up there helps provide some semblance of reason to what once was an unreasonable argument.
edit: holy shit I wrote a novel.