If I were to hypothesize, I would hypothesize that the issue is a cultural outcome of an economic shift.
40 years ago, the wealthy weren't as wealthy and the poor weren't as poor. As such, when they encountered each other the differences weren't so stark. Now, the wealthy are rich and the poor are poor. And the differences would be stark, if the encounters ever happened. The lady with the house in the Hamptons doesn't know anyone without a house in the Hamptons. Forty years ago she would have, and the people who cleaned her house and babysat her kids were likely the same ethnic makeup but lower down the ladder.
Now? Now they're immigrants that she doesn't even have to pretend to have anything in common with.