I think really the difference here is what you're doing with your selfishness. Are you just dedicating all your resources to yourself? I'd say that's within your rights. Are you extending your selfishness to behavior that affects other people beyond their access to your property? That, I'd say, is more of a problem.

If you own a pool and you want to be selfish with your pool, fine. Nobody wants to swim with you anyway. If, on the other hand, you own a car and you like to be selfish about your travel on the road, that's a beast of a different color. Suddenly going the speed you want to go doesn't just mean you're moving slowly, it means you're making everyone behind you late. Finishing your breakfast sandwich instead of looking for a gap to turn into is no longer just a matter of you yourself, but a matter of the poor bastard behind you.

When your selfishness causes you to take my possessions we again have a problem. Or when it means you leave your dog's shit in the middle of the sidewalk. Things like that.

Basically, if your selfishness goes beyond just depriving others of your generosity and becomes actively mistreating or disregarding other people I'd say it's a problem.

We ought to think about ourselves, though. If people can take care of themselves there are less areas where we need to rely on one another. To use a traffic metaphor again, consider the person who keeps letting people in front of them go, regardless of right of way or the natural flow of traffic. To their eyes they're being generous, they're making someone's day easier. Were they to pivot their heads, though (or just look in their rear view mirror for once) they'd see that they're complicating more people's lives than they're improving. Why? Because the flow of traffic is designed for people to concern themselves with their own passage in a timely manner. You're already doing the best thing you can do if you just drive to the place you're trying to go as quickly as you can. I think there's more that can be done socially than that, but it demonstrates how taking care of ourselves helps everyone by making things operate smoothly.

Sorry for all the driving stuff, I spend a lot of time on the road. I think it gets the point across though.