Easy to say, but much harder in practice when people need to overcome internal biases that they might not even be aware of.
Welcome to the world of social justice.
Could the same sort of mentality be at play here?
That's what I was thinking. Pushing the fat man means being directly involved into his death, while switching the railway means that you're basically just moving the bullet slightly, the bullet itself being already in the air set by something else.