I 'm on the same camp. And I dont value life at all: It's kind of ok to die in my book. But you should not be the one making the call in any circumstance. You can, of course, but accept it's morally bad.
Whenever you want, because you feel it serve something more important. e.g: If the fatman decide to sacrifice himself for the 5 other dude, he's a cool guy. He is not morally forced to do so. And I wont push him on the track. And I feel anyone pushing him is kind of a dick.
I must have misread that then. I thought you meant that even in any case even suicide it would be morally wrong. I guess you only meant that when my actions cause another to die it is morally wrong, right? Unless of course the fat man is morally wrong for jumping in front of the train regardless of the people he saves? But you should not be the one making the call in any circumstance.