this situation bummed me out yes, understandable. It's mystifying to be "fired" from something and not know why. This happened to a friend of mine recently. One week he is getting commended for being the best in the department, the next week he's being told he's "not a good fit" and turfed.
keifermiller suggested something about inane office politics. I suspect that there is something that you don't know. That something might have nothing to do with you. Probably some parent phoned the principal complaining about something.
Did the principal attend any of the classes ever?
Suffice to say that I got my past physics teacher to utter the highest compliment I've ever heard from him ("You did a good job", the guy is almost impossible to impress and it showed on our grades.
It sounds like your past physics teacher attended one of your classes and thought you did good. He sounds like not the warmest guy - but I'd strongly consider asking him. There are various ways to approach him, including saying that you very much enjoyed giving the short lessons and would like some mentoring on how to do it better. Share the good feedback from the students, but say you wish you knew why you were asked not to come back.
Also, who is the teacher in charge of the Math and Physics Club. Who did you work with scheduling your short lessons - the principal or someone else?
In conclusion: there is something you don't know. You may never know it -- but I hope you continue your enthusiasm for teaching. It might be a calling. Don't let the bastards grind you down.