Your brother is not necessarily wrong - there is a hormonal basis for the lack of female facial hair. The real evolutionary cause is facial neoteny. Neoteny is the "retention by adults of traits previously seen only in juveniles". The effects of neoteny are massively exaggerated in human females (i.e., larger eyes, smaller noses, and fuller lips). Many physical anthropologists have shown that you can estimate the age of an individual based on information about eye width, nose height, and lip height alone - and it is clear that women (on average) have much larger eyes, smaller noses, and "taller" lips than do men. In this sense, facial hair growth can be seen as part of this neotenous package (as infants and children also do not have facial hair - obviously). This could be adaptive, or perhaps an exaptive - no research has really tested which it is (but the actual selective pressure was very high since lack of facial hair is always seen as a sign of physical attractiveness cross culturally).

I would add that several studies have shown that in the neotenous package - women are always rated as more sexually attractive cross culturally if they exhibit "supernormal" aspects of the neotenous package, and studies of high end fashion magazine and super model magazine model facial proportions also end up falling on the most neotenous end of the facial spectrum.

I would add that in these studies the researchers admit that it is hard to control for what it was - evolutionarily speaking - that men selected for in women - neotenous faces, or maximal waist-to-hip ratio - both of which are indicators of youth and high fertility.

To end - I took a sexual selection theory course in grad school. Two of the most interesting questions to me where the following:

A) Why are humans the only species that have hair that needs to be cut?

B) What is the evolutionary origin of the female orgasm?

One of these questions is currently still a mystery. What one do you think it is?