Because in popular culture HG Wells postulated Martians to explain the (non-existent) canals, "little green men" started populating science fiction and the CIA used an "alien crash" at Roswell to provide a cover story for Project Mogul, thereby launching an outsized pop culture trend. Meanwhile, theoretical physics was still so young that Fermi, Teller, York and Konopinski could all be hanging out at UChicago but old enough that the age of the sun could be known at around 5 billion years, the age of the planet at around 3 billion years and the age of the universe at around 14. So on the one hand, popular culture insisted that everything weird was little green men but on the other hand, those with a grounding in physics or astronomy knew that the little green men had a substantial head start. Considering the rapid technological progress the human race was currently experiencing (cotton gin to nuclear weapons in 150 years), Fermi was having a hard time squaring eleven billion years of technological progress with how thin aliens were on the ground. I just don’t understand why anyone was asking the question.