That is, how do you know that religion helped our species to dream of the infinite before the infinite was in our grasp?

Religions primary evolutionary function is to help us cope with the fact that life is finite. Current studies from thanatology suggest that chimpanzees have a poor understanding of death. But at some point in our evolution (most definitely before the arrival of modern humans) our genus grasped that life was finite, and likely invented a concept of afterlife in response to this realization (e.g., Homo neanderthalensis buried their dead).

We also know from studies of the first societies with recorded literature and from studies of hunter gatherer tribes from the 19th and 20th centuries that most early human societies had no concept of religion, because everything was religion. This would not take the form of modern institutionalized religion, but all of life would be heavily influenced by supernatural belief (which is why I define was a requirement for "religion").

I really don't think this is that controversial of an assertion and rooted in evolutionary theory and available evidence.

I think atheism is more likely to be the default position of ancient humans. After all, we are all born atheists. Religion is a learned phenomenon. Is there any reason to believe this would have been different in the past?

I don't even know how to respond to this. It has been demonstrated over and over again that humans have a natural inclination to have experiences that can be categorized as "spiritual" and all known human cultures have had a belief in some type of supernatural entity. So how could atheism be a our "default position". A supernatural belief structure helped pre-literate hunter gatherer societies A) explain the unexplainable, B) cope with death, and C) help them believe that someone "was on their side" and "cared about their struggle" in a world filled with death, hunger, disease, etc.

Religion, is by its very nature, is coercive. It has been constantly been used by people in positions of power as a tool to dominate others (or to obtain power in the first place). Given that this has been a constant in all nearly all religions throughout recorded history it seems much more likely that religion originated as means of control.

This is a major function of religion that was co-opted by early agricultural civilizations. It doesn't mean that it was the origin of religion (or supernatural belief structures). It just means that it was a function (one of many; with others being more fundamental as ultimate cause) that developed in complex agricultural civilizations.