Mathematician/engineer here. This discussion of art and its relation to mathematics is very common among mathematicians. Less so among engineers in my experience. I think you've struck on an important point: that aesthetics is not the only element of art. This is why I think using the term "beautiful" is not always appropriate to describe the way a mathematician feels about a particular piece of work.

My opinion is that art (a very hard term to define) is intimately related to sharing perspective. I think that our attraction to various kinds of art is the connection it provides with the artist, the story they tell us and the emotions they share with us. For example, a photographer understand and manipulates light in order to convey a view on a frozen moment in time. Similarly, an impressionist painter exhibits his process without being tied so close to a single physical scene. Even comedy works best by making a connection to the audience while providing a fresh and maybe unexperienced perspective on a topic.

This is the context I use while discussing the art of mathematics. The formulas and proofs used often do provide satisfaction to the practicing mathematician, in the same way an inspiring painting, photo, musical piece, or comedic bit provides pleasure while inspiring new work. However, I don't think they're "beautiful" in and of themselves. Rather, it's the new perspective that comes with a deeper understanding of the underlying structures you come across. It's the "ah, look at that" aspect that can sometimes be very striking and even emotional. Made all the more powerful by its timelessness and transcension of humanity.