I completely agree that we should be using Average Utilitarianism as well. The only downsides I can think of are only "downsides" because they are counter intuitive, but intuition doesn't define morality. Here's an example: a population of 100 people with 100 happiness each or a population of 1,000,000 with 99 happiness each. Intuition tells us that the larger population with almost the same happiness is better when in reality population size is only important insofar as it adds to happiness.
I also agree with your point that utility or happiness isn't well specified. Personally, I think that an answer will arise the more we learn about the brain through neuroscience so that we can find an objective quantifiable measure that pleasure, joy, ecstasy, etc. all add to.
Edit: I forgot about an issue with average utilitarianism. See below.