> the opinions of 320 million Americans are difficult to sort through when it comes down to constitutional interpretation
> "Well, we can't agree on this. What do you say?"
Opinions of 320 million Americans don't and shouldn't matter in Constitutional interpretation—that's the whole point of an unelected judiciary. The court's job isn't to reflect the balance of public opinion, nor to anticipate the long-term direction public opinion is heading in. The court's job isn't to be the final arbiter of public debate, it's to assess whether statutory laws are in violation of the Constitution.
> This didn't remove the rights of 320 million Americans
I'm not convinced. Every single referendum on the subject of gay marriage came down against it until Iowa. The majority of states that had legal gay marriage before this decision had enacted it via judicial fiat, overturning public referenda or statutory law passed by the public's representatives. What is happening when five justices invent something unwritten in the actual text of the Constitution in order to overturn both the existing results of the democratic process and any future democratic initiatives/revisions?