I think the basic problem is that when we argue architecture should blend into the background, we make it invisible from ourselves.
Here's Chicago "then and now." The only people who know the names of a handful of these buildings are the true nerds. Everyone else just notices that a new building went up maybe. Put up something hideous, though? Everyone knows its name.