When you really think about it, believing whole-heartedly in meritocracy is a dangerous thing. It enables you to look down on others and justify inequality. Believing that the world runs on meritocracy is a disincentive for fixing problems of unfairness in society and just leads farther and farther from actual meritocracy. There certainly is an aspect of meritocracy in society, but when you really think about it, it seems surprising how little the big forces in the world actually run on it. Celebrities, politics, people with power; it seems a lot of those people owe a lot to connections and privileges. Maybe they are indeed capable, but would they have gotten there if it weren't for some special advantage? With so many capable persons in any given pursuit, you think maybe the special advantages are the only things that matter.
To piggy back off your comment, specifically the portion that says, "(Meritocracy)..enables you to look down on others and justify inequality," struck me as especially poignant. The dangers of meritocracy don't stop at economic disadvantage, but continue into the religious arena with concepts such as karma, (the "untouchables" deserve to starve because of what they did in a past life, etc.). This sort of "bleed" is fascinating, and serves to further illuminate your point on the dangers of a merit-based society in any capacity. Why are so many societies obsessed with forcing people to pull themselves up by their (often non-existent) boot straps? I think it has a lot to do with the staple Steinbeck quote which states, "Socialism never took root in America because the poor see themselves not as an exploited proletariat but as temporarily embarrassed millionaires." Who promotes the ideal of the temporarily embarrassed millionaire though?