I think there is just a difference in how reading and math are understood. Being literate seems to only imply the barest minimum of standards. If you can translate text into the sounds that it symbolizes then you can read. Most kids will be able to do that by the time they're 5. However, it really doesn't say anything about reading comprehension. A person might be able to take away the relevant information from a menu at a restaurant, Facebook posts or even news articles, but not be able to draw anything of note from a great novel. Such a person might say, "I don't enjoy reading," but not actually consider themself to be a bad reader simply because they would be considered literate. "Bad at reading," to me at least, really implies being illiterate. Contrast that with math. I think the bar is just placed higher. For instance, being able to count is just taken for granted. That same person who has no trouble reading menu items but forgoes literature can also probably pull out exact change without much effort. Only now, because they had trouble with algebra in school, they'll consider themself to be bad at math. Being able to say you're good at math seems to require more ability, and with things often being seen in black and white, if you aren't good at it then you must be bad. In fact, that even brings up the other side of reading. I don't think anyone would ever say, "I'm good at reading." People just don't think in those terms on that subject. So, I think there is more to this than one being more socially acceptable.
This nonrequirement that individuals be especially talented at math I think is real, which is unfortunate, because while it's fine for individuals, in the aggregate, you don't want a country full of citizens with weak math skills. The future is won on innovation, and we need to be generating people with strong skills in engineering and the sciences (married with a high level of creativity). But yeah. It's ok to be bad at math because you don't really need to do it any better than you did in middle school IMHO. Nt saying thats a good thing...