The word "nutty" wouldn't have the connotation of "bad idea" if we didn't stigmatize mental illness, and we stigmatize mental illness because we use words like "crazy" and "nutty" to describe bad ideas. See how it's a self-perpetuating cycle?
Whenever people phrase things like that, what they really mean is "I don't understand why you want to do X." But instead of phrasing the dialogue in a way that asks why they want to do X, instead the mentally ill are scapegoated. This is literally the definition of scapegoating. You don't want to actually talk about why you think the idea is bad, or just tell someone that their judgement is bad... instead you choose a group of people with a biological illness with negative connotations to associate with the behavior that you don't understand the motivations for, because OF COURSE no one would want to be associated with that group of people.
People with mental illnesses are not stupid. They are not irrational. I am stable the majority of the time. But because of language like the example you gave, and because of the way the mentally ill are portrayed in fiction, I have to keep my disability a secret at work because otherwise people will question every single thing I do and use my disability to discredit me when I disagree with them. That is the life I have to live because people use the same words to refer to my disability as they do to refer to unpredictable, dangerous behavior, even though there is no correlation.
It hurts me a lot more than the benefit you gain from being able to use that word. By using that sort of language, you are directly contributing to the complex of memes which hurts me directly on a daily basis. And when I say it hurts me, I don't mean "oh you made me feel bad by saying that". I mean you contribute to a culture where people mistreat me and get away with it in the workplace, in my own family, in school, etc.