Not crazy at all - that's actually a pretty big part of my plan/hopes for the immediate future. If possible, I'd like to go one step further and make clothes specifically for gender neutral / trans* folx. My bf is a trans guy, and I have a bunch of genderfluid/non-binary friends that I've been getting the measurements of/making clothes for to try and figure out what specific alterations work well for different types of bodies. My big goal for the year is to learn a bit more about pattern drafting, and try and release a free shirt pattern for non-binary / masc. presenting folks. (But also, just adding approximate sizes for mens/womens is a really solid idea, and somehow one I hand't thought of yet despite all the above...)
you should talk to the lady who runs Origami Customs - she might have some good info for you. She does a lot of intimates and swimwear for trans and non-binary people on both sides of the spectrum. Who know, she might have some more measurements to add to your alteration charts?