You do your best. And that's all you can do. Because different people prefer different terms, depending on their age, geography, culture and just personal preference.
And if someone throws a shitfit at the term you use, you remind yourself that your intent was to be polite, and it is their choice to get offended. Don't apologise. Merely say: no offence was meant, if you prefer a different term to be used, then please make that known.
Ultimately whether you describe someone as "coloured" or "black" it doesn't actually harm them. Likewise "gay" or "homosexual".
If you're a well-meaning, non-bigoted person, and they react badly, then fuck them, seriously.
These days, I am no longer extending any tolerance to the offenderati. If you are not in some shithole like ISIS-run Syria, being raped and tortured on a daily basis, then you can essentially fucking get over yourself. (I don't mean you, OP, I mean people who make a hobby out of getting indignant).