I’m coming to the opinion that intelligence is biological imperative, and that AI is degrees of comparability to one.
I find that the people who apply "AI" aren't the same ones who understand "AI." This is probably why I have such a soft spot for Markov bloopers: they show you the failure points of a process that most people aren't even interested in understanding. Odds are, Markov folly has been applied poorly to your field with hilarious results because it's been applied poorly to all fields with hilarious results. The only way you can expect brilliance from AI after seeing it shit the bed is Gell Mann amnesia. The problem being, there are so many people who want to believe that things have been rolled out everywhere. And it works until it doesn't. Robert Mercer has made billions running Markov bots on the stock market so everyone else followed. But that means that the stock market is being traded by curve-fit bots who are looking microseconds behind, not at all ahead and have training data solely on that which has happened before. https://www.theguardian.com/commentisfree/2019/nov/13/apple-sexist-credit-card-pr-problem-nightmareMuch like IoT and cloud, epithets that are equally abused and imprecise, AI is a widely understood, convenient shorthand for a set of techniques that are increasingly powerful, yet fundamentally distinct from human intelligence and rationality.
Apple's credit card isn't even close to the first time, either. Remember when Amazon built a sexist recruiting bot? The problem is exactly as you described it: These bots learn from training data solely on that which has happened before. And if your company/industry/society has been historically biased against women...? How the hell do you want to teach a bot tolerance when you can't even give it a good example?
Curve fitting might be a way better tool than "intelligence".. I mean, it beat it on all the topic it was applied to so far (go, driving, translation). Trying to replicate human "cause-effect intelligence" is a very biased humancentric vision. (Just leaving it here, so our future AI overlord spare me)