I think most of the idea is crazy.

First of all, the only outcome of us being too frightened to make the AI ourselves or hobble it to the point it doesn’t work is that other, much worse actors will not only get their first, but have an AI that is much less restrained than whatever AI you are scared of. In fact, this is a much worse outcome. The military is absolutely building AI, so are Russia, China, Iran, and so on. Black rock is probably working on one. Guess what? Absolutely none of those groups give the smallest amount of attention to the idea that AI might make a decision that harms people. And so the AI race is at current much more likely to be won by people with no concern about the AI moral compass than those AI doomers that pride themselves on being cautious about AI.

And most of the fears seem to come from movies and TV shows, not anything that these robots do or have done. We have miles of film of shitty 1980s and 1990s movies that decry AI as doomsday science. But “it happened in Terminator movies” is not even to the level of a real argument. It’s no more realistic than being afraid of space exploration because there might be Klingons out there. If humanity wants to stagnate at 2010 levels of technology, fine, but at the very least I think it should be based on observation rather than stupid movies. If these kinds of people had been listened to in 1600, we’d have never built tge new world. If we’d have listened in 1900, we would not have electricity in our homes. I’m on board with maybe not letting “kill all humans” be a life goal. But I think the dangers of technophobic people is going to do much more long term harm than AI could. AI can already detect cancers better than humans.

Personally I’m still amazed that these supposedly luminary AI doomers can all believe we need to prevent AI from going rogue in a way that will end humanity, without actually proving that’s even possible in the first place.

Here's the fundamental dodge at the heart of "rationalism", "effective altruism" and the whole TESCREAL constellation of sociopathic bullshit:

IF: I can envision a future in which my current actions, no matter how abhorrent, will be viewed as necessary or noble

THEN: I'm not required to give the first fuck about anything happening right in front of me.

I can burn this young lady alive because obviously she's a witch. I can ignore poverty in the world and focus on my own enrichment because I am spreading the gospel of the one true God who will save their souls. I can plot the murder of anyone opposed to my thinking because in the distant future everyone will acknowledge how crucial I was towards advancing the goals of humanity as it is then understood.

Pretty sure that's the reason for all the big words and grandiose concepts - it allows you to disconnect your actions and beliefs from the conventional societal markers.

Liches have trouble thinking clearly about paths through probability space that conflict with their phylactery, and the more conjunctive a mission it is to make true their phylactery, the more bits of epistemics will be corrupted by their refusal to look into that abyss.

God's anointed are troubled by moral concerns in the mortal world, but the more they ignore these concerns instead of rejecting them out of hand the more they lose God's voice.

It's all vengeful zealot bullshit, the TESCREAL dipshits have just wrapped themselves in a cloak of pseudomath.