I enjoyed the thought experiment, but could've done with less focus on Yudkowsky and LessWrong. The guys who are worried about it as an actual real life threat seem...fucking wacky, to be honest with you.

I skimmed some of his papers over coffee this morning, and discovered that Eliezer Yudkowsky doesn't believe in the scientific method, and thinks it can be replaced with pure Bayesian reasoning.

I kind of hate rationalwiki, but their article on him is pretty thorough.

It is important to note that, as well as no training in his claimed field, Yudkowsky has pretty much no accomplishments of any sort to his credit beyond getting Peter Thiel to give him money. He claims to be a skilled computer programmer, but has no code available other than Flare, an unfinished computer language for AI programming with XML-based syntax

...

Yudkowsky is almost entirely unpublished outside of his own foundation and blogs and never finished high school, much less did any actual AI research. No samples of his AI coding have been made public.

...

His actual, observable results in the real world are a popular fan fiction, a pastiche erotic light novel, a large pile of blog posts and a surprisingly well-funded research organisation — that has produced fewer papers in a decade and a half than a single graduate student produces in the course of a physics Ph.D (and the latter's would be peer reviewed).

Even good ole' fashion Wikipedia is uncharitable:

Yudkowsky has no formal secondary education, never having attended high school or college. Nevertheless he [self] reported scoring a perfect 1600 on his SATs.

...All if this research just because the quote from the article frustrated me. "Listen to me very closely, you idiot." What a crock of bullshit.