a thoughtful web.
Good ideas and conversation. No ads, no tracking.   Login or Take a Tour!
comment
wasoxygen  ·  2868 days ago  ·  link  ·    ·  parent  ·  post: Superintelligence: The Idea That Eats Smart People

It's fairly certain that malaria will continue to kill people. It is very likely that your contributions to AMF will reduce this bad outcome, buying some time until a better solution is found. (Previous improvements in our response to polio and smallpox give reasonable hope for such progress.)

Meanwhile, if the risk of AI catastrophe is 1%, then it is 99% certain that resources dedicated to averting that problem will be wasted (disregarding side benefits of the research, which could occur with malaria research as well).

There is also some concern that a project like OpenAI could increase risk of a disaster.

Asteroid impact could render all these problems trivial; it's hard to prioritize giant problems that have tiny probabilities.

I agree that a lot of the essay is not very rigorous, but I think it makes some salient points:

· It is not clear what "hyperintelligence" means, and not obvious that it's possible for anything to be exceedingly more intelligent than people.

· We are not good at "baking in" robust reliability to complex systems; we make gradual improvements through trial and error. Such improvements are easily defeated, often unintentionally.

· The cats and emus demonstrate that superior intelligence does not guarantee the ability to dominate inferiors.