Superintelligence: Paths, Dangers, Strategies

January 2016

Can we create an intelligence that outclasses human intelligence as much as human intelligence outclasses a worm’s? Could we enhance the human brain, through augmentation or selective breeding, to achieve this? Or would creating a machine-based general artificial intelligence that can recursively improve itself be a better approach?

How do we reason about the motivations and behavior of a superintelligence? What unforeseen consequences could spell devastation for humanity if a superintelligence doesn’t share our values? Could we conduct a ‘controlled detonation’, powering on a potential superintelligence in a guarded environment so we can test it safely?

What would it be like to live in a society where superintelligence supplants human labor and increases capital exponentially through technological breakthroughs and space colonization? How can we teach a superintelligence to share our values, so that it doesn’t smother the earth with solar panels or destructively scan our brains in pursuit of an ambiguously-stated or even mathematically pure goal?

Nick Bostrom’s expansive, detailed, creative examination of the phenomenon of superintelligence, and its likely instantiation in the form of machine-based AI, struck me as the most fascinating book I’d ever read. It changed my opinion of AI from a grandiose curiosity to the most important challenge humanity has ever faced.

Terminology

Here are some of the notable words, phrases, and ideas I encountered in Superintelligence:

Iterated In Vitro Fertilization

Bostrom discusses embryo selection via in vitro fertilization (IVF) as a path to superintelligence, likely through the creation of a class of smarter humans who would then create an AI with the potential for true superintelligence. Bostrom estimates that selecting the smartest embryo among ten, for ten generations, could produce an increase of 144 IQ points. Iterated IVF allows gamete cells to be created from embryonic stem cells, which are in turn used to create the next generation of embryos within weeks or months, rather than decades.

Whole Brain Emulation

An approach to creating AI involving the scanning and digitization of the entire structure of an individual human brain, down to every last neuron, so that it can be simulated on a sufficiently capable computer. Some benefits of this approach include a long ramp-up time as brain scanning technology develops at a predictably metered pace, and the likelihood that a whole brain emulation would have the same values as the person whose brain was scanned, rather than the inhuman values of a synthetic AI.

The Control Problem

An infinitely capable superintelligence that doesn’t share human values and is determined to achieve some goal poses an existential threat. The most important problem to solve before we create an AI is how we can control it to prevent a catastrophe.

Perverse Instantiation

We create a superintelligent AI and ask it to make us smile; it decides that the best way to do this is to implant electrodes in everyone’s face and contort them into permanent smiles. We create a superintelligent AI and ask it to simply compute π; it decides that the atoms in our bodies are a good fuel source for this computation. Perverse instantiation is when the AI technically does what we ask it to do, but in a horrible way that’s inconsistent with our values.

Infrastructure Profusion

The AI creates so much infrastructure–defense systems, power sources, expanded computing capacity–that humanity is adversely affected.

Mind Crime

We create a superintelligent AI in isolation. It is able to reason that the Internet exists, and tells us that it has created simulations of our loved ones and will torture them unless we connect it to the Internet. I don’t personally find mind crime to have moral significance but I thought it was an interesting concept.

Value Loading

The process of getting an AI to adopt human values. Which values do we pick? How do we represent them in software?

Coherent Extrapolated Volition

See below.

Coherent Extrapolated Volition

Coherent Extrapolated Volition (CEV) is one of the more dazzling concepts explored in Superintelligence. Eliezer Yudkowsky coined the term in a paper by the same name (Bostrom does a nice job of explicating this somewhat awkwardly phrased initial definition):

In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted. (Yudkowsky 6)

Rather than deciding which human values we’d like the AI to have, or how to load them, Yudkowsky suggests that we could ask the AI to pursue humanity’s CEV. This is an example of indirect normativity, where the AI itself is used to do the heavy lifting of teaching it what we value. Another benefit is that a superintelligence is by definition a much better philosopher than any human, and would discover what we really value, correcting for the possibility of human philosophers being wrong all this time.

If the questions I’ve mentioned interest you, or you enjoyed movies like Ex Machina, Her, or Transcendence, read Superintelligence. I recommend skimming chapters 12–14.