AI Foom Debate Conclusion: Post 50 – 52

50. What Core Argument? (Hanson)

Hanson asks again for Yudkowsky’s core argument(s) and lists his objections.

Firstly, it must be said that most AI-researchers and growth-economists consider Yudkowksy’s Foom-scenario to be very unlikely. Which of course doesn’t mean much if you believe the world is mad.

He also thinks that the small differences in brain size between chimpanzees and humans do not show that there aren’t any diminishing returns or low-hanging-fruit slow-downs.

But I think this argument is rather persuasive: It’s likely that there isn’t anything special about the level of human intelligence (due to the copernican/mediocrity principle) so if we don’t see diminishing returns from chimpanzees to humans why should there be any speed bumps anytime soon after human intelligence? Sure, eventually there will be diminishing returns, but probably not very soon. The second part of the argument is that human cognition is deeply flawed in many aspects such as the known biases, poor memory and arithmetic skills, etc. Thus it should be fairly easy to go beyond human intelligence.

I, personally think that the main argument against Yudkowsky’s thesis is the multi-dimensionality of intelligence. Indeed, Yudkowsky wrote in an earlier post:

Yesterday I exhausted myself … asking … “What do you think you know, and why do you think you know it?” with respect to “How much of the AI problem compresses to large insights, and how much of it is unavoidable nitty-gritty?

But, as far as I can see, he didn’t provide any compelling arguments for why the solution of the “mystery of intelligence” should only require a few large insights.

51. What I Think, If Not Why (Yudkowsky)

Pretty illuminating post. Yudkowsky summarizes his positions:

A well-designed mind should be much more efficient than a human..

• An AI that reaches a certain point in its own development becomes able to (sustainably, strongly) improve itself…This point is at, or probably considerably before, a minimally transhuman mind capable of writing its own AI-theory textbooks…

• It is likely that this capability increase or “FOOM” has an intrinsic maximum velocity that a human would regard as “fast” if it happens at all… the core argument does not require one-week speed and a FOOM that takes two years (~1e17 serial ops) will still carry the weight of the argument.

The default case of FOOM is an unFriendly AI, built by researchers with shallow insights…

The desired case of FOOM is a Friendly AI, built using deep insight…

• The good guys do not write an AI which values a bag of things that the programmers think are good ideas, like libertarianism or socialism or making people happy or whatever.  … It is dealt with at length in the document Coherent *Extrapolated* Volition…. I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.

The good guys do not directly impress their personal values onto a Friendly AI.

• Actually setting up a Friendly AI’s values is an extremely meta operation, less “make the AI want to make people happy” and more like “superpose the possible reflective equilibria of the whole human species, and output new code that overwrites the current AI and has the most coherent support within that superposition“. …Describing this as declaring total war on the rest of humanity, does not seem fair (or accurate).

I myself am strongly individualistic:  …When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background, silent as the laws of Nature once were; … I am barred from programming any such consideration directly into a Friendly AI, for the reasons given above.

• Nonetheless, it does seem to me that this particular scenario could not be justly described as “a God to rule over us all”

Friendly AI is technically difficult and requires an extra-ordinary effort on multiple levels...

• White-hat AI researchers are distinguished by the degree to which they understand that a single misstep could be fatal…The story isn’t about good versus evil, it’s about people trying to do the impossible versus others who… aren’t.

• Intelligence is about being able to learn lots of things, not about knowing lots of things….

Ah, he finally addresses my main concern and acknowledges that he so far hasn’t offered strong arguments for the thesis that intelligence is the result of only a few deep insights and not lots of context-specific algorithms.

Intelligence is mostly about architecture, or “knowledge” along the lines of knowing to look for causal structure (Bayes-net type stuff) in the environment; this kind of knowledge will usually be expressed procedurally as well as declaratively.  Architecture is mostly about deep insights.  This point has not yet been addressed (much) on Overcoming Bias, but Bayes nets can be considered as an archetypal example of “architecture” and “deep insight”.  Also, ask yourself how lawful intelligence seemed to you before you started reading this blog, how lawful it seems to you now, then extrapolate outward from that.

I hope he is going to elaborate on that.

52. Not Taking Over the World (Yudkowsky)

Yudkowsky tries to persuade us that he’s not trying to take over the world. Indeed, the CEV-strategy is pretty meta and explicitly weighs the (extrapolated) preferences of all humans equally.

To be honest: I wished Yudkowsky would try to take over the world. EV_Yudkowsky or CEV_{Yudkowsky, Shulman, Rayhawk, Kaas, Salomon, Muehlhauser,….} sounds awesome. It may be the case that CEV_humanity is almost the same as CEV_SIAI-members but that’s far from certain.

But if one of my enemies were to build a superintelligence I would try to persuade him to follow the CEV-proposal because a) there is a reasonable chance that I could persuade him (if I said “hey, implement EV_wallowinmaya I obviously wouldn’t have a chance) and b) it’s not unlikely that CEV_humanity is in accordance with EV_wallowinmaya.

In short: CEV is just one of the most promising Schelling points.

Yudkowsky then asks Hanson what he would do with unlimited power, but Hanson kinda dodges the question. This is indeed a very interesting question and Yudkowsky will expand on this in the Fun-Theory-Sequence.

Leave a Reply