465. Dreams of Friendliness – 473. The True Prisoner’s Dilemma

465. Dreams of Friendliness

FAI-proposals like Oracle AI or AI-boxing won’t work. Oracle AI is FAI-complete and AI-boxing ist just plain unworkable.

Mind you, I’m not saying that you couldn’t build an Oracle.  I’m saying that the problem of giving it a goal of “don’t do anything to the outside world” “except by answering questions” “from the programmers” “the way the programmers meant them”, in such fashion as to actually end up with an Oracle that works anything like the little XML-ish model in your head, is a big nontrivial Friendly AI problem.  The real world doesn’t have little discreet effect-rays leaving the AI, and the real world doesn’t have ontologically fundamental programmer.question objects, and “the way the programmers meant them” isn’t a natural category.

466. Brief Break

I’ve been feeling burned on Overcoming Bias lately, meaning that I take too long to write my posts, which decreases the amount of recovery time, making me feel more burned, etc.

So I’m taking at most a one-week break.

467. – 472. Rationality Quotes Part 12 -17

Just some rationality quotes.

473. The True Prisoner’s Dilemma

The normal Prisoner’s Dilemma is not a “true” Prisoner’s Dilemma because (most) humans are not entirely selfish and the fact that your confederate has to spend more years in prison influences your decision, even if you try to imagine being entirely selfish. Here is a “true” Prisoner’s Dilemma:

Player 1:  Human beings, Friendly AI, or other humane intelligence.

Player 2:  UnFriendly AI, or an alien that only cares about sorting pebbles.

…The payoff matrix is as follows:

1: C 1:  D
2: C (1 billion people tortured, 2 paperclips gained) (nobody tortured, +0 paperclips)
2: D (+5 billion tortured, +3 paperclips) (3 billion tortured, +1 paperclip)

Leave a Reply