In the lasts posts Yudkowsky described morality as a computation but this word may have evoked wrong connotations. So he calls morality an abstracted idealized dynamic. Now, what does this mean?
We sometimes develop different moral views just by thinking about the topic in question, that means armchair philosophizing can change our moral beliefs. This is similar to math/computations where we also can arrive at true beliefs just by thinking about a problem. Or in Yudkowsky’s words:
…it seems like that-which-is “morality” should have the property of latent development of answers—that you may not know right away, everything that you have sufficient in-principle information to know. All the ingredients are present, but it takes additional time to bake the pie.
Morality, likewise, is something that unfolds, through arguments, through discovery, through thinking; from a bounded set of intuitions and beliefs that animate our initial states, to a potentially much larger set of specific moral judgments we may have to make over the course of our lifetimes.
And there is another similarity between math and morality: The answers to mathematical as well as moral questions don’t depend on specific physical instantiations. If we have 10 calculators and we type 13+34 in all of them we know that the true answer is 47 even if some of the calculators malfunction and produce false answers.
And when we ask 10 people if torture is wrong we know that the right answer is “yes” even if some of the folks answer “no” because they are psychopaths or otherwise confused.
…I have to idealize the answer, … talk about this ethereal thing that is not associated with any particular physical process known to me—not even arithmetic done in my own head, which can also be “incorrect”.
It is this ethereal process, this idealized question, to which we compare the results of any one particular calculator, and say that the result was “right” or “wrong”.
To sum up:
- Morality, like computation, involves latent development of answers;
- Morality, like computation, permits expected agreement of unknown latent answers;
- Morality, like computation, reasons about abstract results apart from any particular physical implementation;
- Morality, like computation, unfolds from bounded initial state into something potentially much larger;
- Morality, like computation, can be viewed as an idealized dynamic that would operate on the true state of the physical world—permitting us to speak about idealized answers of which we are physically uncertain;
- Morality, like computation, lets us to speak of such un-physical stuff as “error”, by comparing a physical outcome to an abstract outcome—presumably in a case where there was previously reason to believe or desire that the physical process was isomorphic to the abstract process, yet this was not actually the case.
Good comment by Steven (Kaas, I guess) who summarizes Yudkowsky’s metaethics in “professional” terms at the request of Toby Ord:
“I’m not Eliezer nor am I a pro, but I think I agree with Eliezer’s account, and as a first attempt I think it’s something like this…
When X judges that Y should Z, X is judging that Z is the solution to the problem W, where W is a rigid designator for the problem structure implicitly defined by the machinery shared by X and Y which they both use to make desirability judgments. (Or at least X is asserting that it’s shared.) Due to the nature of W, becoming informed will cause X and Y to get closer to the solution of W, but wanting-it-when-informed is not what *makes* that solution moral.”
Yudkowsky agrees with this description.