Yudkowsky plays “Rationalist’s Taboo” with the word “arbitrary”.
Here is his first definition:
A piece of cognitive content feels “arbitrary” if it is the kind of cognitive content that we expect to come with attached justifications, and those justifications are not present in our mind.
But what does he mean by “justification”?
“Your mind labels X as a justification for Y, whenever adding ‘X’ to the pool of cognitive content would result in ‘Y’ being added to the pool, or increasing the intensity associated with ‘Y’.” How about that?
….We have the idea that different propositions, to the extent they are held, can create each other in the mind, or increase the felt level of intensity—credence for beliefs, desire for acts or goals. You may have already known this, more or less, but stating it aloud is still progress.
Of course justifications that sound reasonable to humans don’t have to sound reasonable to paperclippers:
…the feeling of “justification” cannot be coherently detached from the specific algorithm we use to decide justification in particular cases; that there is no pure empty essence of justification that will persuade any optimization process regardless of its algorithm…
Yudkowsky elaborates on an earlier dialogue of his, named “The Bedrock of Fairness“.
Here are just some quotes ’cause the whole post is neither important nor particularly convincing (IMHO):
You can argue about how to divide up the pie, or even argue how to argue about dividing up the pie, you can argue over what is fair… but there finally comes a point when you hit bedrock. If Dennis says, “No, the fair way to argue is that I get to dictate everything, and I now hereby dictate that I get the whole pie,” there’s nothing left to say but “Sorry, that’s just not what fairness is—you can try to take the pie and I can try to stop you, but you can’t convince that that is fair.”
The obvious reason to care what other people think is “fair”, is if they’re being moved by similar considerations, yet arriving at different conclusions. If you think that the Other’s word “fair” means what you think of as fair, and you think the Other is being honest about what they think, then you ought to pay attention just by way of fulfilling your own desire to be fair. It is like paying attention to an honest person who means the same thing you do by “multiplication”, who says that 19 * 103 might not be 1947. The attention you pay to that suggestion, is not a favor to the other person; it is something you do if you want to get the multiplication right—they’re doing you a favor by correcting you.
Which is to say that you can’t turn “fairness” into an ideal label of pure emptiness, defined only by the mysterious compulsion of every possible agent to admit “This is what is ‘fair’.” Forget the case against universally compelling arguments—just consider the definition itself: It has absolutely no content, no external references; it is not just underspecified, but entirely unspecified.