It is a bit more than that: CEV is what he would want if he were wiser and less confused. Yudkowsky’s vision was that we want a lot of things which are contradictory or conflict with others or will make us sad, but Friend Computer could sort that out. But talking your friend into going to an event or trying a new food which she actually likes when she tries it is definitely in the spirit.
CEV is what he would want if he were wiser and less confused
Isn’t that just steelmanning?
I gathered the “idealized version of myself” was because it’s supposed to be applied to a superintelligence, because of course it’s an alignment thing.
Steelmanning is making the best possible argument for a position, whereas CEV is sorting out all the delusions and contradictions in someone’s thinking and giving them what they would want if they were wise enough to know it. Central bankers engage in extrapolated volition when they try to make the economy run in a way that will make people happy, even if what they do is not what the woman on the street wants them to do because the woman on the street has no idea how the economy works. Friends engage in extrapolated volition when they intervene in a marriage or a drinking bout and say “you are ruining your life, and we are stopping it now.” Extrapolated volition is paternalistic (“you think you want that, but I know better …”) and Yudkowsky’s CEV would demand God the Father. Yud’s original paper is available.
So, CEV presupposes false consciousness - that the average person’s belief system is misaligned from reality due to incomplete information, incorrect presuppositions, and so forth.
And the idea is that a wise leader will choose for the people what the people would choose for themselves if they had a correct understanding of reality, whether the people think they want that or not?
I guess today in LessWrong, we are re-inventing Marxism.
One must always keep in mind that the Rationalist project is explicitly a high-modernist effort; it is a permanent fight against postmodernism which it can never win, a philosopher’s lost cause. They can only look at Marxism as low art which must be elevated by sanctifying it with the nebulous ointment of “Western civilization”.
It is a bit more than that: CEV is what he would want if he were wiser and less confused. Yudkowsky’s vision was that we want a lot of things which are contradictory or conflict with others or will make us sad, but Friend Computer could sort that out. But talking your friend into going to an event or trying a new food which she actually likes when she tries it is definitely in the spirit.
Isn’t that just steelmanning?
I gathered the “idealized version of myself” was because it’s supposed to be applied to a superintelligence, because of course it’s an alignment thing.
Steelmanning is making the best possible argument for a position, whereas CEV is sorting out all the delusions and contradictions in someone’s thinking and giving them what they would want if they were wise enough to know it. Central bankers engage in extrapolated volition when they try to make the economy run in a way that will make people happy, even if what they do is not what the woman on the street wants them to do because the woman on the street has no idea how the economy works. Friends engage in extrapolated volition when they intervene in a marriage or a drinking bout and say “you are ruining your life, and we are stopping it now.” Extrapolated volition is paternalistic (“you think you want that, but I know better …”) and Yudkowsky’s CEV would demand God the Father. Yud’s original paper is available.
So, CEV presupposes false consciousness - that the average person’s belief system is misaligned from reality due to incomplete information, incorrect presuppositions, and so forth.
And the idea is that a wise leader will choose for the people what the people would choose for themselves if they had a correct understanding of reality, whether the people think they want that or not?
I guess today in LessWrong, we are re-inventing Marxism.
One must always keep in mind that the Rationalist project is explicitly a high-modernist effort; it is a permanent fight against postmodernism which it can never win, a philosopher’s lost cause. They can only look at Marxism as low art which must be elevated by sanctifying it with the nebulous ointment of “Western civilization”.