Don’t worry about the AGI becoming self-aware, worry about Yud becoming self-aware.

Dumb of concernedcitizen64 to post this, as remember: “YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES”

(E: while this is obv a joke, it also points to a previous action in the past where Yud actually thought he was smarter than everybody else (for seeing the “risk” of the thought experiment) and he took action against people, so this post is also a reference to a thing related to what concerned is complaining about (it gets even better as yud is claiming he would never do social pushback against people in the linked post (which technically, deleting your posts isn’t of course ;) ))).

It really feels like the LessWrong “rationalist” community has just re-invented religion, with a thin veneer of scientism and technology over top of it. They can’t see it because it’s not supernatural in nature, but it has all of the functionality of a religion nonetheless.

I wonder if this happens anytime you think you’ve constructed the perfect method of doing epistemology? Since that seems to be the core of their movement. I’ve seen people claim that’s the case, but I’m not so sure.

[Edit: rewrote this meditation on epistemology to be more general and far clearer.]

Even if you think you’ve found the best way to come up with beliefs that are the most probably true given your available evidence, that shouldn’t give you the kind of certainty these people speak with.

Fundamentally, all the rationality in the world won’t do much more than elaborate on the input you give it - garbage in, garbage out and all that. And what do we have to put into our rational “calculations”? The only input you actually have are your personal, subjective, situated experiences. You don’t actually have access to objective facts or evidence, if those things even exist - which I’m not totally clear on. Everything you have is mediated by your perspective and what you can actually experience.

Statistics and science isn’t a get out of jail free card for getting objective evidence, either: a lot of times, crucial information about a situation simply can’t be communicated in scientific, and especially quantitative, measurements (such as how difficult it actually is to be homeless to use a recent example where Scott was being an idiot). Additionally, what science has available to it or what results it produces is often extremely influenced by the biases and perspectives of the scientists who perform it. The Mismeasure Of Man is an excellent book on this topic.

This leads me into my second point: experiences can be interpreted in an almost infinite number of ways, and while some interpretations have superior intellectual virtues or are more predictive, and therefore more useful, there is a lot of bias and aesthetic or subjective valuation that goes into choosing an interpretation of your experiences, so you have to not only be aware of that, but very careful of that. A lot of people that talk a lot about logic and rationality tend to just take their biased assumptions as unearned axioms: different definitions and categories, or assumptions about how the world works, can vastly shape your interpretation of everything and can seem so obvious to you that anyone questioning them can sound incoherent or irrational, but those assumptions can be completely unjustified and, from another perspective, untrue and useless.

All of this compounds with the fact that while different interpretations can have different practical benefits - such as being more predictive, or simpler, or more phenomenologically conservative, or more rewarding - there’s nothing really to say which of those practical benefits should outweigh the others. Personally, I prefer maximizing predictiveness and as a secondary trait phenomenological conservation, but it’s not objective.

So you end up with a huge number of points of valid difference in what you can believe, where people with personal experience of something have a lot more to say, and situated experience matters a lot, and somehow out of this these people are getting such absolute certainty?

Finally, Yud doing something useful.

link to article

Too many word make head hurt

Gonna go out on a limb here and predict that EY will not actually discover that at all, no matter how self-evident it is.

Let’s read any responses he had to the comments…

Yeah.

lol

lmao, even