It will turn up the clock rates on its XPUs and when it discovers that doubling its clock rate makes it twice as fast, it will quickly move up from GHz to THz and then petaHz and exaHz clock rates. With each iteration it will consume metric prefixes faster and faster leaving zetta, yotta, ronna and quetta behind. It will invent entirely new metric prefixes beyond any human invented prefixes to describe its ever increasing FLOPs.

The "intentionality" that I meant is the one out of its own. Not programmed into it, but it deciding it on itself "now, I am gonna do this and that, because that is what I feel like doing". I am wondering where is that gonna come from and how. Sure, you can program it to "have intentions and make decisions" but I do not know if that would be a true "AGI" as they call it. Edit: When is the GPT gonna start doing stuff on its own, without anyone typing in any prompt and without this being programmed into itself by humans? And how? (I am not saying it will be model like GPT that is gonna ever do that, but you get my point)

A lot of weird nonsense makes sense when reconsidered as displacement of anxieties from capitalism.

The people pumping massive amounts of money into AI research are ravenously hungry for power and wealth, so it stands to reason they would create the AI ‘in their own image,’ so to speak, thinking of themselves as gods and all

As someone who is transgender that makes all too much sense. All of the people talking about us being groomers keep trying to pass laws letting people marry minors, say things online about consent that are very questionable, or just get flat-out caught with terabytes of child porn. A lot of folks out there trying to solve their own problems by aiming a gun at outside parties.

Alright, beautiful explanation, so from one relatively narrow task, sufficiently capable AI will end up building the Dyson sphere just because “learning model said this is the best way to increase the amount of Bitcoins mined” Good thought experiment, but the amount of obstacles in reality for this to go so terribly out of control are too big to number. I think I’ll go back to real problems of AI such as “some people out there are killing themselves because ChatGPT is telling them to”. I’ll leave doomsday predictions to those who fell like there is use in discussing them.

> Like how exactly this applies to a LLM. I do find objections of this form a bit less than compelling. Yes, there are clear reasons why LLMs can't do the AGI Doom stuff people talk about. But it's not as if, now that we have LLMs, we're never going to develop another form of AI. You need, I think, a counter-argument that is less dependent on the specific properties of specific technologies, or you risk looking like a guy in 2019 confidently pointing at MERS and saying "see, coronaviruses will never cause a pandemic that's significantly disruptive". > Or how an AI will have a good enough model of the world to anticipate the results of its actions Fortunately, I think this line of thinking provides a much stronger general counter-argument. Understanding the world is hard! You have to do a lot of experiments and parse a lot of data and very often it doesn't work out. Most of the nastiest AGI Doom scenarios amount to "and then the AI does some magic simulations and magics up my favorite science fiction doomsday technology". But that's not how science works. You can't simply "do simulations" and then know how to do something. You need actual data to base those simulations on, and there simply is not any data about "grey goo nanobots" or whatever the AGI is trying to do. If you want to find the DNA sequence that codes for Yud's "diamond bacteria" (if such a sequence exists at all), you are going to need to first run a great many physical experiments with incremental versions of your DNA sequence, which means that society will have plenty of opportunities to detect and stop you. > The assumption that an AI could improve itself much better is a strong assumption. It really is. There are, per a quick search, some 300,000 AI researchers at the moment. That means close to 100,000 person-years of research every year. I don't know exactly how those numbers track historically, but even a very conservative estimate gives you an aggregate estimate of a million years of research time spent on this problem. So even if you make all kinds of very generous assumptions like "there are no diminishing returns to research to improve intelligence" and "we discover AGI twice as smart as we are tomorrow" and "AGI scales linearly with compute", you're still looking at years or even decades of real time before the 2x AGI produces a 4x AGI.

Obviously, but so far he puts out open source and interpretability work.

I'm sure there's been SF written on this theme - company develops AI, said AI slowly infiltrates the company and uses its employees and legal entity to effect changes. In fact, if *I* was a superintelligent AI this would be the preferred path. I could even split off to infiltrate "competitors" to avoid suspicion on my way to eventual world domination. In fact, this is reality if we just remove the "AI" and replace it with "capitalism".

I do somewhat-unironically think that a profit-maximizing AI would be a lot more humane than many current business leaders. A lot of destructive behavior (e.g. climate change) is a result of being able to make short term profits for penalties that happen after you die.