LessWrong post: GPTs are Predictors, not Imitators
In an attempt to probe the ineffable mysteries of ChatGPT, Eliezer Yudkowsky poses the following question:
Imagine yourself in a box, trying to predict the next word […] for all the text on the Internet. […] Is this a task whose difficulty caps out as human intelligence, or at the intelligence level of the smartest human who wrote any Internet text?
The correct answer is “neither”. And indeed that’s the answer that Yudkowsky arrives at, sort of: the original question is a clever ruse in which we have been set up to fail, so that a greater truth can be revealed to us by one of history’s greatest thinkers.
Being unfamiliar with any of the established literature in the field of computer science, though, Yudkowsky’s answer is simultaneously too short and too long, and ultimately a copout:
GPT-4 is still not as smart as a human in many ways, but it’s naked mathematical truth that the task GPTs are being trained on is harder than being an actual human.
What does “harder than being an actual human” mean, exactly? Yudkowsky doesn’t seem to know, as indicated by his only response in the comments section to one of his thoughtful critics.
So what’s the real, correct answer to the original question? Consider the following somewhat more technical rephrasing of it:
What is the shortest computer program that can generate a given sequence of symbols?
This is a well-studied question whose answer has a well-known name: Kolmogorov complexity. It is, in general, known the be non-computable.
This is a lot like if Yudkowsky had asked “which would halt execution first: ChatGPT, or a human?”, and then gone on at length about metaphors involving rabbits and hares or something. The average undergraduate computer science student, by contrast, would immediately recognize that question as being ill-considered. Asking that kind of question is often an indication that you have fundamentally misunderstood the larger task that you’re asking it in service of.
As mentioned above, one commenter gives a thoughtful response explaining why Yudkowsky is fundamentally wrong about his approach to this, and Yudkowsky rewards them with petulant dismissal, as is his way.
Lmao the linked commenter also caught this but uh… Yudkowsky telling on himself a bit here assuming people have to write rap battles in advance
Also, much like your grandma’s facebook posts, Yudkowsky apparently uses inexplicable capitalization rules: he always writes “Mind” instead of “mind”.
Guess a problem of Yuds way of being an autodidact is not picking up any of the relevant literature and reading it yourself. Think Claude Shannon might be a good start.
Anyway the thought experiment is misformed because if you are in the box like chatgpt you should also have an abridged/compressed copy of the whole internet with you. Which chatgpt has. (Op is slightly wrong here in linking to Kolmogorov complexity as iirc that is lossless way, while chatgpt is lossy, but still weird that Yud doesn’t talk about information theory at all).
Now if you were to plan to be put into a box and take something with you to reproduce (lossy) text from the internet and you cannot take the whole internet with you the problem is actually not hard. It is in fact very simple. You just make chatgpt, train that on the internet and take it with you. ;). We are a species of tool builders after all.
The whole ‘make words up’ argument is also a bit weird, esp as that is a known weakness of GPT. If you ask it to make up a new sentence it cannot really, while for humans we just throw some grammatically incorrect shit together and boom, brand new sentence (An LW expertise! ;) ). I get that isn’t what he is talking about btw, I just wanted to mention this GPT inability. Anyway, the whole make up random words thing is also odd as making up random numbers is right there, and a lot more predictable due to how keyboards are layed out, of course that wouldn’t make it predict a specific human, but just all humans who didn’t know that you should use a random number generator to make up numbers. (iirc this was a flaw of the Qanon posts, where the numbers were clearly random picked this way)
The reply is also a bit off btw.
People have predicted winning lottery numbers. If the method used to generate the lottery numbers is flawed. Which has happened, in fact globally there are enough flawed systems that somebody worked out which numbers are more likely. (this will not help people personally obviously, it is just somebody writing a few fun math science papers). And iirc people have also broken specific lotteries, but that might also be insider attacks or just other flaws in the system.
I also don’t agree with Yuds conclusion. Due to chatgpt having a lossy compressed version of the internet, it is both an imitator and a predictor. It uses prediction to create an imitation of the internet. It is both! (And wow this ‘stochastic parrots’ line really got under his skin).
🎵 Imagine yourself in a box on a river
🎵 With tangerine trees and marmalade skies…
The argument Yudkowsky makes here is one he’s made several times, and it has the same fundamental mistake. He considers an ML model whose goal is to predict the next token in a variety of everyday contexts. He correctly observes that to accomplish this task perfectly, the model would have to be capable of impressive things like being able to invert hash functions, or simulate the inner workings of human brains.
Yudkowsky clearly wants the reader to take from this that predictors are magic and will be able to invert hashes in the future, which is absurd. (Though he stops short of actually reaching this conclusion in the post.) The obvious correct conclusion is that predictors will never be able to perform the task of prediction perfectly due to limits on computational power.
(By the way, Kolmogorov complexity is not relevant here. The relevant formalism for discussing hashes is the notion of a one-way function; or the discussion could be rephrased in terms of NP-complete problems rather than hashes.)
“As Ilya Sutskever compactly put it, to learn to predict text, is to learn to predict the causal processes of which the text is a shadow”
This kind of mistake is fundamental; it’s precisely to predict the shadow.
Sorry, but I’m lost somewhere in the middle of your post. Why would Kolmogorov complexity being incomputable, or the Halting problem for that matter, make Yud’s questions ill-considered?
Being incomputable doesn’t mean it doesn’t have a truth value. Kolmogorov complexity of a given language is a number, a well-defined one. A machine always either halts or doesn’t, and if two of them halt then we can ask which does it in fewer steps. You can’t write an algorithm that computes those answers, but the answers themselves exist, and are well-formed. You can ask the question.
In particular, the problem of “does this particular machine halt” is not uncomputable. It’s uncomputable to answer “for any machine, tell if it halts”. If you ask whether ChatGPT halts the answer is… yes. Yes it does. You could look at the source code (if it was public) and analyse it to arrive at that conclusion. We can use various formalisms like Hoare’s logic to derive a proof of halting for a concrete source code. And from how it behaves it’s clear that it always halts, simply because after some time it will just tell you “sorry, I timed out”.
The problem of “which formal language is more complex, A or B” for some concrete A and B can also be solved. For example, if both languages were regular, we can precisely tell the sizes of their respective automata that compute them. It’s not enough to say “Kolmogorov’s complexity is an uncomputable function” to dismiss such questions as ill-considered.
[deleted]
I don’t think predicting text is quite the same as computing its Kolmogorov complexity. However, if we model human beings as Turing machines, then the problem of predicting any nontrivial semantic facts about the future behavior of humans is undecidable by Rice’s theorem.
If we restrict the problem so that we are only interested in looking for solutions we can verify quickly, then it becomes NP-complete. In fact, most of the problems that would associate with “intelligence” are going to be NP-hard in the worst case. So, in that sense, all intellectually hard problems are basically the same; they all efficiently reduce to solving elaborate 3-SAT instances. But I think Yudkowsky is using the word “hard” in a slightly more loose and informal way.
For example, it is known that, in general, finding a solution to a Mario level is NP-hard, but we don’t usually consider playing Mario games to be especially hard in practice. Though, by this standard, I’d still have to somewhat disagree with him. It intuitively doesn’t feel like there’s much of a difference in difficulty between predicting when I will make mistakes and preventing myself from making those mistakes.
Although, I think he’s right that GPT can do a bunch of amazing things that humans cannot do, but it seems like a leap to go from this to these crazy godlike AGI conclusions. Computers already regularly solve problems that humans consider informally “hard”. It would take me ages to accurately compute 2^(1/50) to more than a few decimal places, but a computer can do it in seconds. Likewise, it would probably take several human lifetimes to solve a 3-SAT instance with over a million variables, but computer 3-SAT solvers have been able to do this in days or weeks.
Yudkowsky is familiar with Kolmogorov complexity (of course). He has written about this.
You do agree that the more intelligent you are, the easier it will be to predict text, right?
I fail to see any valid criticism here.
Is there a meaningful distinction here between ‘prediction’ and ‘imitation’?
I’m not at all deeply informed about this topic. It comes off to me like he’s getting hung up on semantical differences between these two words that don’t really matter with regards to the larger converstion.