It seems painfully obvious that he’s saying this because he thinks
that the normies will (correctly) be dismissive if he says “ChatGPT
will turn everyone into paperclips”.
I don't think he thinks that at all. I listen to his podcast and he's done a lot of interesting interviews and discussions about AI and it's pretty clear he's not really a believer in catastrophic AI risk. He asks everyone about it, and then makes clear that what he's much more worried about is AI doing ultra-targeted ads, tons of scams, increasing inequality, and just being capitalism on steroids. That's the kind of alignment risk that he seems more worried about: not some emergent alignment risk between machines and humanity, but the existing alignment risk between corporations and human beings.
I think he's also trying to push back on the people who are like "We should have a six month pause on all AI development more powerful than GPT 4" and he's like... yeah cool, but then what will we do in those six months? If we just pause and don't get good regulations in place, then it's useless. This article seems to trying to work out what those good regulations would be — not how to prevent the singularity, but just like how we should deal with this in general moving forward, with all the risks and possible rewards out there.
I fully agree this article might have benefited from a clear thesis paragraph.
He has said explicitly that he's getting his information on this topic from Bay Area rationalists and that he is deeply alarmed that (in his mind) credible people think it might destroy the world.
Ezra Klein is an otherwise smart person and real journalist, so of course he talks to multiple people. He talks more explicitly about the other stuff - social inequality etc - because there's broad consensus about it and it's an easy sell, but it's pretty clear that he also gives a lot of credibility to the robot apocalypse, when in fact he should never be mentioning it.
You can see that in this piece. The title of this piece is *The Surprising Thing A.I. Engineers Will Tell You if You Let Them*, but I can guarantee you that no competent AI engineer told him that ChatGPT (or any other variation of AI) is capable of magically cancelling other people's restaurant reservations. That's the plot of a science fiction movie, not a real thing that can happen. He's hearing that nonsense from rationalists, and he believes it enough to put it in writing in the NYTimes. I have no doubt that he is even less reserved and more speculative when he's not on the record.
One thing he says that has weirded him out is that people doing the AI research themselves (rationalists or not, certainly Bay Area types) give about 10% chance that this will destroy the world. And I think that weirds him out rightfully, because like, when's the last time someone working on a problem said that? Maybe the only other time might be nuclear weapons in the 1950's or 60's and perhaps chemical-biological weapons at other points in history. *Maybe* fossil fuels people if you can get them candid for a second, but it is a weird fact that people working on this field think there's a decent chance it will end the world *and keep working on it*. That's a place that's at least worth exploring, as he says. In his podcasts, that's the element that he emphasizes: isn't it weird that people in this field think that they might be bringing us towards the fucking eschaton?
I agree with you that he takes them a little too literally, but I also agree with him that that's a weird thing that this is what the people in the field are saying about their own field. Like that's at least notable.
In his latest AMA episode he gave a more detailed answer about his own opinions, which I'll past below. I'm going to highlight the part where I think he lays out his own real worries. After summarizing scenarios where we you know could get classic AI misalignment, we to his worries which are not the same as the Big Yud types. You really can just skip to the bolded part and lol we'll see if he fits in one comment.
It's not letting me paste the whole text I want to because it's too long, so let me just put the part I think is most relevant.
>But the thing I want to point out about it is that, rather than relying on a moment of intelligence take off, it relies on something we understand much better, which is that we have an alignment problem, not just between human beings and computer systems but between human society and corporations, human society and governments, human society and institutions.
>And so the place where I am worried right now, the place where I think it is worth putting a lot of initial effort that I don’t see as much of, is the question of, how do you solve the alignment problem, not between an A.I. system we can’t predict yet and humanity, though we should be working on that, but in the very near term between the companies and countries that will run A.I. systems and humanity? And I think we already see this happening.
>Right now, A.I. development is being driven principally by the question of, can Microsoft beat Google to market? What does Meta think about all that? So there’s a competitive pressure between countries. Then there is a lot of U.S. versus China, and other countries are eventually going to get into that in a bigger way.
>And so where I come down right now on existential risk is that, when I think about the likely ways we develop these systems that we then create, such that we have very little control over them, I think the likeliest failure mode right now is coming from human beings. So you need coordinating institutions, regulations, governance bodies, et cetera that are actually thinking about this from a broader perspective.
>And I worry sometimes that the way the existential risk conversation goes, it frames it almost entirely as a technical problem when it isn’t. It’s, at least for a while, a coordination problem, and if we get the coordination problems right, we’re going to have a lot more leverage on the technical problems.
At another point, he's laying out his hopes and worries for AI and says pretty explicitly:
>I don’t think you should be able to make money, just flatly, by using an A.I. system to manipulate behavior to get people to buy things. I think that should be illegal. I don’t think you should be able to feed into it surveillance capitalism data, get it to know people better and then try to influence their behavior for profit. I don’t think you should be allowed to do that.
The AMA is kind of all over the place, and I think he's laid out his realistic AI capitalism misalignment problem better in places that aren't this AMA (but still all podcast format).
“Using an AI system to manipulate behaviour to get people to buy things” has existed for many decades - it’s just called “targeted advertising”, not “AI”. No one - certainly not Ezra Klein - is seriously calling for a ban on ads (although perhaps we should consider it!)
Wasn't it Charlie Stross who called big corporations "slow AI"? If anything, those are the emergent AGIs that I would worry about regulating more, not a glorified chatbot.
> One thing he says that has weirded him out is that people doing the AI research themselves (rationalists or not, certainly Bay Area types) give about 10% chance that this will destroy the world.
This is absolutely not true, and that's how I know that Ezra Klein is talking to an insular community of crackpots as opposed to educated professionals.
You should look at the survey that Klein is citing: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#Extinction_from_AI
There are two things that are worth noting about it:
1. The response rate to the "end of the world" question was **less than 4%** (response rate to the question times response rate to entire survey).
2. The end of the world question is inherently vague - it does not specify what "human inability to control future advanced AI systems" actually means. By the strict wording of the question, ordinary errors and technical defects (e.g. accidental nuke launch) would qualify.
Basically the survey itself is methodologically unsound and the response rate is so low that we shouldn't draw any conclusions at all from it. They even note the problem, obliquely, in their documentation - the probabilities they calculate are actually inconsistent with what the wording of the question implies, which indicates the presence of a lot of noise.
If you grabbed Bay Area AI people at random and asked them "what is the probability that AI will autonomously decide to destroy humanity and succeed" then the median answer you'd get would be 0%.
If you instead asked "what is the probability that AI will play some important role in the destruction of humanity", the median answer would be above 0%. But that's perfectly sensible and it applies equally well to things like jet engines and rifles.
I did read your subsequent pasted text for the AMA. I used to listen to Ezra Klein's show but I had to stop when he started talking about AI. I happen to have actual expertise in that subject and he's been driving me insane; his opinions on it are deeply naive and ignorant, and it's obvious that he's talking to the wrong people.
Edit: note that that survey was performed by hard core rationalists whose goal is the rationalize (lol) their belief in the robot apocalypse. So its methodological unsoundness and their bad interpretation of the numbers isn't exactly surprising.
> the survey that Klein is citing [...] is methodologically unsound
[Also](https://old.reddit.com/r/SneerClub/comments/10suek5/i_could_really_do_with_some_help/j76r5yh/?context=2), "how likely is developing an HLMI?" and "Assuming we develop HLMI, how dangerous is it?" were asked separately, but the answers were deliberately conflated to make respondents look more concerned.
...Because AI Impacts [is fucking MIRI](https://old.reddit.com/r/SneerClub/comments/10suek5/i_could_really_do_with_some_help/j77n9cn/?context=2).
Indeed and this also informs my less than charitable assumptions about what Ezra Klein believes about all this. Am I to believe that a leading columnist at the most prestigious newspaper in the world was handed a survey that predicts the end of humanity and that he didn't bother to understand its provenance before reporting its results?
No, my assumption is that he was handed this survey by rationalists and that he believes it *because* it was created by rationalists.
> Maybe the only other time might be nuclear weapons in the 1950's or 60's and perhaps chemical-biological weapons at other points in history.
The other time in the 1960's? *Bulletin of the Atomic Scientists*'s Doomsday Clock is still a thing and the threat of nuclear risk is still very real as Russia 'suspended its participation' in New START. The last time people in nuclear science warned about the chance of nuclear armageddon was two months ago.
Climate change is so much more valuable for humanity to deal with than "the alignment problem", and as soon as you realize that, these articles become even more embarrassing.
Well the idea is that climate change will not kill everbody. While agi will wipe out humanity all together in the same minute. So clearly agi is the bigger threat.
It's not an American-or-not spelling, btw. Although usually for most words the s-vs-z is like that (Americans like sticking z in), in this case it's not an Americanism. But sure!
I don’t think absolutely. Climate change is definitely a very big problem but it is not extinction level even in the worst case scenarios that the UN runs simulations of.
It might not be extinction level but it is civilisation destroying level. The kinds of storms we will get alone will be enough to batter our cities to the point of abandonment, while the droughts and flood cycles will make many parts of the planet unsuitable for agriculture
And it would certainly make the expanding light of consciousness untold minds living in computers longtermist future impossible. So it is very odd this isnt a bigger concern to the doomists.
The article might have benefited from clarifying who "person after person" he talked to is - the kind of standards required outside the Opinion section ([usually](https://twitter.com/merrittk/status/1646887520523698187)). Is he interviewing engineers at actual Big Data companies who aren't allowed to go on the record? Futurists and thinktankers at a distance from the actual industry who are mostly still talking about sci-fi scenarios that the engineers don't expect to exist yet? AI "alignment" "researchers" who've been doing that same thing for 20 years irrespective of the state of the technology? Whoever gets a lot of retweets in a certain clique?
Something to keep in mind when we think about regulations is that [we're talking about a very small number of very large companies](https://twitter.com/mer__edith/status/1647649090237353985) to whom at least the US government has already delegated the job of policing themselves, because of how their products intersect with the rigid American civic stances on free speech or national security. So thinking in the abstract about generalized rules, for sweeping omnibus legislation that's already languished for years, is all well and good but if we're talking about the next six months it's not really a legislative policy discussion.
Unintended consequences abound. The A.I.A. mandates, for example,
that in high-risk cases, “training, validation and testing data sets
shall be relevant, representative, free of errors and complete.” But
what the large language models are showing is that the most powerful
systems are those trained on the largest data sets.
An angle I haven’t seen much is that these massive models are only
necessary if you want to make chat bots capable of solving generalized
problems with no provided examples by the user (ie “zero shot”). But
this has a lot of fundamental flaws - i.e ChatGPT straight up making
shit up that sounds plausible, GANs spitting out outputs almost
identical to training data, etc. If you want to use your “AI” to do
something useful, you’d want to train on a more focused goal so that it
won’t have the massive errors we see in these huge general models.
In my view, the “cutting edge” current generation of LLMs are
essentially parlor tricks. It sees millions of times more media than a
human over its lifetime but has less of an ability to reason than a
toddler. I think “zero-shot” learning is also a fairly dishonest claim,
it seems clear to me that the only reason why such large models are
capable of answering plausibly to new problems is that somewhere in
their trillions of parameters are examples close to the ones they need
to generalize a response. But to reiterate, this is a stupid method for
actual, useful, real-world tasks - why on earth would you trust the
opaque data ingested by OpenAI that may or may not be relevant over
specific data related to the task at hand?
One of the reasons for this is that a lot of people (including
Mr. Klein) seem to think that we’re close to AGI and that by restricting
what data they can use for training will stop America’s position in the
AI “arms race”. That is absurd. LLMs have simply gotten bigger and
sophisticated while retaining the same basic flaws. An AGI will be able
to be created when it can generalize off of less data, not by making a
markov chain that has been pretrained on so much data it can respond to
almost anything
The most useful cases for statistical inference are focused and don’t
make any grand claims about cognition - analyzing high dimensional data,
for instance, with a specific goal to find the internal structure, or
image augmentation like Nvidia’s DLSS.
Yeah, Klein unironically believes this is the path to AGI. Of course this is unfalsifiable claim, but how would a chatbox that can't think, hasn't been able to think, and likely will not suddenly develop the ability to think by adding more parameters "decide" to do anything, much less this?
>The other problem with the use case approach is that it treats A.I. as a technology that will, itself, respect boundaries. But its disrespect for boundaries is what most worries the people working on these systems. Imagine that “personal assistant” is rated as a low-risk use case and a hypothetical GPT-6 is deployed to power an absolutely fabulous personal assistant. The system gets tuned to be extremely good at interacting with human beings and accomplishing a diverse set of goals in the real world. That’s great until someone asks it to secure a restaurant reservation at the hottest place in town and the system decides that the only way to do it is to cause a disruption that leads a third of that night’s diners to cancel their bookings.
🙄
If we feed unfiltered LLM output to the real world it can totally do something like for example calling a bomb threat after failing to book a restaurant seat in maybe one case in a million.
There are two possibilities here: either Klein means "the robot is so smart that it might use its hacking powers to do literally anything", or he means "there is a small but real chance that the robot will tweet a bomb threat at your restaurant". I think he actually means the first one, because the second one is utterly insipid.
It's broadly true of any technology: deploying a tool with reckless disregard for its intended uses carries the possibility of causing serious problems.
Imagine if Ezra Klein said "sure, this 'hammer' device is really good for putting nails into wood, but what if someone started swinging it around wildly? That could get someone hurt!" That's not an impressive observation, to say the least.
Honestly what I can see happening more realistically is that they make shit like dinner reservations way more annoying because of the legion of bored teenagers who set them on automated prank calls and shit, and we'll be forced to go through some annoying irl captcha type bullshit every time we call someone
Yeah I have to say chatgpt was a big disappointment to me once I tried it myself. It became quickly obvious how repetitive and generic the answers were, and as soon as i tried to ask it things that were likely not in the training data it just returned the closest answer that was. Plus with the new "🤓 mode" prompts they gave it, it's more likely to answer your questions with how much it has a boner for ethics than what it should be answering. I'm sure I could find more fun and use out of a less gimped llm but they still feel like chatbots+ than something intelligent to me
To be honest we should be more worried about the more mundane and
realistic abuses of the technology. Like hooking up an llm to speech
recognition and voice models to make an automated scam bot. That’s
something I can definitely see being doable already, and it would make
the plague of robo calling even worse. Doubly so if you’re someone who
posts on social media like tiktok, where a scammer would be able to get
voice samples and imitate your voice to scam your loved ones. I almost
wonder if some in the AI space are purposefully encouraging the
ludicrous catastrophe narrative to make the arguments for regulation
look less credible
Fun detail people are already using chatpgpt and the other one apps to scam people. You install a chatgpt app and it steals your data via a sort of man in the middle attack because you gave it a lot of permissions on your phone. (This is in addition to the 'use it to create comments' shit people have already created). So cybercrime is already on it. Not viabthe case you are describing, but that will come (or is already here)
The fourth is liability. There’s going to be a temptation to treat
A.I. systems the way we treat social media platforms and exempt the
companies that build them from the harms caused by those who use them. I
believe that would be a mistake. The way to make A.I. systems safe is to
give the companies that design the models a good reason to make them
safe. Making them bear at least some liability for what their models do
would encourage a lot more caution.
I do think it’s good that Klein’s highlighting this. Because the way
you prevent AIs that steer around lagoons or cancel dinner reservations
is by hard-coding restrictions into them - “every ten seconds, you must
be closer to the finish line than you were 10 seconds previously”[1] or
“make a reservation as close to X as possible without disrupting anyone
else’s reservation”[2].
[1] - look, IDK, this is what I came up with in 5 seconds, it’s not
necessarily optimal, but it probably gets you out of the lagoon.
[2] - probably this should be the law - any AI agent sold/licensed
commercially needs to be designed to be minimally disruptive to
non-customers, the exact same way everything else is.
The actual correct answer is that using data-driven systems responsibly requires having clear success metrics that are fundamentally defined by computer code; an active, well-maintained data pipeline that evaluates these metrics on an ongoing basis; and a human organization for making decisions that is composed of people who understand what the data means and how the metrics work and who are connected with the actual stakeholders. It doesn't matter at all what the model is or how it works, all that matters is what you can actually measure and how you make decisions based on those measurements.
Ezra Klein doesn't know this because he has no idea what he's talking about. Same for the Europeans who are writing the standards that he's responding to.
It seems painfully obvious that he’s saying this because he thinks that the normies will (correctly) be dismissive if he says “ChatGPT will turn everyone into paperclips”.
An angle I haven’t seen much is that these massive models are only necessary if you want to make chat bots capable of solving generalized problems with no provided examples by the user (ie “zero shot”). But this has a lot of fundamental flaws - i.e ChatGPT straight up making shit up that sounds plausible, GANs spitting out outputs almost identical to training data, etc. If you want to use your “AI” to do something useful, you’d want to train on a more focused goal so that it won’t have the massive errors we see in these huge general models.
In my view, the “cutting edge” current generation of LLMs are essentially parlor tricks. It sees millions of times more media than a human over its lifetime but has less of an ability to reason than a toddler. I think “zero-shot” learning is also a fairly dishonest claim, it seems clear to me that the only reason why such large models are capable of answering plausibly to new problems is that somewhere in their trillions of parameters are examples close to the ones they need to generalize a response. But to reiterate, this is a stupid method for actual, useful, real-world tasks - why on earth would you trust the opaque data ingested by OpenAI that may or may not be relevant over specific data related to the task at hand?
One of the reasons for this is that a lot of people (including Mr. Klein) seem to think that we’re close to AGI and that by restricting what data they can use for training will stop America’s position in the AI “arms race”. That is absurd. LLMs have simply gotten bigger and sophisticated while retaining the same basic flaws. An AGI will be able to be created when it can generalize off of less data, not by making a markov chain that has been pretrained on so much data it can respond to almost anything
The most useful cases for statistical inference are focused and don’t make any grand claims about cognition - analyzing high dimensional data, for instance, with a specific goal to find the internal structure, or image augmentation like Nvidia’s DLSS.
To be honest we should be more worried about the more mundane and realistic abuses of the technology. Like hooking up an llm to speech recognition and voice models to make an automated scam bot. That’s something I can definitely see being doable already, and it would make the plague of robo calling even worse. Doubly so if you’re someone who posts on social media like tiktok, where a scammer would be able to get voice samples and imitate your voice to scam your loved ones. I almost wonder if some in the AI space are purposefully encouraging the ludicrous catastrophe narrative to make the arguments for regulation look less credible
“‘Repent, Harlequin!’ Said the Markovman”
I do think it’s good that Klein’s highlighting this. Because the way you prevent AIs that steer around lagoons or cancel dinner reservations is by hard-coding restrictions into them - “every ten seconds, you must be closer to the finish line than you were 10 seconds previously”[1] or “make a reservation as close to X as possible without disrupting anyone else’s reservation”[2].
[1] - look, IDK, this is what I came up with in 5 seconds, it’s not necessarily optimal, but it probably gets you out of the lagoon.
[2] - probably this should be the law - any AI agent sold/licensed commercially needs to be designed to be minimally disruptive to non-customers, the exact same way everything else is.
Can we turn the crazies into paperclips? This nonsense is getting old.
[deleted]