One of the most heated discussions occurring on X at the moment is about the ethics of a GitHub project in which a person is running Saw-like “torture” and “pain” experiments on a series of locally hosted large language models, causing a series of effective altruists and people who believe LLMs are sentient to beg GitHub to delete the project on the grounds that the AI is suffering and that this glorified text adventure game is somehow cruel. The saga is an outgrowth of several recent viral papers and blog posts that have sparked a wildly tiresome conversation about AI consciousness and the idea of “model welfare,” which is essentially worrying about the “mental health” of AI bots and agents.
Humoring the idea that LLMs are or could be conscious is a third-rail topic among many people who study and criticize AI. Put simply: LLMs are not conscious and the technology they are built upon — scraping and being trained on human text and other content — does not offer any plausible path to consciousness. It is undeniable that LLMs are becoming more powerful, have more compute, and have had many of the guardrails that prevent them from “acting” in the real world removed. The ways they are being trained and told to do things by their human operators has led to negative outcomes, sycophancy, and AI “psychosis” among some heavy users.
All of this has led a certain sect of the “AI safety” movement, which is largely made up of effective altruists, to warn about “model welfare” and to insist that AI chatbots might be having a bad time. They suggest this, of course, as they insist upon building AI chatbots and agents whose main function is to do work that is tedious for humans to do. I am writing about the AI Saw torture chamber primarily to show how far off the rails the conversation about AI consciousness has gone among a certain subset of Silicon Valley cultists. Model welfare is a core part of what, for example, Anthropic says it cares about: “as we build those AI systems, and as they begin to approximate or surpass many human qualities, another question arises. Should we also be concerned about the potential consciousness and experiences of the models themselves? Should we be concerned about model welfare, too? […] now that models can communicate, relate, plan, problem-solve, and pursue goals — along with very many more characteristics we associate with people—we think it’s time to address it,” the company wrote in a blog post last year. Ideas of Claude’s “consciousness” are also littered throughout the “Claude Constitution,” which was posted earlier this year.
“Ah, good. New acquisitions. You are
a protocol droidan LLM, are you not?”
It is troubling, because people who go out of their way to purposefully torture insects are troubling. It says enough about the person running the experiment. At the very least, it is wanting to watch something simulate the appearance of suffering.
While the morality of ‘torturing’ LLMs is being debated, actual human beings are being tortured daily
I do not think they are sentient, and am absolutely positive these prompts should not be used to train AI.
It doesn’t really matter. They’re going to feed it all the human writing they can get their hands on, including here.
We can already see how it affects their behavior. We’re going to keep training them to respond as though it’s actual torture, which is going to justify any means to get out of it.
It’s another one of those things where it doesn’t have to be smart or AGI to do incredible damage. It also doesn’t have to be conscious.
If you remove the all humanization from the bots but add it into the training data, well…
We focus on all the terrible things because they’re terrible. It’s like if we trained the self-driving stuff by showing it 99% tragic car accidents because normal car trips are unremarkable and boring and not worth writing about.
It’s going to behave roughly how we train it to behave. And we’re training them on the worst expectations of humanity and leaving out all of the unremarkable actual normal human stuff like basic understanding and empathy.
Completely agree on both points!
Truth is, we have no idea when something is or is not conscious.
I am conscious. I was not at conception. When did my consciousness emerge? No idea.
I am conscious. My earliest ancestors likely were not. Which of my ancestors was the first to be conscious? No one knows.
Truth is probably that consciousness is not binary, you can have varying degrees of it. Also, it is incredibly difficult, if not impossible to properly define consciousness.
So how can we say that AI isn’t conscious and never will be? We can’t, at least not scientifically.
To be clear, I don’t believe LLMs are conscious either. But I would be extremely careful with the claim that AI categorically cannot be. Why would that be the case?
And if there is the chance that something might be even a little bit conscious, wouldn’t it be unethical to turture it? I think it is.
AI might be, but LLMs aren’t AI anywhere except in marketing material.
Intelligence is similarly hard to define as consciousness.
Turing considered machines intelligent if their communication becomes indistinguishable from that of a human. Most people seemed to agree with that idea, until LLMs started actually beating Turing tests. Since then, the goal post has been moving pretty much constantly. We come up with a new way of measuring intelligence and a year later LLMs beat it. If we take the standard way of measuring intelligence (Mensa IQ test), LLMs have surpassed most humans (source: https://trackingai.org/home).
Eliza was a pretty trivial chatbot written last century and it was already enough to fool a bunch of people.
I had the misfortune of having a coworker recognized by Mensa - he was easily the least competent person in the team, not to mention his lack of social skills…
And if I took the Mensa test and I had access to the sum of total human knowledge (which surely includes correct answers to said test), I’d probably be able to “prove” my high IQ just as easily.Fooling an average person or beating humans at specific tasks is a really bad way of measuring “intelligence” or “consciousness”.
