One of the most heated discussions occurring on X at the moment is about the ethics of a GitHub project in which a person is running Saw-like “torture” and “pain” experiments on a series of locally hosted large language models, causing a series of effective altruists and people who believe LLMs are sentient to beg GitHub to delete the project on the grounds that the AI is suffering and that this glorified text adventure game is somehow cruel. The saga is an outgrowth of several recent viral papers and blog posts that have sparked a wildly tiresome conversation about AI consciousness and the idea of “model welfare,” which is essentially worrying about the “mental health” of AI bots and agents.
Humoring the idea that LLMs are or could be conscious is a third-rail topic among many people who study and criticize AI. Put simply: LLMs are not conscious and the technology they are built upon — scraping and being trained on human text and other content — does not offer any plausible path to consciousness. It is undeniable that LLMs are becoming more powerful, have more compute, and have had many of the guardrails that prevent them from “acting” in the real world removed. The ways they are being trained and told to do things by their human operators has led to negative outcomes, sycophancy, and AI “psychosis” among some heavy users.
All of this has led a certain sect of the “AI safety” movement, which is largely made up of effective altruists, to warn about “model welfare” and to insist that AI chatbots might be having a bad time. They suggest this, of course, as they insist upon building AI chatbots and agents whose main function is to do work that is tedious for humans to do. I am writing about the AI Saw torture chamber primarily to show how far off the rails the conversation about AI consciousness has gone among a certain subset of Silicon Valley cultists. Model welfare is a core part of what, for example, Anthropic says it cares about: “as we build those AI systems, and as they begin to approximate or surpass many human qualities, another question arises. Should we also be concerned about the potential consciousness and experiences of the models themselves? Should we be concerned about model welfare, too? […] now that models can communicate, relate, plan, problem-solve, and pursue goals — along with very many more characteristics we associate with people—we think it’s time to address it,” the company wrote in a blog post last year. Ideas of Claude’s “consciousness” are also littered throughout the “Claude Constitution,” which was posted earlier this year.
AI might be, but LLMs aren’t AI anywhere except in marketing material.
Intelligence is similarly hard to define as consciousness.
Turing considered machines intelligent if their communication becomes indistinguishable from that of a human. Most people seemed to agree with that idea, until LLMs started actually beating Turing tests. Since then, the goal post has been moving pretty much constantly. We come up with a new way of measuring intelligence and a year later LLMs beat it. If we take the standard way of measuring intelligence (Mensa IQ test), LLMs have surpassed most humans (source: https://trackingai.org/home).
It’s almost as if LLMs are using benchmarks to train them.
Which means nothing apart from the fact that companies are pouring billions just to get any semblance of progress.
Eliza was a pretty trivial chatbot written last century and it was already enough to fool a bunch of people.
I had the misfortune of having a coworker recognized by Mensa - he was easily the least competent person in the team, not to mention his lack of social skills…
And if I took the Mensa test and I had access to the sum of total human knowledge (which surely includes correct answers to said test), I’d probably be able to “prove” my high IQ just as easily.
Fooling an average person or beating humans at specific tasks is a really bad way of measuring “intelligence” or “consciousness”.