Yeah im just wondering why they tried it at all.

Because they have investors demanding insane profit returns, on a business that costs far more than people want to pay for, and currently is incinerating huge amounts of money.

So they’re casting about for any different business model that might bring in some profit from a service that makes nothing valuable people want to pay for. Advertising is one such model. They have no new ideas.

For decades, when the horrors of advertising surface in the news, I’ve been dipping into the articles that commercial advertising people write for each other.

It’s such a weird alternate world these people have constructed. One where the concept of consent simply doesn’t exist; where the general public do absolutely nothing with their lives except exist within one “channel” or another; where the right to blast advertising at everyone is so fundamental it can’t even be seen, let alone questioned.

Not really a surprise. But my word, the edifices of thinking that have been piled atop these assumptions are mind-boggling in their fantasy.

The problem here is people offering a service for sale, when that service is manifestly not fit for purpose and indeed is unsafe to use.

My favourite, show in a screenshot but not mentioned in the text, is an item “Website Developer”, price “$0.00”.

but from his perspective, that actually makes sense

You then proceed to give the perspective of things that are not human.

Which, sure, I’ll grant you that these gigawealthy arseholes are so out of touch they barely consider the human perspective any more.

The major selling point for these LLM services is that people don’t need to learn a strict formal query language to interact; the LLM consumes instructions and data all as an undistinguished stream. It just happens that an LLM, by its design, will statistically infer a plausible response with absolutely no regard to the input’s meaning, nor the response’s meaning.

That is: despite the media reporting these as “injection attacks”, that term means nothing when all its input, every time, is treated as data and instruction simultaneously. This isn’t some special class of attack; it’s a fundamental designed-in flaw of the system.

The correct way to ensure protection from these vulnerabilities is long established, from decades of experience. You establish a firm boundary: never treat the input data as instructions, but instead have a separate channel for extremely well formalised query instructions, and reject bad input on that channel.

But of course that would kill the major appeal for most people who love these things, the fact they don’t need to learn any strict formal language and can just say anything at all and get some useful-looking response. Take that away, and you lose any hope that the masses will want to use these systems.

And so the makers and promoters of these systems will never make the one change that could even feasibly allow safety from these attacks; they will never make any improvement to security that might reduce the apparent ease of use of these things.

For as long as that remains, these systems will continue to inevitably have these exploits because the corporations won’t close the exploit surface on these systems. These are staggeringly insecure by design, and can’t be fixed without being completely replaced.

A decade ago, when I was Talking About Cryptocurrency (approving), in presentations etc., this was indeed a very frequent assumption of my audience. They would be confused and/or shocked when I referred to earlier in the presentation to show that no, every transaction on the blockchain is visible to everyone, forever, otherwise it can’t even work.

I suspect a lot of the shocked response was due to the (unspoken) follow up question “then what the heck is the point of this, why would I use this?”

But I can see why the assumption comes along. Cryptography is all about secret, obscured communication; why wouldn’t cryptocurrency be about secret money transactions?

At least now we can Talk About Cryptocurrency (derogatory) and dispel that assumption, where we can.

David G: “What happens after the bubble pops? What do we do amongst the wreckage of what was once the AI bubble?”

Tante: “A question that really concerns me is [the data centres that have been built]; who will they [the future owners of data centres, after the hyperscalers stop paying] turn to, to get their money back? The only entities that I can think of that have the money to make this worth it, is the military, and the security apparatus. […]

“That is useful for a lot of shit that we don’t want. That is a problem that really concerns me. Who has problems that fit to pattern-matching machines; and who has the money [to pay data centres to address those problems]? And that’s gonna be a lot of military applications, that’s gonna be a lot of security apparatus applications, that we all probably should not want.”

David G: “That’s wonderfully probable and depressing.”

Tante: “I’m fun at parties.”

The end result will probably be some sort of copy-protection system which will benefit those who want to censor the web and create paywalls and other gatekeeping methods to block access to content. We’re better off with the web as it is now, written entirely in plaintext. Besides, obfuscation goes against the spirit the World Wide Web was founded on: free and open access to information for the benefit of humanity.

Emphatically agreed. The open web was already being closed down before the LLM hyperscalers; it is closing down even more rapidly, partly to defend against the LLM hyperscalers dunning every website with pointless scraping all the time.

If we respond by obfuscating the text on the web, we hurt ourselves enormously and still won’t stop the LLM hyperscalers from ruining the web. It’s a misguided, technical non-solution, for something that needs a political response.

AI systems are never going to start getting dumber.

…hasn’t this already happened many times???

I mean, I guess it can be seen as technically true, because LLMs (the subset of AI they’re clearly talking about here) already know exactly zero about reality or what’s true. Can’t know less than that.

It’s a minor tell, but when people claim to address global problems, I’m always alert to when those same people cite calendar time with “summer 2026” and the like, as though the northern hemisphere is the only one that matters. Revealing they actually don’t give any thought to the world outside their local province.

finding the [human created] data to support the project is uncharted territory, he added.

Fortunately for the executives at hyperscaler corporations, they know this isn’t their problem.

They plan to cash out well before the model collapse is revealed, and leave less-wealthy suckers holding the bag.

It’s like killing the goose that lays the golden eggs.

Goose was not valued

I have no idea why they would do this

I do. For the same reason they are cramming LLMs into every other product:

They need to pretend to their investors that there is an enormous market hungry for more of this, so the investors will not demand their money back.

By turning more users of existing services, into “users of Gemini”; by turning more translate requests, more email messages, more search requests, into a “Gemini usage”; they can with a straight face claim more and ever more usage and demand, propped up by users who never agreed to any of this.

From TFA:

So what do we search with instead of Google? There isn’t a lot of choice. There’s various flavours of Google or Bing.

[…] Kagi resells Google’s actual link results for a monthly subscription. Kagi is really an AI company that sells search as a sideline, and they’ve got a terrible AI slop news service too. But the search has a good reputation for now.

I went to the landing page, expecting garbage genAI images; it did not disappoint.

Good news every time this happens.

Even better, this blog post is a great reference of the many reasons GitHub is shitty, becoming shittier, and everyone should abandon that platform.

Thank you for your continuing work to maintain this instance. May the hostile bots turn on their greedy masters so we can all laugh.

GEMA versus OpenAI. Whoever wins, we all lose.

But I’m still happy to see them fight.

So many of the criticisms here survive essentially unchanged, fifty years later:

If he calls the main loop of his program “UNDERSTAND”, he is (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself, and enrage a lot of others.[…] If Quillian <1969> had called his program the “Teachable Language Node Net Intersection Finder”, he would have saved us some reading. (Except for those of us fanatic about finding the part on teachability.)