Well that sent me on a dark rabbit hole…

(For those that didn’t know, The author of Dumbledore’s Army and the Year of Darkness is a scam artist and a cult leader. I knew that part already, but what I didn’t knowwas how big the cult was at it’s peak or that it was tied to a murder-suicide in 2011…)

https://fanlore.org/wiki/Dumbledore's_Army_and_the_Year_of_Darkness

And of course, sneerclubbers should know by now about the HPMOR to Lesswrong to Cult pipeline. (With MAPLE, Zizians and Leverage Research as most clearly cult examples. Dragon Army and Vassar’s circle as cult-adjacent).

Well, OpenAI and Anthropic have repeatedly degraded their service through unacknowledged quantization and/or dynamically swapping models on the user. /r/singularity has gotten sick of it to the point it is a meme. It is because they can’t actually afford to offer quality SOTA models at the price point people will actually pay for them.

The percentage isn’t total progress to some key point of AI 2027, it is progress relative to their timeline. A constant 1.0 would be staying on track with their predictions, numbers less than that would be falling behind. So they are admitting the real numbers are falling behind their predictions more and more (while still not acknowledging their entire timelines was bs in the first place).

Ed Zitron has also explained his suspicion that lots of GPUs are sitting around in warehouses waiting to be installed, in some cases sold (to juice NVIDIA’s revenue) but not even shipped yet.

And I’m really skeptical speedup from AIs claimed by the LLM companies is in anyway related to reality.

That wouldn’t make sense, because 1.07 would mean progress is negative. I think they meant it as a multiplier? So 1.07 is almost exactly what the predicted, .17 is only 17% of what they expected.

Anyway, it doesn’t really matter, because so much of the input numbers to these calcs are garbage.

That is even worse than using the ARRs, which Ed Zitron has extensively explained are wildly distorted numbers done by shuffling sales and costs around to get really good 30-day to 1-month stretches.

Bioman has already pointed out the “Economic Value” numbers they are using for these tables are probably bullshit (based on self reported run-rate extrapolations that are deliberate distortions at best, based on VC valuation at worst). To add to this… the compute values are also probably bullshit. They are likely based on data center announcements and not confirmed totally complete data centers (Ed Zitron has ripped into how much bs there is in data center announcements). “Coding Time Horizon” is probably METR, which, while some of the best numbers for estimating actual AI improvement for practical purposes, are still really bad in several key ways. (They don’t have enough human task performers for the longer duration tasks even if everything else was right, because they aren’t, and there are several ways systematic bias could have leaked in and compelted distorted the constructed measure of task duration.)

“AI Software R&D Uplift” is the single most important category to their scenario of recursive self improvement… and they have it at a small fraction of what they estimated.

I’ve had the same thought about Factory Porn, it’s is basically what Image slop generators are, right down to the addictive nature, even when it isn’t porn being generated.

Lol, I assume he was just throwing shit at the wall and seeing what would stick?

Skimming the linked nature article… is my understanding correct that you can basically disable the watermarking by turning temperature down to 0?

For example, if the LLM distribution is very low entropy, meaning it almost always returns the exact same response to the given prompt, then Tournament sampling cannot choose tokens that score more highly under the g functions.

Other highlights from the linked paper… to get a true positive rate of 90% with a false positive rate of 1% you need 400 tokens (which should be a few paragraphs worth of text)? (If I’m reading figure 3 right?) That actually isn’t that much, relative to the lengths of essays people write for high school and college classes. …well actually… 1% false positive doesn’t sound too bad, but if you have thousands of freshmen students all taking classes involving writing essays and checking for watermarks becomes the norm, that is dozens and dozens of false positive, which means lots of false accusations, and as we’ve seen from how teachers and institutions have tried utilizing the existing “AI detection” tools that are much much less reliable… I’m getting angry just thinking about it.

Edit: on turning temperature down, it should be noted Anthropic and OpenAI have been increasingly denying the end user internals of their models, such as summarizing or even outright hiding the thinking traces, and not allowing them access to temperature settings either.

This view on the community also explains his recklessness not only with aggressively monetizing HPMOR, but also his willingness to risk drawing JK Rowlings attention to his fanfiction (towards the end of HPMOR, one of his author’s notes was looking for someone to reach out to her).

I would try to get in touch with J. K. Rowling to see if HPMOR could be published in book form, maybe as HJPEV and the Methods of Rationality, with all profits accruing to a UK charity. I’m not getting my hopes up, but I do have a rule telling me to try rather than automatically giving up and assuming something can’t be done. If any reader thinks they can put me in touch with J. K. Rowling, or for that matter Daniel Radcliffe, regarding this matter, I do hereby ask them to contact me at yudkowsky@gmail.com.

Rowling (and Warner Brothers) have been litigious with their IP: https://en.wikipedia.org/wiki/Legal_disputes_over_the_Harry_Potter_series#Other_accusations_of_infringement . So this was a reckless move on EY’s part, possibly indicating that he doesn’t appreciate the delicate place fanfic (and its communities) are in.

Edit: thinking about the way Rowling has shown her true face over the past decade… “with all profits accruing to a UK charity” makes me wonder if in whatever fantasy universe Eliezer did talk Rowling into allowing published fanfic to be monetized for “charities”, that money would all end up going to TERF “charities”.

One of the plot threads was about the invention of AI, in the Data-from-Star-Trek kind of sense.

Going on a tangent from this… Its funny how much AI-related sci-fi is feeling increasing quaint (and often foolishly optimistic).

You know what depiction of AI has held up, and maybe makes even more sense in light of modern AI? Star Wars’ take on Droid. Prior to LLMs it seems obvious that droids were sentient and deserving of rights. Post LLMs… it seems pretty plausible you could get something like C3PO that blabbers on but lacks any meaningful sentience or sapience. Also, it seemed obviously idiotic the way the Trade Federation had humanoid droids acting as pilots and gunners instead of building proper autopilots and targeting system. But after seeing LLMs get shoved everywhere, I totally could imagine a megacorp shoving droids built from standardized pretrained modules into all kinds of applications they aren’t fit for.

Stealing technical terms from other fields to make their fantasies feel more real.

Well you see, if they used a real technical term for an existing genetics technique or technology, their fantasy might be limited by the limits of reality, so instead they user complicated jargon terms to signal they are talking about fantasy “near future” techniques and technologies.

also nice save defining “AGI == humanity” => goal reached

A common game along these lines I’ve seen: define the singularity as knowledge-work inventions allowing for more inventions. Thus we’ve been in the singularity since the invention of the printing press writing language.

Lesswronger argues skynet might still be a decade away… because they’d rather have the eugenics wars instead: https://www.lesswrong.com/posts/iQzxxgJXXaAQjq7Jz/faq-isn-t-agi-coming-too-soon-for-reprogenetics-to-help

They noticed no one was starting a criminal investigation against OpenAI and figured it was safe to brag about.

Maybe Anthropic and Meta started double checking their shitty vibe-coded “sandboxes” in response to OpenAI story? And their “sandboxes” were shit enough their chat-bots managed to slip out just slopping their way through and poking at the obvious things.

I’m looking forward to Elon’s desperate attempts to make Grok have a hacking incident so he can keep up!

a discussion on whether agency and self-determination are integral to the rise of disembodied consciousness might be worth having

The lesswrong crowd is absolutely the wrong people to have that conversation. Reminder that Eliezer thinks qualia are a result of certain level of recursion and self-reference in cognitive processes (basically Hofstadter’s ideas like GEB/I Am A Strange Loop, reserved and reheated with a heaping side of overconfidence), thus he doesn’t think chickens or even babies have any qualia but thinks LLM ChatBots might! Also, I think Eliezer owes some philosophers an apology for making fun of the entire concept of p-zombies, I think LLMs are pretty close to the concept, enough to show the value in the thought experiment at least.

Having weird opinions on stuff philosophers are very unsure on isn’t that bad itself, its the fact that Eliezer thinks his opinions are much more objective than they are and that people are dumb for not agreeing with him and that he has nothing to learn from mainstream philosophy.