Took a peek at what our good friends the rationalists are talking about. That’s gonna teach me not to gaze into the abyss.

Apparently some AI related bigshot CEO is telling Stuart Russell that our best hope for survival is an “AI Chernobyl” which would royally piss everyone off even more.

Our “leaders” are quaking in their boots and hoping a bunch of us would die so we the people take the reins and lead them somehow out of a peril of their own construction. That’s whole another level of completely fucked up.

You all know I’m the last person to expect this shit to get 3 vegetables across the river without going back, and yet the latest evolution of their bullshit is making me feel rather doomer-y.

Reading between the lines, Chornobyl = chain reaction = forkbomb = agents hacking their own company to launch more agents with the same prompt. Do I think this is possible? Sure as shit I think its possible. their cybersecurity is terrible. And LLMs did turn out great at finding exploits: humans screw up in stereotypical ways and there’s a gazillion “fixed blah blah” git commits to train on.

Okay, so we got a tremendous cluster fuck. Ideally all that happens is another “human coding appreciation day”, whatever.

Except as can be told from their recent hacking leaks, what happens when a clusterfuck like this runs, with the context full of hacking, it would keep trying to hack whatever, until stopped. Hopefully, “whatever” is other AI companies, but it’s not like they can’t train or fine tune to diversify the whatever.

Other bit that is very disturbing to me is that they are talking about a billion deaths scenario, instead of talking about outright extinction. Why the sudden modesty?

My cynical take is at the core their concerns are still insincere, but this time they actually think something serious might happen, not just having fun with fantasies of infinite importance. And so they are inventing new scenarios that would make that serious thing seem like we got off lightly, or perhaps turn it into a regulatory capture opportunity.

Which is just an escalation, or perhaps de-escalation, of their old game of talking over any realistic concerns with “existential risk”.