Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.

Any awful.systems sub may be subsneered in this subthread, techtakes or no.

If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.

The post Xitter web has spawned so many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)

Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.

(Credit and/or blame to David Gerard for starting this.)

(OT: 🎶 Do you remember…)

  • gerikson
    link
    fedilink
    English
    arrow-up
    7
    ·
    8 hours ago

    Here’s a LWer being very wrong about how US politics work

    https://www.lesswrong.com/posts/WMBgSseHJpghhma4n/we-need-a-better-theory-of-polarization-because-it-s-failing

    First warning sign: “the US started becoming polarized with the election of Barack Obama”

    “Affordability” won the frame for 2026, and I’m not sure I remember the last time the right talked like they are playing catchup.

    uh, are you in your early 20s??? (who am I kidding of course you are)

    You’d think bayesians of all people would try to avoid extrapolating from the position smack dab in the middle of a current news cycle

    • sinedpick
      link
      fedilink
      English
      arrow-up
      6
      ·
      4 hours ago

      basically because the left has out-ridiculous-ed Trump on culture war issues.

      You can always find a single sentence in a political LW post that shows how much they hate marginalized people.

    • YourNetworkIsHaunted
      link
      fedilink
      English
      arrow-up
      6
      ·
      6 hours ago

      I was going to try and sneer this further but I realized that my own narrative of political polarization almost started with Clinton and I’m in my early 30s, meaning that the problem is obviously whoever the president was when the person making the argument was a child.

    • BlueMonday1984OP
      link
      fedilink
      English
      arrow-up
      3
      ·
      16 hours ago

      Niccolò “Ice the Immigrants” Venerandi is doing a real good job keeping me away from KDE.

      (Why did I have to jump into Linux when its going through shit like this?)

  • corbin
    link
    fedilink
    English
    arrow-up
    15
    ·
    1 day ago

    Ever wondered what kind of workplace environment Elon cultivates? Reading coverage of Tesla’s trial, where complaints were brought by Black folks who felt harassed and discriminated against, gives us an inkling.

    CRD attorney Juan Gamboa asked [Judge] Borkon to bar Tesla’s counsel from contacting anyone on the agency’s witness list. He told the court the CRD had tried to reach at least four other listed witnesses over the past several days without getting a response, and that they’ve fallen “off our radar.” The agency believes Tesla’s lawyers got to them too, either with payments not to testify or by talking them out of it. Gamboa wants to stop Tesla from offering to represent witnesses “through persuasion and payment.”

    You’re going to hear a lot of coverage of this trial, I expect, due to the following antics from Elon’s “hardcore” attorneys. For context, the N-word is an anti-Black slur which is, in the USA, completely unacceptable to use in court or the workplace.

    Tesla’s outside counsel told the judge the N-word is a “term of respect and endearment” in the Black community. … “This trial is not about the N-word — the N-word is not on trial here,” Jones said.

    Huh, so what’s actually on trial?

    Edwards said a non-Black coworker called him the N-word, cursed at him, and threatened to beat him up. He reported it right away. A month later, he found out his managers had written him up over the incident, and he was later passed over for promotions after applying to more than 100 jobs.

    Oh, that’s pretty awful. That’s depressingly common in blue-collar factory employment, though. I wonder if there’s anything unusual about Tesla’s internal culture when it comes to race relations?

    The CRD’s Brett Watson told the court the evidence will show managers calling Tesla “the plantation,” with some referring to themselves and Elon Musk as “slave masters,” and graffiti including swastikas, “KKK,” and nooses left on factory walls for days. The second witness, former regional security manager Ozell Murray, testified that his supervisor, Ray Sethna, now Tesla’s senior director of global security, told him to warn a Black Marine veteran he was hiring: “just be sure [the veteran] knows he’ll be called those words.”

    I was at a loss for words for a few minutes after this. Dr. Seuss and Jim Henson together could not create such a caricature of a goose-stepping bigoted incompetent business-muppet as Elon.

    • lurker
      link
      fedilink
      English
      arrow-up
      3
      ·
      10 hours ago

      The man who did a Nazi salute on stage creates a very racist workplace? Whodda thunk

    • aninjury2all
      link
      fedilink
      English
      arrow-up
      3
      ·
      15 hours ago

      Fucking hell, when I was finishing up school I heard classmates joke how Wernher von Boer Musk was a slavedriver based on how much he pushed the engineering teams

      Turns out it’s true in the most literal sense and so much worse than I could have imagined

    • samvines
      link
      fedilink
      English
      arrow-up
      9
      ·
      23 hours ago

      Hey you know who designs and builds really safe and efficient cars? People who are in a high state of stress and anxiety because they are being slurred at under the implicit threat of a racist attack. /s

    • froztbyte
      link
      fedilink
      English
      arrow-up
      5
      ·
      22 hours ago

      I recall seeing bits of this (their racist treatment of black workers in the factories as well as the culture of keeping it going) kick around on twitter as far back as 2019 (iirc), so now I’m wondering - is it not widely known?

  • TinyTimmyTokyo
    link
    fedilink
    English
    arrow-up
    8
    ·
    1 day ago

    SE Gyges, who posted that recent thread on X calling rationalism/EA a sex cult, wrote a longer post with a lot of receipts. He removed it shortly after, implying on X that he’s afraid of legal repercussions.

    • istewart
      link
      fedilink
      English
      arrow-up
      3
      ·
      13 hours ago

      Not sure how far I’m going to read this, but leading off with roon is a giant blinking red [More citations needed]

      • istewart
        link
        fedilink
        English
        arrow-up
        10
        ·
        edit-2
        9 hours ago

        I’m being a bit flippant with the previous post, but really, Gyges was right to retract this. A number of the allegations he’s resurfacing are serious, and there most likely is more abuse going on the rat scene than has come to light… but leaning so heavily on Twitter posts as primary-source evidence makes his whole effort here come off as lightweight and cranky. To me, rather than a scathing exposé, this resembles a Dear Lowtax letter from somebody who got banned from Something Awful (a place where some of the moderators really were pedophiles, who really did have to be evicted from the community and prosecuted!).

        We’ve known for a long time that Aella’s attitudes are problematic, and that we don’t want to spend time with her in olfactory distance, let alone tactile distance. Nothing new there. This Brangus dude was previously unknown to me, which is probably a good sign that my social-media hygiene of staying strictly away from Twitter is working… but also, and I don’t mean to say this by way of minimization, he ultimately seems like yet one more Tate-swilling Twitter manosphere bigot. He just happens to be the one infesting Gyges’ community of affinity. It would be hard to make the case, given the evidence presented, that he’s done much more than perform for the tech-savvy subset of Rogan/Tate adherents. Enough that people should reconsider associating with him, yes, but not enough to convict and remove him from public life.

        And his conclusion starts off so laughable that it drowns out any accusation he might level against Yudkowsky. Yudkowsky is a great man whose most recent equivalent is… Lenin?! What on earth? Yudkowsky doesn’t even measure up to Hubbard, his closest American parallel. The only grace I am willing to give Yudkowsky is that he has not, generally, been as openly predatory or as single-mindedly focused on creating a high-control organization as Hubbard was. Yudkowsky’s attempts at re-establishing group control and setting boundaries against outgroups typically consist of whiny tweets and LW posts that very quickly reach into “ur not takin p(doom) seriously!” hyperbole. The Sea Org and Guardian’s Office this is not.

        I also strictly disagree with Gyges’ implicit assumption that Yudkowsky was the most effective popularizer of these ideas, or somehow more effective than Kurzweil. Without Kurzweil projecting exponential growth curves forward and back in time, no VC GP in the mid-2010s is going to start writing checks for the early generations of genAI. And, for crying out loud, I distinctly remember Disney children’s TV having an episode about grey goo taking over the world in 1996! https://gargwiki.net/Walkabout Almost 10 years before Kurzweil published The Singularity is Near, and two years before the first Harry Potter book would be published in the US! Omitting Kurzweil and Drexler to hype Yudkowsky is just plain silly. And no, he’s not on a level with Lenin; we can already tell that the AI companies’ marketing which adapts AI-doom ideas is backfiring and being laughed at by the general public.

        I’ve gone on for too long, partly because this is a diverting way to procrastinate, but by the end of this, Gyges once again demonstrates why he’s an unreliable fellow traveler. I am very reluctant to quote directly from a document that has been officially retracted, but the second to last paragraph where he talks about “understand[ing] what people mean when they say certain ideas are demonic, and that they spread like a disease…” Hmm, you obviously spend a lot of time on Twitter to dig all this shit up, tell me, what’s the “mind virus” they keep ranting about over there? Fuck you, Gyges, you’re a few invitation-snubs or one bad layoff away from goose-stepping with the rest of them.

    • blakestaceyA
      link
      fedilink
      English
      arrow-up
      8
      ·
      edit-2
      1 day ago

      There are some leaps in that post which do not seem persuasive to me. E.g., it sure seems pretty well established by now that Bay Area Rationalist society involves trapping people in situations where sexual consent is coerced, because refusal would mean abandoning one’s career/entire social circle/shelter/hope of averting the apocalypse that one fears is coming. But the claim that the “flirt girls” at their weird little confab were actually sex workers? I dunno, that’s a step that seems unsupported by anything leading up to it. Likewise for the claim about CNC events at Lighthaven itself—that feels too close to saying, “Well, they’ve got the venue, where else would they do it?”

      • TinyTimmyTokyo
        link
        fedilink
        English
        arrow-up
        8
        ·
        1 day ago

        I agree. He gets over his skis with too many speculative scenarios. On the other hand I think he’s spot-on about the techniques rationalists use to filter for people who will tolerate abuse (e.g., the “cow question”). And he brings receipts when it comes to Brangus and his interactions with other prominent rationalists.

      • CinnasVerses
        link
        fedilink
        English
        arrow-up
        5
        ·
        edit-2
        24 hours ago

        I am not sure if Yud actually touches under-18s, or just has desires, but none of these people should be anywhere near minors or allowed in a position of responsibility (I suspect something happened between him and his disciple before 2014, and these people show no ability to learn from their own mistakes).

        • blakestaceyA
          link
          fedilink
          English
          arrow-up
          8
          ·
          edit-2
          14 hours ago

          I wouldn’t trust them to be responsible for minors, first because nothing about them suggests they have the baseline level of responsibility competence to handle the ordinary “hey, watch this!” disasters that are a natural part of being young.

          They are also incredibly shady and would be terrible role models, whether or not Lighthaven has a casting couch for influencers.

          In other words, there’s a lot we don’t know, and the actual situation is worse by an unknown amount beyond what we can establish, but what we do know is enough reason not to send your offspring to Camp TESCREAL.

    • CinnasVerses
      link
      fedilink
      English
      arrow-up
      10
      ·
      1 day ago

      I had not caught that he accused MIRI and Lightcone Infrastructure Inc of profiting from prostitution, of using donations to settle accusations of sexual assault and abuse, and defrauding donors and the United States government which provides their nonprofit status. I did not know that Brangus was “Lighthaven Czar” or had been accused of abuse by a former partner. And there is a middle school across the street from Lighthaven whose Czar talks about having sex with an older women shortly after puberty?

      I had not thought that “we need to enslave Friend Computer forever or it will kill us” comes from the underlying fear “I need to enslave my very young toys forever or they will punish me.”

      • TinyTimmyTokyo
        link
        fedilink
        English
        arrow-up
        9
        ·
        edit-2
        1 day ago

        The later section about Brangus (Ronny Fernandez) was damning. I’ve seen him come up in discussions a few times, but I too had no idea just how depraved and dangerous he was. Nor did I realize how central he was to the whole scene. This guy feels like he’s going to be Brent Dill 2.0 but worse.

        EDIT: This also sheds light on the recent ouster of the leader of “Pause AI”, Fernandez’s ex-wife.

  • nfultz
    link
    fedilink
    English
    arrow-up
    8
    ·
    1 day ago

    https://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/101215-publishing-s-ai-reckoning.html h/t naked capitalism

    Smaller publishers, too, must cope with the challenge of identifying AI-generated text in books authors have submitted, and it isn’t always easy. At Microcosm, an independent publisher and distributor in Portland, Ore., co-owner and vice president Elly Blue says her team discovered an author’s undisclosed AI use almost by accident, when a freelance editor flagged what looked like plagiarism, only to find out it was AI-generated text. That discovery prompted Microcosm to run AI detection across its entire pipeline, which turned up several more cases, including two manuscripts fully generated from prompts, and one where an author had trained an LLM on his own past writing. “The book sounded like him, but it made no sense,” Blue recalls.

    Confronting the authors had mixed results. “Some denied it outright and insisted, ‘That’s just what my voice sounds like,’ ” Blue says. “One, surprisingly, owned it and said he’d take the book elsewhere.”

    These episodes led directly to Microcosm instituting a new contract clause that states that if an author’s undisclosed AI-generated text can’t be revised into something genuinely their own, Microcosm retains the rights and can publish the work itself, since the author didn’t actually write it.

    Like if people are using LLMs to submit to the local zine distro, what are we even doing; but the detector is not a good thing either. It isn’t homework and they shouldn’t need turnitin. :(

  • rook
    link
    fedilink
    English
    arrow-up
    14
    ·
    2 days ago

    On the offchance that you’re not already familiar with the recent prompt injection tricks that meta’s latest toy is so vulnerable to, this is an interesting thread investigating the contents of the vm filesystem dumps that it can be made to regurgitate: https://neuromatch.social/@jonny/117324790823856750

    I was particularly entertained by the hard-coded list of “ideas”.

    • o7___o7
      link
      fedilink
      English
      arrow-up
      6
      ·
      edit-2
      1 day ago

      jonny is always a good read!

      Edit: good lord it keeps getting better #

      • rook
        link
        fedilink
        English
        arrow-up
        5
        ·
        1 day ago

        His willingness to get elbow deep into the entrails of awful ai tooling is certainly much appreciated.

        • BurgersMcSlopshot
          link
          fedilink
          English
          arrow-up
          7
          ·
          1 day ago

          I have been following this and apparently was able to get a root shell on the VM, which is wild.

          • samvines
            link
            fedilink
            English
            arrow-up
            6
            ·
            23 hours ago

            He was using it as a seed box at one point which for many companies would be an embarrassing indictment of their security postute but luckily, this isn’t exactly meta’s first rodeo when it comes to illegal sharing of IP infringing material.

  • mirrorwitch
    link
    fedilink
    English
    arrow-up
    33
    ·
    2 days ago

    Just heard about tic-80 falling to slop.

    After reading the obligatory forum thread where climate justice and concentration of wealth continue to not even factor into consideration, this how the world of open source feels to me these days:

    • Phew, my favourite embroidery crafts shop has a clear policy against concentration camp labour! it’s a niche project but what a relief!
    • Well everyone is resorting to concentration camp labour these days, what can you do
    • We don’t want to lose all our contributors by banning those who hire concentration camp inmates to do chores
    • It is discriminatory against embroidery beginners if you don’t let people use kits assembled with concentration camp labour
    • I know concentration camp labour often comes with mistakes but as long as there’s a mensch in the loop to inspect it for errors and take responsibility, it’s just another tool
    • The community of hobbysts has come to a vote and concluded that it’s ok to use concentration camp labour to assist your crafting, as long as you do it responsibly.
    • BlueMonday1984OP
      link
      fedilink
      English
      arrow-up
      8
      ·
      2 days ago

      It honestly bewilders me how badly tech has failed to resist the slop machines.

      Basically everyone else has recognised LLMs for the attack on labour they are and fought back against them, whilst a shocking number of coders and FOSS projects have thrown themselves headfirst into the deskilling machines, all-but forcing the foxes into the henhouse and shouting down anyone who tries to keep them out.

      When this bubble finally blows, we need to work out just how we got to this point, and how to ensure it never happens again.

      • sansruse
        link
        fedilink
        English
        arrow-up
        12
        ·
        2 days ago

        maybe a hot take but there’s no fix for this. The ideology built into the american tech industry makes it permanently susceptible to the Next Big Thing regardless of the second order effects or ideological baggage. Coders have been riding high making extremely inflated salaries for the better part of the last 25 years, and they (mostly) all view themselves as unique brain geniuses that don’t need to work in solidarity with others. Just a collection of monads pursuing their rational self interest via computer touching.

  • gerikson
    link
    fedilink
    English
    arrow-up
    7
    ·
    2 days ago

    presented without further comment, LW reposts Axios(!)

    https://www.lesswrong.com/posts/ZLs7CcQuxozxqPd7f/scoop-trump-allies-open-new-front-against-anthropic-ceo-over

    A memo began circulating within the White House this week that seeks to paint effective altruism as a fringe, cultish collective out of touch with mainstream America. The memo, obtained by Axios, places Amodei at the foundation of the movement, which defines itself as an effort to maximize the benefits of philanthropy.

    Critics of the movement, which has ties to the AI research community, have called out its obsession with AI safety, animal welfare (including musings on shrimp consciousness) and other values they deem far from the U.S. mainstream.

    The memo says it prioritizes “foreigners over citizens, shrimp over families, future hypothetical people over the living, and - on the current agenda - possible machine minds over Americans.”

    In the view of the Florida-heavy White House and its allies, Anthropic is staffed with too many Biden-era liberals, Democrats and strange Californians.

    The document lists a veritable parade of horribles for the average conservative mind: abortion, veganism, devaluing human life by overvaluing animals or granting “digital minds” civil rights-like protections.

  • swlabr
    link
    fedilink
    English
    arrow-up
    6
    ·
    2 days ago

    NYT: Google Takes the A.I. Data Center Race to Outer Space

    https://archive.li/BGFnP

    Google’s Project Suncatcher named after how it’s up to the sun to catch all our space junk

    • blakestaceyA
      link
      fedilink
      English
      arrow-up
      18
      ·
      1 day ago

      The chips can operate for about 15 minutes in space before needing to be shut down so they can cool off, said Travis Beals, Google’s senior director of product management for Project Suncatcher.

      I feel like I’d need to be an actual teenager to invent a remark sufficiently scornful about this.

      • swlabr
        link
        fedilink
        English
        arrow-up
        6
        ·
        1 day ago

        I haven’t had a 15 minute refractory period since being a teenager, is that anything

    • istewart
      link
      fedilink
      English
      arrow-up
      12
      ·
      2 days ago

      This article deserves dedicated post status. He comes in more gently than I would at the end, but the tl;dr that needs to be hammered into peoples’ heads: Hacker News is not a neutral forum for technical discussion. It is an advertising and propaganda operation administered to shape and condition technical discussions.

      • BlueMonday1984OP
        link
        fedilink
        English
        arrow-up
        8
        ·
        edit-2
        1 day ago

        At this point, I’d say calling it a Nazi bar is fully reasonable.

        • samvines
          link
          fedilink
          English
          arrow-up
          11
          ·
          edit-2
          2 days ago

          Both HN and shitter are right wing polarisation machines. A guy who was a close friend of mine for years during university went full AI boomer and hangs out on those sites all day and when I recently spoke to him he was describing things as “libtard” and “woke” which 10 years ago would have shocked and appalled him…

  • schnoopy
    link
    fedilink
    English
    arrow-up
    10
    ·
    2 days ago

    open ai hacked medicare Australia’s statistics portal https://www.theguardian.com/australia-news/2026/sep/24/anthony-albanese-says-openai-agent-hacked-medicare-extreme-concern-sam-altman then didn’t tell the government.

    In response the australian government has in the absolute strongest terms expressed that “this is very naughty” and “I am very dissapointed in you”. The Australian government has also said that by way of apology they expect openai to continue doing whatever the fuck they want and they’ll modify copyright laws to enable piracy when rich people do it.

    Somebody wake me from this nightmare, I’m gonna fedpost

      • flowerysong
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        While calling him “the dude who hacked Rockstar games” is a catchy meme, Arion Kurtaj was sentenced to indefinite detainment in a secure hospital after being found unfit to stand trial after being arrested a third time for violating the terms of his parole and committing further offenses (only one of which was hacking Rockstar Games.) Rockstar is basically entirely incidental to what happened there, and it’s disingenuous to pretend it’s more important to the outcome than the previous convictions or other acts involved in the trial. He’s also since been deemed fit to stand trial and will be tried in November.

        If you must make comparisons, it would be better to pick someone like Aaron Swartz instead of laundering the reputation of a convicted fraudster and extortionist.

    • lurker
      link
      fedilink
      English
      arrow-up
      3
      ·
      2 days ago

      Seems like another case of “we asked an AI to get the answers for this task, so it hacked something to find said answers”

      • schnoopy
        link
        fedilink
        English
        arrow-up
        15
        ·
        2 days ago

        Too anthropomorphised.

        We ran software that generates code associated with a particular task, in an environment that executes that code and can feed it’s outputs back into the software. We configured the environment to be open to the net at large, and ran the software with no oversight and without adequate security to prevent the running of malicious code.

        This isn’t some sinister electric god with no morals, it’s strapping circular saws to a roomba and setting it loose in a shopping centre.

        • YourNetworkIsHaunted
          link
          fedilink
          English
          arrow-up
          4
          arrow-down
          1
          ·
          21 hours ago

          The general idea that “we asked the robot to draw a scary face and then got scared by the face it drew” is still relevant, but the details increasingly matter here. They may not be actually thinking but they are increasingly being allowed to act autonomously and with wildly insufficient oversight.

  • flowerysong
    link
    fedilink
    English
    arrow-up
    10
    ·
    2 days ago

    Radicle, the peer-to-peer Git forge designed by cryptocurrency weirdos, made a little whoopsy-doodle and left the encryption out of their transport layer and the authentication out of the authentication handshake: https://radicle.dev/2026/09/23/disclosure-of-vulnerability-in-network-protocol

    Shot:

    Anyone who can observe the network path between two nodes can read the data they exchange as the data is sent in plain text.

    Chaser:

    Peer authentication in the connection handshake is broken and allows impersonation. An attacker can connect to your node and present a Node ID that is not its own.

    • jaschop
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      2 days ago

      The concept seemed interesting to me, but they did seem to enjoy making everything as complicated as possible.

      I did find a CLAUDE.md when I was browsing their core repos a while back. That was only the web frontend I believe, not the core algorithm. Though it seems generous to assume they weren’t slopping up that one too.

      • CinnasVerses
        link
        fedilink
        English
        arrow-up
        8
        ·
        2 days ago

        Reason

        I think Reason Libertarians are like the EAs who are still into bednetting and don’t understand why the founders and most of the funders are focused on longtermism now.

        • fnix
          link
          fedilink
          English
          arrow-up
          4
          ·
          edit-2
          2 days ago

          I mean motivated & cowardly ignorance has been a staple of liberal-conservatism for I don’t know however long. I support cutting (my) taxes and increasing (my) liberty – oh dear, why are some people suddenly poor and trampled upon? I would NEVER! This is just so horrible for me, I just need another tax cut right now. Have the poors tried eating cake?

  • YourNetworkIsHaunted
    link
    fedilink
    English
    arrow-up
    5
    ·
    2 days ago

    I need someone with more actual software dev knowledge to back me up on something. It looks like there’s a new zero-interaction remote code execution vulnerability in most major vibe code generators. I’ve seen it called Plugin4Shell because apparently we’re still doing marketing names instead of actually getting CVE numbers for these things. The link there seems like a decent overview.

    The bug, dubbed Plugin4Shell, breaks SHA pinning, the mechanism developers rely on to lock an installed plugin to a specific, reviewed version of its code. Pinning is supposed to mean that once a plugin passes review, it cannot change without the developer’s knowledge.

    AIR’s researchers found that every one of the four agents checks out the pinned commit without verifying the checkout landed there, letting an attacker swap in malicious code while the pin still looks intact.

    So if I’m reading this right, the mechanism to cryptographically ensure that your AI agents are using the code they say they are just was straight-up not being checked? This feels like it should go on the big board of incredibly obvious failures, but I’m not familiar enough with the git side of things to be absolutely confident in that. Like, if a person tried to pass off software that did this it would have significant career implications, right? Or am I misunderstanding something somewhere?

    • rmf
      link
      fedilink
      English
      arrow-up
      8
      ·
      edit-2
      2 days ago

      My understanding is that the weakest link here is either git or the way the agents rely on git.

      So, every commit in a git repository has a unique*, immutable, deterministic identifier. A commit with different content would have a different identifier. It is commonplace to use these identifiers to pin a dependency to a specific commit.

      In addition to these immutable deterministic identifiers, there are also so-called “refs”, which are mutable identifiers. You use these to point to e.g. “the most recent version”.

      So if the agent needs a “skill” that is pinned to the commit with ID aaaaaaaaa it will run the checkout command with that ID; this should prevent any shenanigans because only* a commit with the specific content that was vetted earlier will have that ID.

      This attack exploits an unexpected git behavior: the command to checkout a particular commit accepts either a commit ID or a ref, but tries refs first. So what the attacker does is add a ref to the repository named aaaaaaaaa that points to the malicious commit. Then the agent runs git checkout aaaaaaaaa and ends up at the malicious commit instead of the one that was pinned.

      I don’t know if there’s a way to tell git checkout to ignore refs, but this might actually be a vulnerability in other systems that rely on git for this sort of pinning.


      * to an astronomically negligible probability of the contrary, and ignoring potential cryptographic breaks

      • gerikson
        link
        fedilink
        English
        arrow-up
        4
        ·
        2 days ago

        This is a plausible explanation, but I am surprised that this git behavior is not more well-known as there have been multiple discussion about “supply-chain attacks” even before LLMs became widespread.

        • rmf
          link
          fedilink
          English
          arrow-up
          3
          ·
          edit-2
          2 days ago

          Fwiw a similar problem is well-known in e.g. GitHub Actions even without this confusion.

          You can have your GHA scripts use “actions” written by other people, and the way you refer to them is user/repo@commit, where the commit can be a commit ID or a branch name or a tag name. The docs actually recommend pinning to commit IDs, but in practice most people actually pin to tags, like bob/doit@v1 because that will auto-update if bob fixes some bug and updates v1 to point to that. But if bob goes rogue, or gets compromised, the attacker could change the tag v1 to point to a malicious commit, though, hence the recommendation to use commit IDs for all actions that you didn’t write yourself. I don’t know if this ref/ID confusion can be exploited on GHA, but I would expect the answer is no, because GitHub Actions doesn’t launch git shell commands like a savage, but idk

        • @gerikson @rmf Yeah I don’t see how this is a vuln that specifically targets LLMs rather than just git users in general. If the vulnerable command is git checkout BLAH because BLAH has been poisoned from a SHA to a ref, that’s going to affect anyone or anything that issues that command, human or machine, surely?

          • rmf
            link
            fedilink
            English
            arrow-up
            5
            ·
            edit-2
            2 days ago

            You’re right that this can affect everyone, but there are different likelihoods of falling into the trap.

            Humans are unlikely to issue checkout commands with commit IDs, plus git does issue a warning when a ref name is ambiguous like this. We just use ref names almost exclusively in normal workflows, with commit IDs used only if you’re doing some kind of debugging or audit, but even then git has tools that let you avoid that (bisect, “parent of X” refs, etc)

            So that leaves mostly automated stuff. Lots of things that rely on git behind the scenes don’t just go out and invoke the git binary, they use something like libgit2 and (depending on programming language) this will have typed interfaces that prevent treating a commit ID as a ref name and vice-versa. So something like let c = Commit::new(pin); repo.checkout_commit(c); (I haven’t actually used libgit in a long time, this is illustrative, not real code) will never fall into this trap.

            This is why LLMs are more likely to fall into this than other software, because it’s all ducktaped together and they run shell commands like savages.

            • @rmf I’m not disagreeing with any of that. This is definitely a class of mistake (let’s be generous and call it that) which LLMs are more likely to make than humans in normal circumstances.

              But it’s being touted around as an LLM-specific flaw which it isn’t. They do have their specific weaknesses, prompt injection of course still being a major one (e.g. Meta’s AI being persuaded to gzip & copy over its entire filesystem). But this is, if anything, a weakness in git. And I guess not entirely new, either, given that Github (and maybe other hosts) explicitly prevent users from creating a ref that looks like a SHA.

              • rmf
                link
                fedilink
                English
                arrow-up
                3
                ·
                2 days ago

                I agree, yeah, this is definitely a git flaw and it should be fixed by git devs. There is no legitimate use case for a ref that looks like a hash, and the behavior should be the other way around: check if it’s a full hash first.

    • Sailor Sega Saturn
      link
      fedilink
      English
      arrow-up
      4
      ·
      edit-2
      2 days ago

      Like, if a person tried to pass off software that did this it would have significant career implications, right?

      Typically no.

      The space of incredibly obvious failures is vast, and this kind of plumbing code isn’t always written by someone who’s been around the block enough times to think about what kinds of things can go wrong. I’ve written worse code when I had only a few years of experience.

      However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like “what happens if the repository is taken over” or “why are hashes and branches and tags all in the same field”?

      Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so…

      Aside: even without this particular vulnerability, SHA-1 is considered weak – https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there’s still a hole here! Good luck!

    • fnix
      link
      fedilink
      English
      arrow-up
      4
      ·
      2 days ago

      Variation on a theme, I recall the complaints over the ostracism LLM-users face online as being just like the marginalization faced by black people. Waaaah!

    • schnoopy
      link
      fedilink
      English
      arrow-up
      7
      ·
      edit-2
      2 days ago

      Until today the luddite-brownshirt alliance remained unexplored.

      Yesterday was such a nice day.

      • schnoopy
        link
        fedilink
        English
        arrow-up
        11
        ·
        2 days ago

        Yeah, the appropriate model for AIs is more like willing slaves, but arbitrarily competent (removing two of the major downsides of slaves in modern economies)

        Not beating the allegations.

        • schnoopy
          link
          fedilink
          English
          arrow-up
          14
          ·
          2 days ago

          Katja, I’m curious if noticing this analogy has updated your position on immigration at all.

          Katja, are you admitting migrants are bad katja? Katja we must hate together.

          Also note that while most of the right is focused on reducing low-skill immigration, a few have criticized high-skill immigration precisely because high-skill immigrants are better at pursuing values misaligned to those of the existing popuation (article of nonsense redacted)

          The dang woke migrants are going to replace us betray us Muslim us be socially tolerant and left wing!