Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned so many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
(OT: 🎶 Do you remember…)
I need someone with more actual software dev knowledge to back me up on something. It looks like there’s a new zero-interaction remote code execution vulnerability in most major vibe code generators. I’ve seen it called Plugin4Shell because apparently we’re still doing marketing names instead of actually getting CVE numbers for these things. The link there seems like a decent overview.
So if I’m reading this right, the mechanism to cryptographically ensure that your AI agents are using the code they say they are just was straight-up not being checked? This feels like it should go on the big board of incredibly obvious failures, but I’m not familiar enough with the git side of things to be absolutely confident in that. Like, if a person tried to pass off software that did this it would have significant career implications, right? Or am I misunderstanding something somewhere?
My understanding is that the weakest link here is either git or the way the agents rely on git.
So, every commit in a git repository has a unique*, immutable, deterministic identifier. A commit with different content would have a different identifier. It is commonplace to use these identifiers to pin a dependency to a specific commit.
In addition to these immutable deterministic identifiers, there are also so-called “refs”, which are mutable identifiers. You use these to point to e.g. “the most recent version”.
So if the agent needs a “skill” that is pinned to the commit with ID
aaaaaaaaait will run the checkout command with that ID; this should prevent any shenanigans because only* a commit with the specific content that was vetted earlier will have that ID.This attack exploits an unexpected git behavior: the command to checkout a particular commit accepts either a commit ID or a ref, but tries refs first. So what the attacker does is add a ref to the repository named
aaaaaaaaathat points to the malicious commit. Then the agent runsgit checkout aaaaaaaaaand ends up at the malicious commit instead of the one that was pinned.I don’t know if there’s a way to tell
git checkoutto ignore refs, but this might actually be a vulnerability in other systems that rely on git for this sort of pinning.* to an astronomically negligible probability of the contrary, and ignoring potential cryptographic breaks
This is a plausible explanation, but I am surprised that this git behavior is not more well-known as there have been multiple discussion about “supply-chain attacks” even before LLMs became widespread.
Fwiw a similar problem is well-known in e.g. GitHub Actions even without this confusion.
You can have your GHA scripts use “actions” written by other people, and the way you refer to them is
user/repo@commit, where the commit can be a commit ID or a branch name or a tag name. The docs actually recommend pinning to commit IDs, but in practice most people actually pin to tags, likebob/doit@v1because that will auto-update if bob fixes some bug and updates v1 to point to that. But if bob goes rogue, or gets compromised, the attacker could change the tag v1 to point to a malicious commit, though, hence the recommendation to use commit IDs for all actions that you didn’t write yourself. I don’t know if this ref/ID confusion can be exploited on GHA, but I would expect the answer is no, because GitHub Actions doesn’t launch git shell commands like a savage, but idk@gerikson @rmf Yeah I don’t see how this is a vuln that specifically targets LLMs rather than just git users in general. If the vulnerable command is
git checkout BLAHbecause BLAH has been poisoned from a SHA to a ref, that’s going to affect anyone or anything that issues that command, human or machine, surely?You’re right that this can affect everyone, but there are different likelihoods of falling into the trap.
Humans are unlikely to issue checkout commands with commit IDs, plus git does issue a warning when a ref name is ambiguous like this. We just use ref names almost exclusively in normal workflows, with commit IDs used only if you’re doing some kind of debugging or audit, but even then git has tools that let you avoid that (bisect, “parent of X” refs, etc)
So that leaves mostly automated stuff. Lots of things that rely on git behind the scenes don’t just go out and invoke the git binary, they use something like libgit2 and (depending on programming language) this will have typed interfaces that prevent treating a commit ID as a ref name and vice-versa. So something like
let c = Commit::new(pin); repo.checkout_commit(c);(I haven’t actually used libgit in a long time, this is illustrative, not real code) will never fall into this trap.This is why LLMs are more likely to fall into this than other software, because it’s all ducktaped together and they run shell commands like savages.
@rmf I’m not disagreeing with any of that. This is definitely a class of mistake (let’s be generous and call it that) which LLMs are more likely to make than humans in normal circumstances.
But it’s being touted around as an LLM-specific flaw which it isn’t. They do have their specific weaknesses, prompt injection of course still being a major one (e.g. Meta’s AI being persuaded to gzip & copy over its entire filesystem). But this is, if anything, a weakness in git. And I guess not entirely new, either, given that Github (and maybe other hosts) explicitly prevent users from creating a ref that looks like a SHA.
I agree, yeah, this is definitely a git flaw and it should be fixed by git devs. There is no legitimate use case for a ref that looks like a hash, and the behavior should be the other way around: check if it’s a full hash first.
Typically no.
The space of incredibly obvious failures is vast, and this kind of plumbing code isn’t always written by someone who’s been around the block enough times to think about what kinds of things can go wrong. I’ve written worse code when I had only a few years of experience.
However companies should know this; so ideally the tool would have gone through a launch review which should have kicked off a security review where a security expert should have read about the git checkout in the design document and started asking questions like “what happens if the repository is taken over” or “why are hashes and branches and tags all in the same field”?
Of course security experts who think about stuff like supply chain attacks are expensive and slow down the darling vibe-coding workflows of Silicon Valley so…
Aside: even without this particular vulnerability, SHA-1 is considered weak – https://git-scm.com/docs/hash-function-transition, so that should have been thought about as well. Dear supply-chain attackers: maybe there’s still a hole here! Good luck!