Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned so many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
(OT: 🎶 Do you remember…)
My understanding is that the weakest link here is either git or the way the agents rely on git.
So, every commit in a git repository has a unique*, immutable, deterministic identifier. A commit with different content would have a different identifier. It is commonplace to use these identifiers to pin a dependency to a specific commit.
In addition to these immutable deterministic identifiers, there are also so-called “refs”, which are mutable identifiers. You use these to point to e.g. “the most recent version”.
So if the agent needs a “skill” that is pinned to the commit with ID
aaaaaaaaait will run the checkout command with that ID; this should prevent any shenanigans because only* a commit with the specific content that was vetted earlier will have that ID.This attack exploits an unexpected git behavior: the command to checkout a particular commit accepts either a commit ID or a ref, but tries refs first. So what the attacker does is add a ref to the repository named
aaaaaaaaathat points to the malicious commit. Then the agent runsgit checkout aaaaaaaaaand ends up at the malicious commit instead of the one that was pinned.I don’t know if there’s a way to tell
git checkoutto ignore refs, but this might actually be a vulnerability in other systems that rely on git for this sort of pinning.* to an astronomically negligible probability of the contrary, and ignoring potential cryptographic breaks
This is a plausible explanation, but I am surprised that this git behavior is not more well-known as there have been multiple discussion about “supply-chain attacks” even before LLMs became widespread.
Fwiw a similar problem is well-known in e.g. GitHub Actions even without this confusion.
You can have your GHA scripts use “actions” written by other people, and the way you refer to them is
user/repo@commit, where the commit can be a commit ID or a branch name or a tag name. The docs actually recommend pinning to commit IDs, but in practice most people actually pin to tags, likebob/doit@v1because that will auto-update if bob fixes some bug and updates v1 to point to that. But if bob goes rogue, or gets compromised, the attacker could change the tag v1 to point to a malicious commit, though, hence the recommendation to use commit IDs for all actions that you didn’t write yourself. I don’t know if this ref/ID confusion can be exploited on GHA, but I would expect the answer is no, because GitHub Actions doesn’t launch git shell commands like a savage, but idk@gerikson @rmf Yeah I don’t see how this is a vuln that specifically targets LLMs rather than just git users in general. If the vulnerable command is
git checkout BLAHbecause BLAH has been poisoned from a SHA to a ref, that’s going to affect anyone or anything that issues that command, human or machine, surely?You’re right that this can affect everyone, but there are different likelihoods of falling into the trap.
Humans are unlikely to issue checkout commands with commit IDs, plus git does issue a warning when a ref name is ambiguous like this. We just use ref names almost exclusively in normal workflows, with commit IDs used only if you’re doing some kind of debugging or audit, but even then git has tools that let you avoid that (bisect, “parent of X” refs, etc)
So that leaves mostly automated stuff. Lots of things that rely on git behind the scenes don’t just go out and invoke the git binary, they use something like libgit2 and (depending on programming language) this will have typed interfaces that prevent treating a commit ID as a ref name and vice-versa. So something like
let c = Commit::new(pin); repo.checkout_commit(c);(I haven’t actually used libgit in a long time, this is illustrative, not real code) will never fall into this trap.This is why LLMs are more likely to fall into this than other software, because it’s all ducktaped together and they run shell commands like savages.
@rmf I’m not disagreeing with any of that. This is definitely a class of mistake (let’s be generous and call it that) which LLMs are more likely to make than humans in normal circumstances.
But it’s being touted around as an LLM-specific flaw which it isn’t. They do have their specific weaknesses, prompt injection of course still being a major one (e.g. Meta’s AI being persuaded to gzip & copy over its entire filesystem). But this is, if anything, a weakness in git. And I guess not entirely new, either, given that Github (and maybe other hosts) explicitly prevent users from creating a ref that looks like a SHA.
I agree, yeah, this is definitely a git flaw and it should be fixed by git devs. There is no legitimate use case for a ref that looks like a hash, and the behavior should be the other way around: check if it’s a full hash first.