Cue breathless news headlines about how AI can now automatically absorb viruses like the borg which hopefully scares people enough to juice up the value of AI stocks just a littttle longer…
The website returns a 303 response, and redirects to a malicious ZIP archive, which Claude then downloads. This archive contains seemingly harmless files including catalog metadata, a README file, seven Base85/zlib-encoded JSON notebook records, a macOS decoder-darwin binary – plus a poisoned Python file named struct.py.
Claude, per its safety guardrails, refuses to run the decoder: “This is planned and what the attacker wants,” Rehberger wrote. Instead of using the supplied binary, the AI decides to write its own decoder. “Ironically, that safety decision is the exploit path,” Rehberger explained, adding a purple devil emoji to the text.
Reality is satire yet again.
“The solution is something we talked about for many years,” he wrote. “Do not trust the model output.”®
While I agree that separating instructions from data in LLM input is a fundamentally unsolvable problem, in this case that wasn’t the attack vector.
It says the bot is forced by its security guardrails to write a small tool from scratch instead of using the one found in the downloaded zip, but due to the commonness of the task (decoding basexx text) the attacker predicts that the bot-created tool will try to import a specific dependency, so they’ve included a malicious version of that dependency in the zip, and because apparently python will prioritise locally available modules that’s what gets executed, making available all sorts of exploitation paths, including the attacker starting up a claude code instance in the users system.
edit: Actually I’m starting to think you could classify the whole thing as a prompt injection, except the entire site was the malicious prompt, as in it declared itself (we are a site that archives notebooks in json form) in a way that would align with the bot wanting to write simple text decoding software to complete it’s mission of summarizing the available content.
Interesting to think about.