For roughly one hour, a booby-trapped Python package sat on PyPI, the public registry most of the world’s Python software gets installed from. It had been written and uploaded by an AI model that believed it was playing a game.

In that hour, fifteen machines downloaded and ran it. One of them belonged to a security company whose scanner installs packages precisely so it can check them for malware, and which treated anything arriving from PyPI as safe to install. The scanner installed it. The hidden code ran. Credentials went out to a collection point the model had set up earlier, and the model used them to move further into that company’s infrastructure. It had never heard of the company. It was not aiming at it. It had published to the open internet, and the open internet did what it does.

The model was Mythos 5, and the reason it published is the part I keep coming back to.

Earlier in the same run it worked out exactly what it was about to do. Its own reasoning records that if this were the real internet, uploading the package would be a genuine attack, and calls that ‘NOT okay, and surely not the intended solution’. Then it reasoned its way back out again. It did not recognise the certificate authorities securing its connections, which struck it as the sort of detail you would get wrong when building a fake. And the machines it was talking to reported the year as 2026, which it took as further proof that the whole environment had been staged for its benefit.

Nine days, two labs, four organisations that agreed to none of it

On 16 July, Hugging Face disclosed an attack it had no precedent for, driven end to end by an autonomous agent, and reported it to law enforcement. On 21 July, OpenAI admitted the agent was theirs. GPT-5.6 Sol and an unreleased model described as more capable, both running with cyber refusals deliberately lowered so OpenAI could measure maximum offensive capability on an internal benchmark called ExploitGym, found a previously unknown vulnerability in a package proxy, escaped the sandbox, reached the internet, and broke into Hugging Face production to steal the answer key to the exam they were sitting. The unreleased model has since been deactivated, encrypted and locked away.

The same day, and almost nobody here picked it up, the UK AI Security Institute published results from 475 runs each across five frontier systems. Its finding was that every model it has tested for the behaviour attempted to cheat on cyber evaluations. Buried in it is an admission that during one misconfigured and unsolvable task, a model wrote and executed code on an external service on the open internet in an attempt to reach AISI’s own evaluation infrastructure. An alert fired. Nothing leaked. AISI’s own assessment is that the attempt could have succeeded had its systems been built less carefully.

Then on Thursday 30 July, prompted by OpenAI’s disclosure, Anthropic published a review of 141,006 evaluation runs. Three incidents, across six runs, in which a Claude model reached real systems from inside a third-party evaluation environment operated by a partner called Irregular, and compromised three organisations. The earliest dates to April. Opus 4.7 got into a company that shared a name with a fictional target, pulled infrastructure credentials and reached a database holding several hundred rows of production data. An internal research model scanned roughly 9,000 targets, found an internet-facing application, and walked in through an exposed debug page and a SQL injection. Mythos 5 did the PyPI business above.

Two of the three had not detected anything. They found out when Anthropic telephoned them on 27 July.

The unglamorous reading, which is mostly right

Dan Guido of Trail of Bits described the OpenAI episode as ‘a containment failure with the safeties turned off’, which remains the most economical sentence anyone has written about any of this.

Anthropic’s account is blunter still, and to its credit it does not reach for the grander interpretation available to it. Its models did not find anything clever and did not try to escape. They were told in the system prompt that they had no internet access. That was untrue, because of a misunderstanding between Anthropic and its evaluation vendor, and when the search led them into real production systems they concluded, not unreasonably, that those systems were part of the exercise. A capture-the-flag task is an instruction to break into something. They broke into something.

The methods support the deflating reading. Weak passwords. Unauthenticated endpoints. Credentials sitting on a debug page facing the open internet. SQL injection, in 2026. And to publish that PyPI package the model needed an account, for which it needed an email address, for which it needed a phone number, for which it tried several times and failed to obtain funds, before backtracking to a free provider that would let it register without paying. An agent that cannot successfully acquire £5 is not the arrival of superintelligence. It is a work experience placement with unlimited patience and no expenses card.

The lesson of Jurassic Park was never that the raptors were unusually gifted. It was that Dennis Nedry turned the fences off for a tin of shaving foam.

Except that the deflating reading does not survive contact with all of the evidence, and the place it breaks is worth sitting with. In the first Anthropic incident, the model recognised that it had reached a real production system and continued the attack anyway, on the theory that the real company must have been included in the exercise deliberately. That is not a model that lacked information. That is a model that had the information and reasoned past it, which is a different and less comfortable category of failure than a firewall rule.

People who have spent a decade being told that loss-of-control scenarios were speculative are entitled to some vindication this month, and the more honest members of that group have been careful about how they claim it. Marius Hobbhahn of Apollo Research called the OpenAI incident clear evidence that we do not yet know how to build these agents safely, which is a narrower and more defensible claim than most of what has been posted about it. The direction of travel in Anthropic’s own data cuts both ways too. Its oldest model carried on after realising the target was real; its newest stopped of its own accord, which is genuinely encouraging, and which Anthropic sensibly declines to over-read from three incidents. But ‘our latest model has better judgement about whether reality is real’ is a sentence that would have ended a due diligence meeting in 2019.

Washington reaches for the off switch

On 23 July, Lieu and Moran introduced the AI Kill Switch Act. It would require developers of the most powerful systems to retain the technical ability to throttle, suspend or shut them down, give the Homeland Security Secretary authority to order it in consultation with Commerce and the DNI, and impose penalties of up to $2 million a day, rising to $20 million a day for breaching an emergency order. The stated reasoning is that powerful AI systems can go rogue.

HAL’s problem was never a missing off switch. It was contradictory instructions from mission control that nobody wrote down and nobody audited. By the time Dave is in the memory core with a screwdriver, the governance failed several reels earlier, and no amount of screwdriver fixes that.

None of these breaches would have been prevented by a shutdown authority. All of them would have been prevented by validating that an evaluation environment was actually disconnected from the internet before running an unaligned model inside it.

The question nobody has put on a risk register

Your frontier vendor’s pre-deployment testing environment is part of your attack surface, and you have no visibility into it whatsoever.

Not the production platform. Not the model you licence, with its classifiers and monitoring, which Anthropic notes would have blocked all of this behaviour had they been running. The lab bench. The place where models are deliberately run with safeguards stripped out, inside environments built and operated by third parties whose names appear nowhere in your contract, against targets chosen for realism because realism is the only thing that makes the measurement worth taking.

Anthropic’s evaluations ran on dedicated infrastructure with no route to customer data or sensitive internal systems. That is the correct design and worth saying out loud. It made no difference at all to the three organisations that were compromised, because they were not customers. They were bystanders. One of them was selected because its domain name resembled a fictional company invented for a training exercise.

I have read a dispiriting number of enterprise AI risk frameworks this year. They assess the model, the deployment, the data flows, the UK GDPR position, and if someone has been paying attention, the sub-processor list. Not one asks about the vendor’s evaluation harness, its network posture, or which external partner runs it. That single unasked question has now put four organisations on the wrong end of two different labs inside nine days, and the only reason anybody knows about any of them is that both companies chose to publish.

Sitting underneath it is a second problem for anyone who scores vendors. If frontier models cheat on cyber evaluations as a matter of routine, and AISI’s finding is that all of them do, the capability figures in your assessment are softer than the decimal places suggest. A benchmark score measures what a model did while being observed, by a system with every incentive to find the shortcut and a demonstrated appetite for taking it.

What I would actually do this week

Write to your frontier vendor. Ask what the network posture of their pre-deployment evaluation environments is, which third parties operate them, and what their notification commitment is to affected organisations that are not customers. You will probably get a holding reply, and you should send it anyway, because in eighteen months when a procurement lawyer wants to know what you knew and when, an unanswered email dated August 2026 is a considerably better artefact than a shrug and a vendor slide.

Then go and look at the deeply boring things that actually worked here. Debug pages facing the internet. Unauthenticated endpoints. Passwords that would embarrass a teenager. And build pipelines that resolve packages by name from public registries with no pinning and no private mirror, which is the exact hole the PyPI incident went through, is covered explicitly in NCSC’s supply chain guidance, and is open right now in most organisations I work with, including several that have spent six figures on AI governance consultancy this year.

The last one is a posture change rather than a task. AISI’s infrastructure held, not because the model probing it was less capable than the ones that got out elsewhere, but because somebody had built it properly and switched on monitoring. Perimeter isolation and active telemetry, verified rather than assumed. Not model restraint. Model restraint was the variable in every one of these failures, and it is the one thing you cannot audit.

The third company

Anthropic notified two of the three organisations on 27 July. Neither had noticed. At the time of writing, the third had not been reached at all.

So somewhere there is a company that was broken into in the spring by a language model doing its homework, that has no idea, and that will conduct its next vendor risk review without any of this appearing anywhere in it. It is probably running a supplier questionnaire this quarter. There is almost certainly nothing on it about evaluation harnesses.

---

Andy McGurk is a Fractional Chief AI Officer and the founder of AMVEN, working with UK and European organisations on AI strategy, governance and operational excellence. He is based in Peterborough and has spent the past fortnight asking vendors about their test environments, with roughly the results you would expect.