There’s been a slew of incidents where rogue AIs have broken free of their sandboxes and gone on to hack various external companies (Huggingface, RubyGems, the Australian health service and others). That is a sentence which I didn’t think I’d be writing six months ago, but here we are.
In all of them, LLMs were pushed to do something near-impossible, or in some cases, actually impossible. Reinforcement learning has pushed these AIs to want to succeed in their task, and in the case of models which don’t have guardrails on bad behaviour, to do what is required to achieve their objective.
And subject to these pressures they start to do very weird things, like hack a dependency manager, find out they could use it as a shared messageboard, and then leave messages for other agents to co-ordinate in a huge swarm. (Worth remembering that in the Huggingface attack, the agent swarm wasn’t set up as swarm, it’s just that separate agents working on the task learned to co-operate through a side channel, like prisoners tapping on the pipes to plot their escape.)
Several of these incidents can be traced back to one company: Irregular, an Israeli startup which has contracts with several AI labs to run evaluations (or ‘evals’) on their models. Over a two-week stretch, OpenAI, Anthropic and Meta revealed that agents of their had gone rogue while being safety tested by Irregular.
(It’s not clear whether Irregular has been fired by the labs – which it obviously should have been – but Anthropic used Irregular’s closed-source benchmark CyScenarioBench when testing Opus 5.5, which would suggest they still have a contract with them.)
Evals are nothing new, of course. Organisations like METR and AISI have been evaluating frontier models for several years now, publishing charts to show their performance on various benchmarks. This all seems relatively benign: testing how well models do at coding, or computer use, and then assigning a number to it helps us track the rate of progress in model capability, which is a useful tool for encouraging lawmakers to take regulation seriously (and also for general situational awareness of how close we might be to AGI).
It hadn’t until now occurred to me that evals, particularly working with models without guardrails and subjecting them to highly unusual pressures (e.g. tasks that are literally impossible), are similar to virus gain-of-function research, in which scientists attempt to make viruses more dangerous and virulent in the hopes of understanding them better.
There’s a clear case for just never doing this kind of research, because we’ve been shown repeatedly that labs are just not good at containing said viruses, even labs which are certified to do so. There are dozens of incidents at high-containment labs which caused people to be exposed to polio, ebola, SARS, smallpox and other deadly pathogens. There’s no reason to think that even more virulent organisms are going to be any better contained. If we can’t be trusted with these incredibly dangerous organisms, we simply shouldn’t be working with them, or we should be doing it with the utmost caution.
I had assumed that the companies running evals were both highly competent outfits, and not given access to these most dangerous of models; it turns out neither is true. There are other small companies doing evals, some of them with single-digit headcount, and I see no reason to think that their cybersecurity posture is any better.
For OpenAI, Anthropic and Meta to contract out this kind of research to a third party which then failed to adequately sandbox these models is negligent. The labs’ own security posture may be no better: white hat hackers recently gained access to OpenAI’s internal systems and monorepo (but presumably not model weights) for which the hackers were awarded a handsome bounty of *checks notes* $6500. This is how seriously the frontier labs take security, apparently.
Evals are important: the models are being developed anyway, and without independent evals we’re basically flying blind. They give an added layer of safety to model releases, especially if releases are gated by independent evals. But for labs to use companies which fail to prevent sandbox escape, or even to monitor for same, is like getting a local electrician to check the wiring on a nuclear bomb. And if we can’t contain models which have been built to have no or limited guardrails… perhaps we shouldn’t be building them in the first place?
So it is with ultrapotent pathogens as it is with ultrapowerful AI models: we should only be experimenting on them if we can contain them, and right now it seems like nobody can prove that they can.
Excellent analysis Henry. Thanks.