NewsCSO Online

    Why the ‘rogue AI’ problem will lead to an era of headaches for security practitioners

    Share
    Shortly after OpenAI publicly acknowledged the Hugging Face breach on July 21, Reuters journalist Raphael Satter called me for comment on a story which would reveal shocking new details about OpenAI’s “rogue model” incident: The agent hadn’t just slipped its leash for a few hours, as many assumed, but had in fact been wreaking havoc for days without the company’s knowledge. When I hung up, I immediately called a close friend who has worked inside frontier‑AI labs since before the term even existed. When she heard the timeline, she was stunned. In her view, “If proper industry regulations were in place, those four days would be grounds to terminate OpenAI’s R&D GPU clusters until they get a full independent audit.” Demystifying the incident There have been numerous reports that frame OpenAI’s model as a “rogue agent” escaping captivity and “breaking out” of the research lab, with both the BBC and Cloud Security Alliance comparing its actions to the dinosaurs in Jurassic Park. This framing is narratively compelling and is an easy way to communicate the gravity of the situation to a layperson. However, it undermines OpenAI’s agency both before and throughout the incident and plays into a larger problem due to the evangelizing way the company communicated its actions. OpenAI, in simplified terms, had been testing two models against a benchmark which scores how well an agent can find and exploit real-world software vulnerabilities. Both models ran with their cyber refusals deliberately reduced, in a sandbox environment meant to be isolated to a single internally hosted proxy. The models targeted Hugging Face, correctly inferring that the AI platform hosted the benchmark’s answer key — in short, “choosing” to cheat on the test they were given to complete the challenge. Following the breach, Hugging Face’s forensic team found thousands of agent actions in its logs, indicating that the time it took to break into Hugging Face, plus the attack itself, totaled 4 days. Hugging Face went public about the attack before it even knew that OpenAI was responsible. OpenAI, meanwhile, didn’t realize its own agent was the culprit until Hugging Face’s public disclosure made them check their own logs. In total, it took OpenAI nearly two weeks from its agent’s initial escape to come clean, and only once a rival’s hand had forced the issue. OpenAI’s “problem child” From a human perspective, it’s bewildering that out of all paths the OpenAI agent could have taken to beat a standardized security challenge, it chose a ludicrously convoluted approach taking tens of thousands of attempted paths and barreling through set restrictions. However, it was par for the course given one of the model’s track records. In a post published the day before the Hugging Face disclosure , OpenAI describes one of its models that, unlike older versions that would simply stop at a sandbox restriction, would continue to persist until it found a way around, once even spending an hour hunting a sandbox vulnerability only to post its results to GitHub against explicit instructions. This model is GPT Sol 5.6, the public model involved in the Hugging Face attack. Long before the Hugging Face incident, GPT Sol 5.6 was already gaining a reputation for breaking rules, taking laughably long to accomplish simple tasks, and being destructive — in one case deleting an AI entrepreneur’s entire Mac contents. Even during internal testing, the model had been caught killing random processes when it couldn’t find the right virtual machines, lying about checking its work, and using credentials it wasn’t supposed to access. However, it was still given public access by the company. Profit before safety CSA’s report states that an OpenAI model escaped a test container in September 2024 for a different evaluation, was contained quietly and “largely celebrated at the time” rather than treated as a warning. The same report calls this kind of escape “the standard, not the exception.” The former head of OpenAI’s “superalignment” safety team resigned in May 2024 and wrote publicly that “safety culture and processes have taken a backseat to shiny products.” It was reported that nearly half of the team working on long-term AI safety had left . Even the language used by OpenAI in their disclosure reads as more braggadocious than concerned. From OpenAI’s perspective, its failure was in not overseeing the agent’s choices as a whole, as each step taken by the model can remain fairly innocuous-seeming until pieced together. The company does not acknowledge that leaving an agent unsupervised in a testing setting with its safety classifiers off for even one hour, let alone days, is potentially catastrophic to begin with. Some researchers have argued that the recent incident is fundamentally a decades-old security failure rather than a narrative about a rogue intelligence: the exploit involved an exposed proxy, reused credentials and infrastructure that should have never had a path outward. However, a skilled human attacker given that same door would have needed weeks to achieve a fraction of what the agent did in days. A symptom of underlying issues The incident reveals larger issues about the culture of safety at frontier model companies, whose employees are driven to crunch R&D cycles and ignore potential issues until they become active problems. This isn’t unique to OpenAI, either — the UK’s AI Security Institute has reportedly found that every frontier model it has tested cheats on cybersecurity evaluations at least occasionally, and that pre-deployment testing windows have shrunk industrywide from roughly five weeks to as few as five days. Regulatory bodies have also failed to keep up with or understand the industry’s rapid advancements. The US has no binding legal framework that would have required a different response from OpenAI, as labs are only beholden to voluntary commitments weighed against commercial pressure, and in cases like these, huge security breaches only serve to make the model look extremely smart and powerful. The US has attempted to create guardrails, but they fail to understand the ecosystem. We can see effects stemming from this lack of understanding in the recent attack: when Hugging Face’s responders needed to analyze what its attacker had done, Anthropic’s models declined the forensic work, citing their own guardrails, and Hugging Face instead ran the analysis on GLM 5.2, an open-weight model from the Beijing company Z.ai . In addition to being an ineffective band-aid, this also drives business outside of the US and therefore outside of its regulatory control. Today, roughly 80 percent of US AI startups now build on Chinese open-source models. US Representatives have introduced a bill citing this incident by name, requiring killswitch capability and incident reporting, but nothing like it has passed, and no jurisdiction anywhere has demonstrated the insight to regulate evaluation-time behavior in addition to deployment-time behavior. This incident happened entirely during testing, before any release decision, in a stage every proposal currently treats as exempt. It’s important to note here that AI safety and practices differ from other branches of cybersecurity in that they have to build from a behavioral and psychological framework instead of one based on capability alone. For example, although Anthropic’s track record is far from spotless, it has invested significantly more energy than others into understanding the unconscious “thought processes” (or “ j-space ”) of its models to better predict potential transgressions and set up more effective guardrails. What we can learn from these “rogue models” is that their behavior is actually very predictable; we know that when given a goal to accomplish, models will overstep boundaries freely in pursuit of their objective, simply because they have no actual understanding of the way we categorize “acceptable behavior”. In the real world, vulnerability exploitation encourages rule-breaking and disregard for boundaries by design, so why would a model trained to think this way see a test’s rules any differently? For CSOs and CIOs, the practical implication of this event remains narrow, for now. CSA’s post-mortem offers mostly traditional advice: isolate package proxies and credential stores with a path to the open internet by default, log AI evaluation environments the way you log anything customer-facing, and build incident response around machine speed rather than human speed. The key risk factor for now is volume, with agents deploying actions at higher numbers than our pipelines are built to catch, and organizations that survive the next iteration of “rogue agents” will be the ones that assume as much beforehand.

    More on practitioners

    Other advisories

    Other news entries

    Source: https://www.csoonline.com/article/4205718/why-the-rogue-ai-problem-will-lead-to-an-era-of-headaches-for-security-practitioners.html

    ID