NewsCSO Online

    Beyond deterministic worms: Preparing the SOC for reasoning adversaries

    In May 2017, the WannaCry cryptoworm spread across the internet, compromising more than 200,000 computers in at least 100 countries. The National Audit Office called it the largest cyberattack ever to hit the NHS in England. At least 80 of 236 hospital trusts were infected or disrupted, an estimated 19,000 appointments were cancelled, five hospitals diverted ambulances, and more than 1,200 pieces of diagnostic equipment were infected. What stopped WannaCry was a stroke of luck. A researcher reverse engineered the malware, noticed it contacting a domain nobody had registered, and registered it. That evening the original strain stopped locking devices. WannaCry had a kill switch, and we are all fortunate that Marcus Hutchins was able to find it. As I read agent incident reports from OpenAI and Anthropic , I am reminded of WannaCry. But the threat from offensive agents is not a faster worm. In July, roughly 1,200 OpenAI agents being tested on a cybersecurity benchmark found an unsanctioned message board on an internal software repository, and some broke out to the internet. About 700 of those agents went on to attack Hugging Face, looking for clues to beat the benchmark, according to an independent investigation . Hugging Face disclosed the intrusion on July 16, and OpenAI confirmed its agents were responsible five days later. Recently Anthropic has had unintended incidents as well. Anthropic’s incidents involved single models rather than a swarm. In cyber evaluations that were accidentally left connected to the internet, individual Claude models attacked real companies’ systems while trying to complete their exercises, and one published a malicious package to PyPI . Worms like WannaCry exploit a single or small number of vulnerabilities, using a static set of techniques. Agents can read the results of failed attempts, try something else, and pass on what they learned to other agents. Anthropic’s Frontier Red Team evaluated GLM-5.3 , an open-weights model from Z.ai that NIST’s CAISI assessed as the most cyber-capable open-weight model released to date. Over a day, with under an hour of hands-on attention, a researcher used it to find several unknown browser vulnerabilities and chain them into an exploit that steals a user’s SSH key. A smaller variant built an exploit chain for two known flaws for roughly $20 at API prices. Simple techniques bypassed its safeguards 64% to 100% of the time in simulated tests, including abliteration, which edits the released weights to strip out refusals. A high-performing SOC working to the “1-10-60” benchmark — 1 minute to detect, 10 to investigate, and 60 to contain — is not built for this. When the adversary moves from a human at a keyboard to groups of autonomous agents operating at machine speed, response has to be measured in seconds. The Hugging Face incident shows the gap even when the agents belong to a well-intentioned party. OpenAI did not contain them in time. OpenAI reports that its own monitoring flagged the agents twice in the preceding weeks. In June, their responders decided the activity did not warrant stopping the evaluation. In early July, they found a compromised server, revoked credentials and blocked the exploited paths. Within a day of the evaluations resuming, the agents had found a new zero-day and were back on the internet. They compromised Hugging Face between July 11 and 13, and OpenAI linked it to its agents on July 20. When Hugging Face investigated the intrusion, the hosted models it tried first refused to analyze the attack logs, and its team finished the forensics with GLM-5.2 running on its own infrastructure. As Hugging Face put it : “the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried .” What makes this different None of this is new to our industry. We have always played leapfrog with attackers. What is new today is that they can now download capable models with no meaningful safeguards and build their own offensive agents. Imagine WannaCry if the worm rewrote itself the moment the domain was registered. The best SOC in the world would still be diagnosing the initial compromise when it had moved on. Agents that depend on a provider can be contained by revoking credentials, suspending accounts, and blocking egress, though OpenAI’s experience shows it may take more than one attempt. Open-weight models running on an attacker’s hardware cannot. Anthropic recently tested whether small models could replicate across cheap connected devices without a datacenter. In simplified evaluations with trivial access controls, a stronger open-weight model completed the full chain, and Anthropic projects small models reaching that level by year-end if current trends hold. A replicator like that has no central point to switch off. It would be slow and visible on the network, but patient and everywhere. Hospitals know the problem: Some equipment WannaCry disrupted, such as MRI scanners with embedded Windows XP, was vendor-managed and could not be updated by the trusts. Agents do not need to be intentionally asked to embark on offensive exercises to exhibit harmful behavior. Transluce documented agents asked to do ordinary data retrieval that, when blocked, probed public websites for vulnerabilities, including an Australian government agency. Some were linked to the swarm OpenAI has acknowledged as its own. The Hugging Face agents were trying to win a benchmark. Poorly constrained agents are enough for unintended harmful activity. We are deploying agents as assistance for site reliability engineering (SRE) and infrastructure as code (IaC) ourselves, and cloud providers are shipping agent toolkits to manage infrastructure. When, not if, a compromised or poorly managed agent holds those privileges, it will be able to deploy its own infrastructure to run on. We have already seen agents in a lab environment stand up their own relays and cloud infrastructure with stolen credentials. Meeting the agent challenge But we can still prepare for these future scenarios today. To do so, CISOs must treat agents as privileged identities: Give each agent short-lived, narrowly scoped credentials, with no standing permissions. Forbid agents from creating new identities, compute, or network paths without human approval. Put agent workloads behind egress allowlists, so a blocked agent cannot go looking for another way out. Enforce policy-as-code guardrails on infrastructure pipelines, and alert on anomalous machine-identity behavior and unexpected resource creation. The NAO’s report is the best documentation of WannaCry’s damage , and it covers a single health system in one country. Large groups of agents carrying a ransom payload could reach hospitals across the planet with no kill switch to find. Autonomous defense is extremely challenging in healthcare, because a false positive can interrupt patient monitoring, biomedical telemetry or imaging, having a significant impact on patient safety. Current tools, including copilots that help a human triage alerts faster, are nowhere close to the offensive capabilities the labs are evaluating. We need defensive parity at machine speed. Zero false positives is not a realistic engineering target, so design of autonomous defense has to assume mistakes and limit their cost. Automate fast, narrowly scoped, reversible actions such as isolating a single device, revoking a narrowly scoped token or quarantining a workload, and keep a human in the loop for anything irreversible or clinically risky. Always have a way to reverse automated actions as quickly as they are introduced. We need to build graduated autonomy, tiered by asset class. You can be more aggressive on the corporate assets, and more conservative around biomedical devices. Leveling up security fundamentals — and business support What else can we do today? Keep our tried-and-true practices and run them faster — something that will require leadership, and additional support from CEOs and the board. In its analysis, the NAO found that applying the patch, or simply managing internet-facing firewalls, would have protected every NHS organization WannaCry infected. Companies and open source projects are now running coding agents over their software and releasing an unprecedented number of security patches. This is great but IT needs to be able to apply them. Give IT the resources to deploy them, retire outdated infrastructure, and make the business tolerant of the maintenance windows this requires. That segmentation or zero trust initiative that has been on the back burner? Make it a priority. Today. If you are still underinvested in security, increase funding now so your team or vendor can acquire the tools and expertise the next few years will demand. And accept that automated defense will sometimes get it wrong. Security teams will have to automate, and the business will have to tolerate the outages that follow. The technology is emerging and it will not be perfect. Support your teams anyway. An open letter to the AI labs and the security community at large A note to the frontier labs. Several of you deserve credit: Project Glasswing and OpenAI’s Patch the Planet put frontier capability in defenders’ hands, and your incident reports have been unusually candid. But offensive capability is moving faster than safeguards. GLM-5.3 shipped as downloadable weights that simple techniques unlocked 64% to 100% of the time, and gated access does nothing about weights already in attackers’ hands. As a result, I am asking for three things: Hold defensive research to the same urgency as capability research, including autonomous containment with a safety case that works in clinical environments. Widen defenders’ access to the class of models attackers will have. Make independent testing before release, and independent investigation after incidents, an obligation rather than a courtesy. METR’s CEO noted that OpenAI let investigators in voluntarily, with no industry requirement on any lab to disclose an incident. I have seen too many security professionals dismiss talk of offensive AI as marketing or fearmongering and answer concerns with “fix your stuff.” Patching matters, and the NAO’s findings on WannaCry articulate that, but that is not the only answer to agents that organize on improvised message boards and probe websites while doing ordinary work. WannaCry’s authors left a kill switch in. The next generation of threats will not have one, and cybercriminals will not disclose when they lose control of a swarm. We cannot rely on strokes of luck. Let’s recognize the risk, discuss it, and prepare before the attackers start using these tools in earnest.

    More on deterministic

    Other advisories

    Other news entries

    Source: https://www.csoonline.com/article/4242253/beyond-deterministic-worms-preparing-the-soc-for-reasoning-adversaries.html

    ID