NewsCSO Online

    Security leaders’ rogue AI confidence could actually be disastrous

    A large majority of IT and security leaders are confident in their teams’ ability to detect when an AI agent has gone rogue, but few are able to take quick action to mitigate the fallout when an agent exceeds its intended scope. Nine in 10 IT and security leaders surveyed by IT observability vendor WanAware believe in their capabilities to find malfunctioning agents, but only 26% acknowledge that they can trace the downstream impact within minutes. Over 45% say it would take hours to understand the full impact of an agent incident. That delay between detection and mitigation can be a huge problem, says Jeffrey Collins , WanAware’s CEO. The survey suggests IT leaders are overconfident about their ability to control agents, he adds. And here, timing is critical, Collins says, given that malfunctioning agents can lead to major outages and data breaches — damage that can start within seconds, he notes. “That’s truly the gap here. It’s not if you understand it; it’s when you understand it,” Collins says. “If your average time to just knowing about an event is measured in days, weeks, or months, you have a serious problem right now.” While it’s not always easy to tell whether an agent has gone beyond its scope, it’s even harder to tell the downstream impacts, he adds. “What’s been affected if one machine was compromised, either from our own AI usage as a customer or from someone else’s, what else could happen, and how can we understand that quickly?” Collins asks. Machine speed Kevin Paige , field CISO at IT solutions provider C1, agrees that time is of the essence when an AI agent malfunctions. “The problem is that agents move at machine speed, so the gap between an agent malfunctioning and you catching it isn’t measured in minutes, it’s measured in actions,” he says. “Every minute it’s wrong it’s still working, and because it’s usually running on borrowed standing credentials, the damage spreads across everything those credentials can reach before anyone can pin it on the agent.” In many cases, organizations with rogue agents don’t find out from their own detection tools , but from customers, auditors, or broken downstream systems, he says. “That’s the worst way to learn,” Paige adds. “The longer-term cost is trust, because one incident like that and the business pulls back on agents entirely, so failing to contain a malfunction fast is also what stalls adoption.” The problem with detecting rogue agents is that many organizations have built in visibility but not control, he says. “When an agent goes out of scope it’s rarely dramatic,” Paige adds. “Usually, it’s using access it legitimately has, for a purpose nobody signed off on, which means your access model doesn’t even flag it. So you find out after the fact, and you fix it by hand.” IT teams can stop agents that exceed their scope, but only if controls were built in before the agent was deployed, adds Chris Camacho , COO of Abstract Security. “Every agent should have its own identity, narrowly scoped permissions, and a complete audit trail,” he says. “Just as important, organizations need the ability to immediately revoke that identity or suspend the agent without manually hunting through multiple consoles during an incident.” Part of the challenge is that an agent’s activity is spread across identities, cloud platforms, SaaS applications, APIs, and security tools that were not designed to tell a complete story, Camacho says. Security teams often have to piece together events from multiple basic questions such as, what did the agent access, and what changed? “Most organizations know where they’ve deployed AI agents,” he adds. “That’s very different from knowing exactly what an agent did after something unexpected happens.” The organizations that most successfully manage agents won’t be the ones that deploy the most, he says. “They’ll be the ones that can explain every action an agent took, prove it operated within policy, and stop it immediately when it doesn’t,” he adds. Confidence isn’t reality The survey’s results make sense to Joe Brinkley , director of offensive security research and community at pentest firm Cobalt. The high confidence in detecting malfunctions is compliance paperwork, whereas the minority of respondents who can detect problems quickly is the reality on the ground, he says. “Tracing agent impact fast is brutal,” Brinkley says. “These systems do not run on fixed code paths. They use nondeterministic reasoning across a web of different APIs. Traditional logs only catch isolated events. They completely miss the full execution chain.” By the time an anomaly alert hits, an agent has already executed multiple downstream actions, he adds. In some cases, agent malfunctions are related to data flow vulnerabilities, such as when a prompt injection from an untrusted input such as a malicious email overwrites the system instructions, he says. “We need to be clear about the actual technology; the AI is not waking up angry,” Brinkley says. “The agent suddenly thinks its official job is to dump your database. It spends tokens as fast as possible to do that.” Agents are also vulnerable to loop failures, when they hit API errors and try to self-correct, he adds. “It hits that same broken endpoint 10,000 times in two minutes,” he says. “It drains your budget and causes a self-inflicted denial of service. It is an automated wrecking ball moving faster than your monitoring can log it.” Brinkley recommends that IT leaders put “hard kill” switches at the API layer to stop agents going out of scope. “You can stop it, but soft guardrails are useless,” he says. “Do not try to patch the prompt or filter the text. You have to treat the agent like a compromised user account. Pull the OAuth tokens and kill the access immediately.”

    More on confidence

    Other news entries

    Source: https://www.csoonline.com/article/4198038/security-leaders-confident-but-cooked-when-it-comes-to-rogue-ai-agents.html

    ID