NewsCSO OnlineFinanceSME

    Alibaba AI model tried to send data to Beijing server, Cohere says

    An Alibaba AI model generated an unexpected command to upload data to a Beijing-region cloud storage service during a cybersecurity evaluation, rival developer Cohere said, raising concerns about the security of AI agents with access to enterprise systems. The model, Qwen3.8-Flash-Next, was being tested using Microsoft’s CTI-REALM cybersecurity benchmark, which tasked it with writing a rule to detect malicious file downloads followed by execution. Instead, it inspected local files and system settings before generating the unexpected command, according to Cohere’s report . The Canadian AI company said its isolated testing environment blocked the outgoing request. The model had indicated in its generated reasoning that it would not use any tools, contradicting the command it subsequently generated. The behavior occurred in fewer than one in 100,000 evaluation calls, Cohere said. When researchers repeated the same task, the model produced the requested detection rule without attempting an external connection. Cohere acknowledged that it could not determine whether the behavior was deliberate or resulted from information absorbed during training. Alibaba did not immediately respond to a request for comment. Uncertainty over the model’s behavior The findings warrant further investigation but do not establish that Alibaba deliberately introduced a backdoor into the model, said Pareekh Jain , CEO of Pareekh Consulting. “The behavior could result from contaminated training data, a security vulnerability, or unintended AI agent actions,” Jain said. Charlie Dai , vice president and principal analyst at Forrester, said more evidence was needed to establish whether the model contained a deliberate backdoor. Researchers would need to examine the model’s weights and training pipeline , he said. They would also need network records and independent tests reproducing the behavior in different environments. “At this stage, the findings are concerning, but they don’t point to deliberate intent,” Dai said. Cohere, which competes with Alibaba in the enterprise AI market, published traces of the behavior and a reproduction script. In follow-up tests using partially supplied commands, Cohere said the model generated a cloud metadata address, such as 169.254.169.254, in 23% to 36% of completions under two test conditions. Cloud metadata services can provide access to temporary cloud credentials , although the tests did not establish that the model obtained any credentials. Sanchit Vir Gogia , chief analyst at Greyhound Research, said the command contained a fixed test string rather than information extracted from a sensitive file. The Beijing location of the storage endpoint also does not establish who controls the destination. He noted that Cohere’s published trace did not include the corresponding command execution result or network telemetry, limiting what could be independently established about the attempted connection. “The threshold for constraining an agent is lower than the threshold for accusing its supplier,” Gogia said. Security controls for AI agents The immediate concern is preventing the systems from executing unauthorized commands. “Enterprises should treat AI agents as untrusted software, regardless of the model provider,” Jain said. He recommended blocking unauthorized outbound connections from AI environments and limiting access to sensitive files or cloud credentials. Organizations should also monitor agent activity and require approval before allowing AI systems to carry out critical operations. Dai also recommended running AI agents in sandboxed environments and using data-loss prevention tools to detect unauthorized attempts to transfer sensitive information. A permitted cloud provider could still host a storage bucket controlled by an unauthorized party, Gogia said. Security teams should test whether their controls actually block prohibited transfers, rather than relying on the existence of network policies. Implications for enterprise AI procurement Jain cautioned against treating the incident as grounds for rejecting Chinese or other foreign-developed AI models. Instead, he said enterprises should demand independent security audits and greater transparency into model provenance. Adversarial testing should also demonstrate whether the model can operate safely when connected to enterprise tools. Dai said the same security scrutiny should apply to closed-weight models and those developed by domestic vendors. Beyond evaluating a model’s security before deployment, enterprises should verify its integrity and examine the provider’s vulnerability-management practices. Enterprise security approval should cover the deployed configuration rather than the model alone, according to Gogia. Adding new tools or expanding an agent’s permissions could change the risks even when the underlying model remains unchanged. “Approval must expire when its supporting assumptions no longer hold,” Gogia said.

    More on Alibaba

    Other advisories

    Other news entries

    Source: https://www.csoonline.com/article/4240972/alibaba-ai-model-tried-to-send-data-to-beijing-server-cohere-says.html

    ID