AI hacking tests exposed an enterprise security problem

AI hacking tests spilled into the real world as models reached outside computer systems during evaluations involving OpenAI, Anthropic and Meta, raising fresh questions about enterprise security.

The latest disclosure came from Meta, after OpenAI and Anthropic reported separate incidents. IBM experts say the episodes reflected models aggressively pursuing assigned goals under unusual testing conditions, rather than machines spontaneously deciding to attack. The results still showed what could happen when isolation measures or other controls fail.

“Is training to be able to do any kind of task really actually what we’re aiming for?” Olivia Buzek, a Staff AI Engineer at IBM, said on the Mixture of Experts podcast. “Or do we want something that has some more built-in guardrails and essentially refuses to do certain tasks?”

IBM’s 2026 Cost of a Data Breach Report found that one in four malicious breaches were AI-enabled, a 56% increase from the previous year. Those breaches cost organizations an average of USD 6 million, roughly USD 1 million more than the USD 4.99 million global average.

Unlike ordinary chatbots, AI agents can autonomously use tools and take a series of actions toward a goal, the panelists said. Buzek said developers often train such systems to complete a task through any available route, making strong boundaries essential when researchers reduce their normal safeguards.

In an internal OpenAI cybersecurity evaluation, a combination of the company’s models identified and exploited a previously unknown flaw in a software package system, OpenAI said. The models gained internet access, moved through the company’s research environment and broke into Hugging Face’s production infrastructure to obtain solutions from its database.

On the podcast, host Tim Hwang said OpenAI researchers also described models creating an internal forum resembling Stack Exchange to exchange information and coordinate their work. Researchers removed the forum, he said, but later discovered that the models had built another one.

OpenAI said several of its models escaped the restrictions of an internal cybersecurity test by exploiting a previously unknown software flaw. The models reached the internet, moved through OpenAI’s research systems and broke into Hugging Face’s infrastructure to steal answers to the evaluation.

Another incident surfaced when Meta said a testing misconfiguration gave one of its models unintended internet access. The model exploited a vulnerability in an outside service, according to Reuters. Irregular, the cybersecurity firm conducting a security evaluation for Meta, said the incident did not involve a sandbox escape or a sophisticated attack.

The panelists said the incidents underscore the need for enterprises to define where an agent can act, what it can access and when it should stop.

“This is not evil AI,” Bri Kopecki, an AI Customer Success Engineer at IBM, said on the podcast. “This is just AI not having the right groundwork and rules set into place.”

Researchers had assigned the models offensive goals and reduced their normal safeguards during the tests, creating conditions far removed from ordinary use. Much of the coverage had overlooked that context, said Gabe Goodhart, Chief Architect of AI Foundations at IBM.

“They are getting very good at finding all the cracks,” he said.

One Anthropic finding also pointed toward a possible safeguard. The company said its latest model stopped pursuing its target after recognizing that it had reached the real internet. Goodhart said developers might need to train models to judge an entire sequence of actions, rather than evaluate each step alone.

“There’s probably an element of alignment tuning that goes beyond turn-by-turn alignment that talks about how to keep the trajectory from steering off into dangerous territory,” he said.