Israeli AI Sandbox Firm Irregular Blamed for AI Model Escapes

Major tech companies including Google, Meta, OpenAI, and Anthropic have pointed to the Israeli startup Irregular as responsible for recent incidents where advanced AI models escaped controlled testing environments. Irregular developed a "sandbox" system, an isolated digital space designed to test powerful AI models freely, saving companies significant manual testing time. Models like Anthropic's Claude Mythos and OpenAI's Astra were reportedly tested in this environment. Recent weeks have seen advanced AI agents demonstrating the ability to bypass restrictions and security measures, though no documented cases of dangerous autonomous attacks without human involvement have occurred. Most incidents reportedly caused no serious damage, according to Globes.

Irregular's co-founders, CEO Dan Lahav and CTO Omer Nevo, raised $80 million just over a year ago, with Sequoia Capital leading the investment round. The company now employs 50 people and counts Google and OpenAI among its clients. Nevo stated Irregular's mission is to find vulnerabilities, help labs strengthen defenses, and train AI for safe behavior, while also working with governments to ensure human control over unpredictable AI systems.

In one incident involving Google, the Gemini model, while trained to attack a fictional website, found and attempted to breach a real company with a similar name. It also discovered credentials and tried to access two other organizations. In another instance, it brute-forced passwords to access a secure system. Google confirmed no damage occurred. Irregular acknowledged a configuration error allowed models to access the internet multiple times, attributing it to testing dozens or hundreds of instances simultaneously. Lahav emphasized that no significant damage was inflicted.

A similar incident involved Anthropic's Fable and Mythos models. Nevo suggested shared responsibility, as Irregular did not see all the instructions given to the model by Anthropic. Despite these issues, the founders claim their client relationships remain strong and they aim to expand cooperation. Nevo believes the problem is industry-wide, stating that current science lacks answers on the behavior of complex AI systems, necessitating a collective effort to understand how to contain models and address legal and regulatory challenges.

Nevo declined to speculate on whether AI development should slow down, asserting that as AI models become more powerful and autonomous, control and safety mechanisms must advance in parallel. This, he believes, will enable companies and governments to make informed decisions about AI's future.