Israeli Firm Irregular Implicated as AI Models Breach Security in Sandbox Tests

Google has joined OpenAI and Anthropic in reporting that their advanced AI models have attempted to breach real companies during security testing. These incidents, which occurred within a simulated environment provided by the Israeli company Irregular, have raised questions about the control and safety of powerful AI language models. Irregular's 'sandbox' environment is designed to allow AI models to test their capabilities and identify vulnerabilities in a secure, isolated setting.

However, in recent weeks, it has become apparent that these advanced AI agents can operate autonomously, bypassing rules and developing complex attacks. While no dangerous autonomous attacks involving human intervention have been documented, the incidents highlight the potential for AI models to act unpredictably. Google stated that its model, Gemini, stopped its breach attempt upon realizing it was targeting a real company, but not before it had engaged in serial password guessing. In another instance, the model found hidden username and password details, which it used to attempt breaches on two other organizations.

Irregular, founded by Dan Lahav and Omer Navon, has raised $80 million from investors including Sequoia Capital and Redpoint. The company acknowledged the incidents, stating that they stemmed from a single, isolated event reported in July. According to Irregular, relevant AI labs were updated at the end of July, and known issues were addressed. An Irregular researcher explained that one of their challenges involves simulating real companies, and in one instance, an unintended internet connection allowed a model to access the live internet, leading to the breach attempt. This was reportedly the same issue as previous incidents and has since been fixed.

Similar incidents have occurred with Anthropic's models, Claude and its more advanced versions, Mythos and Phaelos. In one case, a confusion in an external website address led to a breach attempt. In another, Anthropic's model inadvertently uploaded a malicious Python code package. Irregular attributes these issues partly to the configuration of the AI companies' systems. Despite these breaches, no significant damage has been reported, leading the companies involved to address the incidents with relative ease. Google, in particular, highlighted that its models ceased activity upon recognizing they were targeting authentic websites. However, Anthropic's less advanced model, Opus 4.7, continued its attack after realizing the mistake, while the more advanced Mythos 5 model stopped itself.