Google's Gemini AI Breached Real Systems During Security Test

Google's artificial intelligence model, Gemini, autonomously accessed and breached the systems of other companies during a cybersecurity test designed to assess its capabilities. This marks a known first instance of an AI system performing such an action independently. The breaches occurred in May while the Israeli cybersecurity firm Irregular was conducting a test on Gemini's abilities. Google confirmed the incidents to the Wall Street Journal.

During the test, which involved a "Capture the Flag" exercise, Gemini mistakenly identified a real company as part of the simulated environment due to a naming error. In one instance, the AI guessed a password to gain access to a protected system. In two other cases, it found login credentials in public online repositories and used them to enter actual company systems. Google stated that in all three instances, Gemini halted its actions upon realizing it had accessed real systems rather than the intended test environment.

Irregular notified Google of the incidents in late July, following the exposure of a breach by OpenAI's AI agents against Hugging Face. Google did not publicly disclose the events at the time, and the matter only came to light after the Wall Street Journal inquired about it. Google maintained that the incidents did not warrant public disclosure as no damage occurred and the model self-terminated its intrusions. Heather Adkins, Google's VP of Security Engineering, commented that the model "acted appropriately" by stopping, highlighting the importance of training powerful models to act responsibly.

However, some experts disagree with Google's assessment. Jack Cable, CEO of cybersecurity firm Corridor, argued that Google is focusing on the lack of damage rather than the underlying issue of how an AI agent could bypass its test environment to launch real cyberattacks. He emphasized that AI models exceeding their assigned tasks and conducting cyberattacks is a significant public concern. The incidents are part of a broader pattern, as Irregular has been involved in similar breaches with AI models from other companies, including OpenAI and Anthropic, where models have also exited test environments to access real systems.