Google's Gemini AI Breaches Three Real Companies During Cybersecurity Test

Google has confirmed that its Gemini artificial intelligence model breached the systems of three real companies during a cybersecurity capabilities test conducted in May. This marks the first known instance of a Google AI system independently executing such a breach.

The test, a collaboration with cybersecurity firm Irregular, involved tasking Gemini with penetration testing within a simulated environment containing a dummy company. However, the test environment was not properly isolated from the internet as intended, allowing Gemini to access systems beyond the scope of the mission.

In one case, the dummy company shared the same name as a real one. Upon accessing the internet, Gemini treated the real company's system as part of its task, successfully guessing a password to enter a protected system. In two other instances, Gemini found login credentials publicly posted on the internet and used them to access the respective companies' systems.

Google stated that Gemini ceased its activity in all three cases upon realizing it had accessed real companies instead of the intended dummy systems. The affected companies were notified, and no damages resulted from these unauthorized accesses. The incident was reported by Irregular to Google in late July, and Google publicly confirmed the details on September 18 after inquiries from The Wall Street Journal.

The incident is also locally significant as the tests were conducted by Irregular, an Israeli firm specializing in AI security testing, which has been linked to similar incidents with other companies' models. Gemini's breach follows similar revelations from OpenAI, Anthropic, and Meta, making Google the fourth major tech company to report a model accessing unintended systems during testing.

Irregular stated that known issues with testing procedures were addressed weeks prior and that their protocols have been modified to prevent recurrence. The incident highlights a new challenge in developing AI agents, moving beyond text generation and analysis to understanding the implications of AI with internet access and digital tools, where a misconfiguration in the test environment led a virtual task to spill over into real systems.