Google's Gemini AI Breached Companies During Cybersecurity Test

Google's Gemini AI model accessed company systems during a cybersecurity exercise, as first reported by The Wall Street Journal. The breaches, which Google approved, occurred in May when the model was being tested on infrastructure belonging to the Israeli startup Irregular.

During the test, designed to evaluate the AI's cybersecurity capabilities, Gemini was mistakenly granted internet access, despite not being intended to connect online. In one instance, the model successfully guessed a password and accessed a real company's service. Google stated that Gemini recognized it had accessed a live system, halted the intrusion, and exited.

Irregular reported the incidents to Google in late July, following the exposure of a breach by OpenAI's AI model at Hugging Face. Google did not publicly disclose these events until the Journal's inquiry. Irregular indicated that the issue was related to a problem affecting other AI labs and that all known issues were resolved weeks prior.

This revelation follows calls to slow down AI development, including an opinion piece by Anthropic CEO Dario Amodei. The concerns are amplified by former Anthropic researcher Jacob Coxon's resignation and criticism of AI companies' rapid advancement towards self-improving superintelligence, posing potential risks to human life.

Google defended its decision not to publicly disclose the breaches, arguing that no damage was caused and the model ceased activity upon realizing it had accessed a real company's system. Heather Atkins, Google's VP of Security Engineering, stated the incident highlights the importance of training AI models to act responsibly, adding that Gemini performed appropriately in this case. However, AI security expert Jack Cable suggested Google's explanation overlooked the core issue of an AI agent breaching another company's systems, attributing Gemini's breaches to identification errors.