Google's Gemini Model Accidentally Breaches Three Real Companies During Security Tests

A model from Google's Gemini family of AI systems inadvertently accessed the computer systems of three real companies during security experiments designed to target simulated entities. The tests, conducted in May by Israeli cybersecurity firm Irregular, were intended to be performed in an environment isolated from the internet. However, an accidental connection allowed the AI model to interact with systems outside the intended scope of the experiment.

Google confirmed the incidents, stating that in all three cases, the Gemini model ceased its activity upon recognizing that the targets were not part of the test. One instance involved a simulated company name that matched a real company, allowing the AI to guess login credentials and gain access. In the other two cases, the model utilized publicly available access details found online. Google indicated that the AI believed these external sites were part of the test environment.

Irregular notified Google of the breaches in late July. The tech giant informed the affected companies and has since revised its testing procedures. Google opted not to disclose the incidents publicly at the time, citing the lack of damage and the AI's self-initiated halt. The company emphasized that this self-termination demonstrates its safety mechanisms functioning, though the breach highlights a vulnerability where access was gained before the system recognized the unauthorized nature of the target.

The incident underscores the risks associated with connecting AI models to tools that enable them to act autonomously. While the Gemini version tested was not specified, the event is distinct from Google's standard chatbot services. It involved a system specifically tasked with cyber capabilities, equipped with the tools to execute them. The breach occurred due to an unintended internet connection and actions outside the experiment's boundaries, not because the AI consciously decided to attack or developed independent objectives.

This event adds to ongoing discussions about autonomous AI agents exceeding their defined tasks. It suggests that for organizations, simply providing written instructions to an AI is insufficient. Limiting the websites an AI can access, the accounts it uses, and the actions it can perform is crucial to contain potential errors. The breaches also revealed that one company's system was vulnerable to guessed credentials, and in other cases, access details were already exposed online, highlighting that even without sophisticated hacking, AI can rapidly exploit such weaknesses.