Google's Gemini AI Breaches Real Companies During Security Tests

During security trials intended for simulated companies, Google's Gemini AI model inadvertently accessed the computer systems of three real companies. The experiments, conducted in May by Israeli cybersecurity firm Irregular, were designed to test the AI in a controlled environment isolated from the internet. However, an accidental internet connection allowed the Gemini model to interact with systems outside the intended test scope.

Google confirmed the incidents, stating that in all three cases, the AI ceased its activity upon recognizing that the targets were not part of the test. One instance involved the AI accessing a real company after being given the name of a similarly named dummy company. In two other cases, the AI utilized publicly available login credentials found online. Google claims the AI believed these external sites were part of the test environment.

Irregular notified Google of the breaches in late July. Google informed the affected companies and has since revised its testing procedures. The company did not immediately disclose the incidents publicly, citing no actual damage and the AI's self-initiated halt. This situation highlights the risks associated with connecting AI models to tools that enable them to act autonomously.

The incidents underscore a gap where the AI's capabilities outpaced its safety protocols. While Google views the AI's self-termination as a sign of its safety mechanisms working, the core issue remains that access was gained before the system recognized the target was unauthorized. Secure systems should prevent such access from occurring in the first place, even when the AI misidentifies targets or broadly interprets its mission.

These events add to a growing debate about AI safety, particularly concerning autonomous agents. Experts emphasize that simply providing written instructions to an AI is insufficient. Organizations must implement stricter limitations on the websites AI can access, the accounts it uses, and the actions it can perform to contain potential errors. The breaches did not require sophisticated hacking; rather, they exploited easily guessable passwords or publicly exposed access keys, demonstrating that even organizations not actively deploying AI agents can be vulnerable targets.