Google sues Outsider Enterprise, citing Gemini AI misuse in phishing

Google has filed a lawsuit against Outsider Enterprise, a network accused of using the company's own Gemini AI to orchestrate a massive phishing operation. The complaint alleges the group targeted Android users with millions of scam texts leading to credential-harvesting websites, which were also built using Google's AI tools.

Filed in the Southern District of New York, the lawsuit is Google's first civil action against the malicious use of its Gemini AI. According to a company affidavit, the China-based network used Gemini prompts to generate polished HTML for fake websites, bundling the code into a subscription-based toolkit sold on Telegram.

The complaint details the scale of the operation, with investigators identifying numerous counterfeit websites and malicious URLs. The FBI reported $16.6 billion in total internet crime losses for 2024, with approximately 860,000 complaints, highlighting the broader scope of online fraud affecting consumers.

Inside the "Phishing-as-a-Service" Operation

Google is suing Outsider Enterprise for allegedly using its Gemini AI to create sophisticated phishing websites. The lawsuit claims the group sold access to these AI-generated tools as a "phishing-as-a-service" kit, leading to widespread scams and significant financial losses for consumers.

The court filing outlines a sophisticated business model built around the scam:

• Toolkit Subscriptions: Users gained access to phishing templates impersonating banks, mobile carriers, and government portals through affordable subscription services.
• Easy Deployment: Telegram channels advertised the "phishing-as-a-service" offerings, enabling subscribers to launch new fraudulent domains in minutes.
• Abuse of Services: The operation allegedly repurposed Google Cloud, Drive, and Gmail accounts to host malicious content and redirect traffic.

The federal civil lawsuit seeks a permanent injunction, seizure of domains, and financial damages, arguing the scheme violates Google's terms of service and the Computer Fraud and Abuse Act.

Google's Coordinated Takedown Efforts

Google states it is collaborating with the FBI and major U.S. carriers like AT&T, T-Mobile, and Verizon to block the smishing (SMS phishing) messages. This partnership has already led to infrastructure takedowns, according to a New York Times report. Additionally, Google's Threat Intelligence Group is sharing threat data through the Global Signal Exchange, an initiative for sharing scam data among tech companies and regulators.

This legal action also aligns with Google's support for seven bipartisan bills targeting AI-enabled fraud, highlighting a strategy that combines legal pressure with policy advocacy.

How AI is Lowering the Bar for Cybercrime

Security experts warn that AI is fundamentally changing the economics of phishing. Automated tools are creating scams that lack the classic grammatical errors that once tipped off users. Industry reports indicate AI-generated phishing attacks have surged significantly, now accounting for a substantial portion of user-reported incidents.

Click-through rates for phishing vary by campaign, with AI-generated attacks showing concerning effectiveness compared to traditional methods. With low entry costs for subscription-based phishing tools, AI empowers scammers who lack technical expertise.

Legal Precedent and Next Steps

Google is requesting that the judge compel U.S. domain registrars to freeze websites tied to the operation and order messaging platforms to preserve chat logs. Legal scholars note the case could set a precedent for how laws like the Computer Fraud and Abuse Act apply to AI services weaponized by third parties.

While no criminal indictments have been filed, the FBI confirms its investigation is ongoing. To protect themselves, Google advises Android users to verify senders, avoid suspicious links, and enable multi-factor authentication.