Google’s Threat Intelligence Group (GTIG) revealed the use of Google’s Gemini AI tools by more than 40 state-sponsored advanced persistent threat actors (APTs) from Iran, China, North Korea, Russia and at least 16 other countries.
Threat actors used the Gemini large language model (LLM) to support activities in every phase of the attack life cycle, the GITG said in a blog post last week, but this use only resulted in “productivity gains” rather than the development of “novel capabilities,” the researchers noted.
The finding is consistent with a report by Microsoft and OpenAI last year that found that Iranian, Chinese, North Korean and Russian state-sponsored actors used ChatGPT in a limited and experimental manner for tasks such as scripting and phishing help, vulnerability research, target reconnaissance and help with post-exploitation activities.
Iranian threat actors were the most prolific adversarial users of Gemini for hacking activity and influence operations, according to Google’s report, while Russian threat actors were noted to make limited use of the AI tool.
North Korean threat actors used Gemini for activities consistent with the North Korean government’s ongoing IT worker campaign, including reconnaissance on international companies, searches for job listings and generation of work proposals and cover letters, in addition to seeking assistance with malware development, post-compromise activity and other research.
More than 20 China-backed groups also used Gemini in attempts to streamline their hacking activities, including by seeking information on U.S. critical infrastructure, vulnerabilities, Windows exploits and methods for lateral movement across compromised systems.
Threat actor AI requests span entire attack life cycle
GTIG identified threat actor activity spanning across seven attack life cycle segments including victim reconnaissance, tool weaponization, payload delivery, vulnerability exploitation, malware installation, command and control communications, and actions to achieve adversarial objectives like data theft or system disruption.
Reconnaissance efforts differed across APTs from different nation-states; for example, Iranian threat actors sought information on international defense and government organizations as well as information related to the Iran-Israel proxy conflict, while North Korean actors researched US military operations in South Korea and free web hosting providers, and Chinese APTs focused on US IT providers, military and intelligence personnel.
Threat actors also sought a range of information and assistance on the development of malware and exploitation of tools and vulnerabilities. Nearly a third of the Iranian threat actor activity was noted to come from APT42, also known as Charming Kitten, which focused the majority of its effort on researcher publicly known vulnerabilities in products from providers such as Mikrotik, Apereo and Atlassian.
Attempts to get help with malware coding across languages, including Python, PHP and C++ were also observed, although Gemini’s safeguards prevented the AI from following instructions to generate and encode malware, according to Google. The threat actors were noted to have used basic jailbreaking prompts, often copied and pasted from online resources, rather than more advanced prompt attacks, and such jailbreaks were unsuccessful in attempts to perform tasks such as generating a Python-based distributed denial of service (DDoS) tool.
Research on phishing techniques and targets, as well as generation of phishing and influence operation-related content, were also common, with APT42 heavily using Gemini to generate, edit and translate phishing-related text tailored to specific targets such as U.S. defense organizations.
Post-compromise activity, such as detection evasion, privilege escalation, lateral movement and data extractions, was also included among many threat actor requests for coding assistance and research into operating system tools and vulnerabilities. Chinese state-sponsored actors were especially focused on gaining deeper access to victim networks, asking Gemini for ways to sign a Microsoft Outlook plugin that could be silently deployed to all computers on a network, for example.
“PRC-backed APT actors asked Gemini for assistance with Active Directory management commands and requested help troubleshooting impacket, a Python-based tool for working with network protocols. While impacket is commonly used for benign purposes, the context of the threat actor made it clear that the actor was using the tool for malicious purposes,” GTIG wrote.
Google’s efforts to disrupt threat actor use of its AI tools
GTIG leveraged its knowledge of international APTs to detect, attribute and disrupt malicious activity on the Gemini platform. The team wrote that sharing the findings of its investigation into this activity with the public, industry partners and law enforcement is part of its effort to combat and prevent similar misuse of LLMs and other AI tools.
Notably, Gemini’s safeguards successfully prevented it from complying with many malicious requests, including requests related to the exploitation and misuse of Google tools such as Gmail and the Chrome browser. Google said information it has learned from the adversarial activity detected on its platform is continuously used to strengthen the security of its AI models.
“In conduction with this research, DeepMind has shared how they’re actively deploying defenses within AI systems along with measurement and monitoring tools, one of which is a robust evaluation framework used to automatically red team an AI system’s vulnerability to indirect prompt injection attacks,” the GTIG blog post stated. “Our AI development and Trust & Safety teams also work closely with our threat intelligence, security, and modelling teams to stem misuse.”
