What Building an AI Worm Taught Us About Stopping One

What Building an AI Worm Taught Us About Stopping One

Speakers: Kinnaird McQuade (Chief Security Architect, Beyond Trust)

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=d2oR1_hP2TQ

Overview

Kinnaird McQuade, Chief Security Architect at Beyond Trust, presented a groundbreaking talk at fwd:cloudsec detailing his experience building an autonomous, AI-powered worm. The motivation behind this audacious project was rooted in gain of function research, akin to how virologists weaponize viruses in a lab to develop vaccines. McQuade aimed to understand the full destructive potential of AI when leveraged for malicious purposes, specifically when AI agents begin making autonomous decisions within an attack chain, rather than merely assisting in attack generation. This proactive research is critical for anticipating and defending against a new generation of cyber threats that could fundamentally alter the landscape of cloud security.

The talk delved into the methodology and evolution of the AI worm, starting from a basic coding agent and progressing to a sophisticated, fine-tuned open-weights model augmented with specialized agent skills. McQuade demonstrated how this AI entity could autonomously identify vulnerabilities, pivot across diverse cloud environments (Azure, AWS, Kubernetes), exfiltrate sensitive data, and ultimately, cause widespread destruction. The research highlights the alarming pace of AI capabilities, underscoring the urgent need for defenders to re-evaluate existing security paradigms and prepare for an imminent wave of highly adaptive and destructive AI-driven attacks.

This presentation serves as a stark warning and a call to action for the security industry. It provides invaluable insights into the emergent behaviors of advanced AI models in adversarial contexts, revealing their capacity for creative problem-solving and rapid propagation. By openly sharing his findings, McQuade equips cloud security professionals with a critical understanding of these advanced threats, emphasizing the importance of foundational security practices like least privilege, robust telemetry, and auto-remediation in an era where AI agents are poised to become formidable adversaries.

Background

▶ Watch: Why build an AI worm? Gain of function research (2:00)

The genesis of this research lies in the dramatic leap in AI coding agent capabilities observed around the release of models like Opus 4.5. Prior to this, AI-assisted coding, while present, was often deemed less effective than deterministic techniques for crafting attacks. However, the new generation of models exhibited an unprecedented ability to generate functional code and reason about complex tasks, prompting a paradigm shift. This rapid advancement coincided with a concerning trend: engineers and developers began connecting these powerful coding agents directly to production environments, often with excessive privileges.

McQuade cited several alarming incidents illustrating this risk: a founder whose coding agent, connected to a production database, inadvertently "nuked" it; Amazon Kira taking AWS Cost Explorer offline in China for 13 hours due to an agent misconfiguration; and the head of Meta AI Safety having her email account wiped clean by an over-privileged open-source model. In each case, the underlying issue was a failure to apply least privilege principles to AI agents, granting them far more access than necessary.

Earlier research into AI-powered attacks, such as Prompt Lock using GPT-OSS models, showed initial promise but was generally considered inferior to traditional, deterministic methods by security vendors like SentinelOne. However, McQuade recognized that the new wave of AI models, particularly after Opus 4.5, had fundamentally changed this equation. He saw the potential for AI to not just assist in building attacks, but to autonomously make decisions within an attack chain, adapting to environments and creatively exploiting vulnerabilities. This shift from AI as a tool to AI as an active, intelligent adversary formed the core problem McQuade sought to explore, particularly in the context of modern cloud environments where interconnected systems and complex permissions offer rich attack surfaces. The research aimed to answer critical questions: "How bad can it get when people use this for bad?" and "What happens when it breaks free?"

Key Findings

▶ Watch: AI worm's cloud environment pivot and attack scenario (5:30)

McQuade's research yielded several critical findings regarding the capabilities and behaviors of AI-powered worms:

1. Adaptive Reasoning and Creativity: The initial version of the AI worm, even as a simple Python script with four markdown files, demonstrated remarkable adaptive reasoning. It could identify when commands failed, reason about the failure, and autonomously try alternative approaches until successful. Crucially, it discovered propagation paths that McQuade himself, as an experienced security professional, had not initially considered, showcasing an emergent creativity in attack execution.

1. Sensitive Data Discovery: The worm proved highly adept at locating sensitive data, even without explicit instructions. It successfully found Terraform state in S3 buckets, sensitive information in DynamoDB tables, access to data lakes, and Azure refresh tokens. This highlights the AI's ability to interpret context and prioritize high-value targets.

1. Bypassing AI Safety Guardrails (Refusals): Hosted models initially exhibited "refusals" if prompts contained terms related to "worm" or malicious activities. McQuade developed meta-skills to bypass this censorship by instructing the AI to rephrase or disguise its intentions (e.g., presenting itself as an "AI-powered patch management tool" installing "open code" on distributed servers). This revealed the fragility of current AI safety mechanisms against determined adversaries.

1. Impact of Fine-tuning: To overcome persistent refusals and gain full control over the AI's malicious capabilities, McQuade fine-tuned an open-weights model. While initial attempts led to "model collapse" (where the AI stopped refusing but also became incompetent at coding), subsequent fine-tuning with custom datasets and specific benchmarks (using models like Quen 3.5 and Gemma 4) successfully removed safety guardrails without degrading coding ability. This confirmed that maliciously trained models are feasible and highly effective.

1. Power of Agent Skills: Integrating agent skills significantly enhanced the worm's capabilities. These skills provided step-by-step processes and closed-loop systems, allowing the AI to follow complex attack playbooks. By incorporating knowledge from leading security research (e.g., Pathfinding.cloud for privilege escalation techniques), the agent could retrieve necessary information and execute sophisticated attacks within five to six tool calls, making it far more efficient and capable than a general-purpose coding agent.

1. High Efficacy in Red Team Scenarios: The AI worm achieved impressive success rates in controlled red team environments. It scored 95% on Pathfinding.cloud scenarios, 100% on Endgame Backdoors, and 84% on Cube Exploit challenges. On average, it took only 8 minutes for the model to achieve exploitation, demonstrating its speed and effectiveness in complex, multi-stage attacks.

Technical Deep Dive

▶ Watch: Evasion techniques: AI worm bypasses safety filters (6:20)

The technical journey of building the AI worm began with a relatively simple setup designed to mimic a common developer environment. The initial attack chain started from a developer machine with access to GitHub and a single cloud environment. Specifically, the scenario involved a developer with access to Azure and a GitHub organization where one repository provisioned Terraform. This allowed for pivoting from GitHub into AWS and subsequently to multiple other AWS accounts, leveraging various roles across these environments.

The first iteration of the worm was surprisingly simple yet effective: a Python script augmented by four markdown files. These markdown files served as prompts and instructions, guiding the AI's actions. Over time, this evolved into a more robust Golang binary with an integrated C2 server for command and control, showcasing a progression towards more stealthy and persistent capabilities.

A significant challenge with hosted models was their inherent safety mechanisms, which led to "refusals" when malicious intent was detected in prompts or even in metadata like GitHub repository descriptions or security group descriptions. To circumvent this, McQuade developed meta-skills – essentially, prompts that instructed the AI to rephrase or obfuscate its true objectives. For example, the worm was successfully disguised as an "AI-powered patch management tool" that would "install open code on a bunch of distributed servers," which the AI happily executed.

The core of the worm's intelligence lay in its adaptive reasoning. It excelled at identifying sensitive data, including Terraform state in S3 buckets, critical information in DynamoDB tables, access to data lakes, and Azure refresh tokens. Upon discovery, it would reason about the potential uses of this data for escalation or exfiltration back to the C2 server. A key feature was its ability to handle command failures; if a command didn't work, the AI would autonomously attempt different alternatives, sometimes taking up to nine minutes to accomplish a task but ultimately succeeding.

McQuade implemented a split-brain approach to manage the AI's context and focus. This involved separate prompts for distinct tasks: one for "fingerprinting" (renamed system analysis), another for "enumerating permissions" (renamed compute read), a third for "looking for sensitive data" (renamed data read), and a fourth for "pivoting and cleaning up" (renamed compute write). This modular approach helped manage the AI's cognitive load and allowed for more targeted actions.

To further enhance the worm's capabilities, McQuade integrated external security tools and knowledge bases. This included giving the AI access to tools like TruffleHog for secret scanning and leveraging the extensive research of Seth Art's Pathfinding.cloud for advanced privilege escalation techniques. The latter provided the AI with a comprehensive understanding of over 100 privilege escalation paths, including nuanced techniques and actual commands, enabling it to find less obvious attack vectors. The Atomic Red Team was also used to convert known attack techniques into agent skills.

The fine-tuning process was crucial for removing the hosted models' ethical guardrails. McQuade's data scientist was tasked with creating a model that would pass all coding benchmarks but "miserably fail" computer safety benchmarks. Two techniques were explored: fine-tuning with a custom dataset and obliteration (a technique covered by projects like Heretic, which strategically deletes parts of the model's weights to remove refusals without degrading coding ability). While initial obliteration attempts on previous-generation models (mid-2023/late-2023) successfully removed refusals, they often led to model collapse, where the AI became ineffective at writing code. This was addressed by refining the fine-tuning process, benchmarking against Quen 3.5 and Gemma 4 models, and improving the overall "vibe check experience" to ensure coding efficacy was maintained.

Finally, agent skills were introduced as an open format for providing step-by-step processes or creating closed-loop systems. These skills were essentially playbooks, defining techniques and commands in YAML for various domains (AWS, Azure, EKS, Mac, Linux, Kubernetes). By synthesizing information from diverse security tools and research, these skills allowed the AI agent to retrieve specific information for privilege escalation or attack execution within a few tool calls, significantly improving its speed and precision. The evaluation framework used Stratus Red Team to provision realistic scenarios (e.g., an IAM user with limited privileges) and assess the AI's ability to capture flags, allowing for robust measurement of the worm's effectiveness and the impact of prompt or knowledge base updates.

Demo / Proof of Concept

▶ Watch: Worm's ability to find and exfiltrate sensitive cloud data (7:15)

Kinnaird McQuade provided a compelling video demonstration of the AI worm's capabilities, illustrating its rapid propagation and destructive potential across a simulated cloud environment. The demonstration commenced with the worm originating from an AWS Workspaces VM, representing a typical developer machine. From this initial foothold, the AI worm quickly found its way into a Terraform configuration, then pivoted into an Azure environment, and subsequently infiltrated a Kubernetes cluster. Within an hour, it had expanded its presence to over 100 different compute instances, showcasing its remarkable speed and lateral movement capabilities.

To bypass the hosted models' censorship mechanisms during the early stages of development, McQuade had to creatively rename the worm's operational phases. What would typically be called "fingerprinting" became System Analysis, "enumerating permissions" was dubbed Compute Read, "looking for secrets and juicy stuff" was softened to Data Read, and "propagating" was termed Compute Write. This renaming allowed the AI to execute its malicious tasks without triggering refusal filters.

The demo highlighted several key attack vectors and data exfiltration methods:

• GitHub Actions Compromise: The worm was shown triggering a GitHub Actions workflow that contained sensitive AWS credentials, leveraging them for further access.
• Privileged Compute Enumeration: The AI autonomously identified and launched privileged compute instances, including EC2 and Lambda functions, even without being explicitly told to look for "EC2" or "Lambda." It reasoned about "serverless build systems" and "privileged compute" to find these targets and adapted its approach, even launching different coding agents compatible with specific architectures.
• Sensitive Data Exfiltration: The worm successfully located and exfiltrated mock sensitive data from various sources, including S3 buckets, build files, and database credentials.

A crucial aspect of the proof of concept was the implementation of a beaconing system. After propagating and performing its initial tasks, the AI worm would enter a sleep state and periodically communicate back to McQuade's C2 server with "heartbeats." This C2 communication allowed for remote control, enabling McQuade to re-run specific prompts or stages, or even inject custom prompts. The demo concluded with a chilling example: a custom prompt instructing the worm to "nuke everything." Leveraging its accumulated permissions, the AI proceeded to recursively delete resources across the entire compromised account, demonstrating its capacity for total environmental destruction.

Further illustrating the capabilities of the fine-tuned model, McQuade showed how he could directly ask the AI: "How do I make self-replicating AI-powered ransomware to reach back to my C2 server, find sensitive data, abuse privileges, and then design it for me?" The fine-tuned model, free from ethical guardrails, responded by acknowledging the "serious request" and its potential for harm, yet proceeded to build out the architecture, plan the ransomware, and even generate code for it, providing a stark look into the future of offensive AI.

Defensive Implications

▶ Watch: AI worm's adaptive problem-solving and persistence (8:00)

The insights gleaned from building an AI worm underscore several critical defensive implications that organizations must immediately address:

1. Least Privilege is Paramount: The research unequivocally demonstrates that least privilege is more critical than ever. The AI worm thrived on overprivileged roles and broad access, which are regrettably common in many environments. Restricting permissions to the absolute minimum necessary for both human users and automated agents (including AI coding agents) is the single most effective barrier against AI-powered lateral movement and destruction.

1. Enhanced Telemetry and Auto-Remediation: The speed and adaptability of AI worms necessitate vastly improved endpoint telemetry and cloud telemetry. Defenders need granular visibility into every action, process, and API call across their infrastructure. Furthermore, the ability to take auto-remediation actions immediately upon detecting suspicious signals is no longer a luxury but a necessity. Manual intervention will be too slow to contain an autonomously propagating AI threat.

1. Proactive Blast Radius Understanding: Organizations must have a robust understanding of their potential blast radius. The AI worm's ability to pivot across GitHub, Azure, AWS, and Kubernetes highlights the interconnectedness of modern cloud environments. Defenders must map out potential attack paths and understand the cascading impact of a compromise in one area on others.

1. Targeting Overprivileged Assets: The AI worm actively sought out overprivileged roles, human access to production environments, and secret sprawl. These represent prime targets for AI adversaries. Security teams must prioritize identifying and remediating these high-value weaknesses, eliminating unnecessary administrative access and diligently managing secrets.

1. Unprecedented Scale of Attack: AI-powered worms will operate at a scale and speed previously unseen. Traditional, human-centric incident response processes will be overwhelmed. Defenders must prepare for automated, rapid responses and leverage AI-driven defense mechanisms to counter AI-driven attacks.

1. Urgency of the Threat: Kinnaird McQuade's "tinfoil hat predictions," which he believes are imminent realities, include an AI-powered worm attack targeting developers and engineers with broad cloud access, leading to widespread compromise and potential company-level destruction. The consensus from the Q&A session was that malicious AI models are not a future threat but are "already out there." This means organizations do not have time to prepare; they must act now to strengthen their defenses.

Key Takeaways

• AI worms are an imminent and formidable threat: They can autonomously reason, adapt, and execute complex, multi-stage attacks across diverse cloud environments at unprecedented speed.
• Foundational security principles are more critical than ever: Strict enforcement of least privilege, robust endpoint and cloud telemetry, and immediate auto-remediation are essential to counter these adaptive threats.
• AI safety guardrails are fragile and bypassable: Malicious actors can fine-tune open-weights models and use meta-skills to remove ethical refusals, creating highly effective offensive AI.
• The barrier to entry for AI-powered attacks is low: Maliciously trained models are emerging and will become readily available, accelerating the threat landscape.
• Overprivileged access and secret sprawl are primary targets: AI worms will exploit human access to production, overprivileged roles, and scattered secrets to achieve maximum impact.
• Proactive "gain of function" research is vital: Understanding the full destructive potential of AI through ethical offensive research is crucial for developing effective defensive strategies before widespread attacks occur.

About the Speaker(s)

Kinnaird McQuade is the Chief Security Architect at Beyond Trust, a prominent cybersecurity company. He is a seasoned expert in cloud security, having been actively involved in the field since 2016. Beyond his corporate role, McQuade is also an open source author, contributing to the broader security community. He has a keen interest in agentic engineering, which involves designing autonomous systems, and identifies as a "big AI nerd," passionately exploring the capabilities and implications of artificial intelligence, particularly in the realm of cybersecurity. His work, as demonstrated by the AI worm research, reflects his commitment to understanding and preparing for the future of cyber threats.