Zapocalypse: Compromising every Zapier user through a Lambda memory leak

Zapocalypse: Compromising every Zapier user through a Lambda memory leak

Speakers: Yair Balilti, Security Researcher, Token Security

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=FEtJFekr_cs

Overview

This talk, titled "Zapocalypse: Compromising every Zapier user through a Lambda memory leak," details a critical vulnerability chain discovered in Zapier, a prominent AI-driven workflow automation platform. Presented by Yair Balilti, a Security Researcher at Token Security, the research outlines a sophisticated five-stage attack that allowed the team to achieve a full platform account takeover, theoretically compromising every authenticated Zapier user. The presentation highlights the dangers of insecure sandbox environments, the persistence of "deleted" data in memory, and the risks associated with misconfigured build processes and supply chain vulnerabilities in modern cloud-native applications. This research is particularly significant given Zapier's extensive user base and its role as an integration hub for over 8,000 applications, making a comprehensive compromise a high-impact event.

The core of the "Zapocalypse" attack chain began with a sandbox escape within Zapier's code execution environment, leading to the recovery of deleted AWS credentials from Lambda function memory. These credentials, despite being associated with an "allow nothing" role, permitted broad enumeration of Zapier's internal Elastic Container Registry (ECR) repositories. Further investigation, including a clever metadata-based scanning technique, led to the discovery of highly privileged Zapier Actions MCP API keys and, critically, a high-privilege npm publish token. This npm token, possessing write permissions across all Zapier packages and bypassing two-factor authentication, presented a direct path to a supply chain takeover and stored Cross-Site Scripting (XSS), enabling a full account takeover of any Zapier user. The talk serves as a stark reminder for organizations leveraging serverless functions and complex build pipelines to re-evaluate their security assumptions and defensive strategies.

Background

▶ Watch: Introduction to Zapocalypse and research agenda (0:46)

The motivation behind targeting Zapier stemmed from a strategic decision by Balilti's research team to focus on an AI-driven, workflow automation platform that was not a hyperscaler. The team observed a pattern of critical sandbox escape vulnerabilities in similar platforms, specifically citing four critical bugs discovered in N8N within a single year. This indicated a potential blind spot in security research for platforms offering user-controlled code execution, suggesting that Zapier, despite its significant market presence, might have similar underlying issues that had yet to be publicly scrutinized. Zapier fit the criteria perfectly: it's an AI company, not a hyperscaler (avoiding lengthy disclosure processes), a workflow automation platform (offering a wide attack surface), and had no public CVEs in the preceding year. Crucially, Zapier provides a "Code by Zapier" feature, allowing users to execute custom Python or JavaScript code within their automation workflows – the same primitive that had proven vulnerable in N8N. This direct code execution capability became the initial entry point for the "Zapocalypse" research.

The fundamental problem exploited by the researchers lies in the common but flawed assumption that sandbox environments are impenetrable and that deleting sensitive data from memory effectively eradicates it. Many platforms, including Zapier, attempt to isolate user code and remove credentials from the execution environment to mitigate risks. However, as demonstrated, these measures are often insufficient. The talk underscores that a sandbox escape should always be a default assumption for attackers, and that robust defensive layers must extend beyond the sandbox itself to include strict Identity and Access Management (IAM) scoping and network egress filtering. Furthermore, the Python del statement, commonly used to remove variables, only removes the reference from the Python dictionary, leaving the actual string bytes of the secret intact in the process memory, making them recoverable through memory forensics techniques. This crucial detail formed the foundation for the attack's second stage, allowing the recovery of seemingly "deleted" AWS credentials.

Key Findings

▶ Watch: Executing OS commands in Zapier's code block (4:40)

The "Zapocalypse" research uncovered a sophisticated, multi-stage attack chain that exposed critical vulnerabilities across Zapier's infrastructure:

1. Sandbox Escape and Memory Leak: The initial finding demonstrated a successful escape from Zapier's Python code block sandbox, allowing arbitrary OS command execution within the AWS Lambda environment. Critically, AWS credentials (access key ID, secret access key, session token) were recovered from the Lambda's heap memory, despite Zapier's explicit attempt to del them from environment variables. This highlighted the misconception that del truly removes data from memory.
2. Broad ECR Enumeration: The recovered AWS credentials, associated with an "allow nothing" IAM role, surprisingly granted ecr:DescribeRepositories permission. This allowed the researchers to enumerate over 1,000 internal Zapier Docker repositories, revealing the vast scope of Zapier's internal containerized applications.
3. ECR Image Pulling Bypass: Despite lacking the ecr:GetAuthorizationToken permission required for standard Docker login, the researchers successfully pulled container images by directly calling AWS ECR APIs (ecr:BatchGetImage and ecr:GetDownloadUrlForLayer) and reassembling the layers from S3. This granted "open source" access to Zapier's entire container image fleet.
4. Discovery of Zapier Actions MCP API Key: Through targeted metadata scanning of ECR image config.json files (specifically the history field), a hardcoded Zapier Actions MCP API key was found in a light-LLM repository. This key, belonging to the co-founder and CTO of Light LLM, allowed full access to the associated Zapier user's connected integrations, including the ability to send emails from their Gmail account.
5. Discovery of High-Privilege npm Publish Token: Further metadata scanning using the ARG Dockerfile feature revealed a high-privilege npm publish token. This token possessed write permissions for all Zapier-scoped packages (scope: all) and, critically, bypass_2fa: true. This meant the token holder could publish malicious code to core Zapier npm packages, including zapier-platform-core (with over 100,000 weekly downloads).
6. Supply Chain Takeover and Stored XSS Potential: The npm token's capabilities enabled a theoretical npm supply chain takeover. By injecting malicious postinstall scripts into Zapier's public npm packages, an attacker could execute arbitrary code on developer machines during package installation. Furthermore, by targeting frontend packages like the "Zapier Design System" (loaded on every authenticated Zapier page), a stored Cross-Site Scripting (XSS) vulnerability could be introduced, leading to a full platform account takeover for any authenticated Zapier user by stealing cookies or manipulating their sessions.

Technical Deep Dive

▶ Watch: Understanding the AWS Lambda memory leak mechanism (6:10)

The "Zapocalypse" attack chain was meticulously executed, leveraging several technical nuances and overlooked security assumptions.

The first stage, sandbox escape, commenced within Zapier's "Code by Zapier" block, which allows users to execute Python or JavaScript. The researchers initially tested the boundaries of this environment by attempting to run basic operating system commands. A simple os.system('env') command successfully printed all environment variables, immediately confirming that the code block was running on AWS Lambda. Key environment variables like AWS_LAMBDA_FUNCTION_NAME were visible. Further os.system('ls') commands revealed files in the working directory, including lambda_function.py, which was the Lambda handler. Upon reading this file, the researchers discovered that Zapier explicitly attempted to remove sensitive AWS credentials (access key ID, secret access key, session token) from the environment variables using the Python del function. A comment in the code indicated this was done to "avoid responding of thousand of annoying false positive security reports," rather than for security directly, as they assumed an "allow nothing" role would prevent access.

This led to the critical second stage: recovering deleted credentials from memory. The del function in Python, while removing a variable reference from a dictionary, does not immediately purge the actual string bytes from the process's memory heap. Recognizing this, the researchers crafted a Python memory scanner designed to run within the Zapier code block. This scanner searched the process's memory for specific regex patterns associated with AWS credentials (e.g., AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). The scanner successfully recovered the full AWS credentials, including the session token, directly from the Lambda's heap memory.

With the recovered credentials, the next step was validation and enumeration. A short Python script using the sts:GetCallerIdentity API call confirmed the credentials were valid and belonged to an IAM role named "allow nothing." Despite its name, this role proved to be surprisingly permissive. The researchers utilized enumerate-iam, an open-source tool, to brute-force all possible AWS API actions for the "allow nothing" role. The results revealed that ecr:DescribeRepositories was permitted. This allowed them to list over 1,000 internal Zapier repositories within their Elastic Container Registry (ECR), a private Docker registry service.

The challenge then shifted to pulling these ECR images. Standard Docker pulling requires ecr:GetAuthorizationToken, which the "allow nothing" role did not possess. This was a roadblock for a conventional approach. However, the researchers understood that Docker's convenience layer ultimately calls underlying AWS ECR APIs. They identified that ecr:BatchGetImage (to get image manifests) and ecr:GetDownloadUrlForLayer (to obtain pre-signed S3 URLs for individual layers) were permitted. A custom Python script was developed to directly call these ECR APIs, download each compressed layer from S3, and reassemble them into full Docker images. This effectively bypassed the lack of GetAuthorizationToken and granted "open source" access to Zapier's entire container image fleet.

Faced with over 1,000 repositories, a smart triage strategy was needed. Initial secret scanning on a single pulled image yielded no results. The researchers implemented two filters: name analysis (prioritizing repos with terms like AI, LLM, internal, prod, infra) and last push time (focusing on recently updated repos). This led them to light-LLM, an open-source AI gateway, which had been pushed just an hour before their discovery. Pulling and scanning this container revealed a hardcoded Zapier Actions MCP API key in plain text. The Zapier Actions MCP API is Zapier's Machine Comprehension Platform (MCP) compatible interface, used by AI agents to interact with Zapier user connections. This key, belonging to the co-founder and CTO of Light LLM, granted access to their entire Zapier connection graph, including Gmail, Drive, and CRM integrations.

Recognizing the single-user blast radius of the MCP key, the researchers sought a broader impact. They shifted their secret scanning strategy from full image file systems (gigabytes per repo) to image metadata (config.json, kilobytes per repo). Specifically, they targeted the history field within the config.json, which logs every Dockerfile command, including build-time arguments (ARG). The common assumption is that ARG variables are temporary and disappear after the build; however, they are permanently baked into the image history. Scanning the history field for ARG values across all repositories quickly revealed a high-privilege npm token. This token was validated against the npm registry API and found to possess write permissions for all scope Zapier packages and, critically, bypass_2fa: true. This meant the token holder could publish, modify, or delete any Zapier-maintained npm package without human intervention. The token maintained public packages like zapier-platform-core (over 100,000 weekly downloads) and zapier-platform-cli.

This discovery opened the door to a npm supply chain takeover. By publishing a new version of a critical Zapier npm package containing a malicious postinstall script, an attacker could execute arbitrary code on any developer's machine that installed or updated the package. Furthermore, by targeting frontend packages like the "Zapier Design System," which is loaded on every authenticated Zapier page, a stored Cross-Site Scripting (XSS) attack could be launched. This would allow an attacker to steal user cookies, modify Zaps, deploy MCP servers, and effectively take over any authenticated Zapier user's account and all their connected integrations.

Demo / Proof of Concept

▶ Watch: Successful recovery of deleted AWS credentials from memory (7:30)

The talk demonstrated several stages of the "Zapocalypse" attack chain with concrete proof-of-concept executions:

1. Sandbox Escape and Environment Variable Display: The initial PoC involved running os.system('env') within a Zapier code block. The output clearly showed the environment variables of the underlying AWS Lambda function, including AWS_LAMBDA_FUNCTION_NAME, confirming code execution outside the intended Python logic.
2. File System Enumeration: Following the environment variable display, os.system('ls') was executed, revealing the lambda_function.py file, further solidifying the sandbox escape and access to the container's file system.
3. Memory Credential Recovery: A Python script designed as a memory scanner was run inside the Zapier code block. This script successfully identified and extracted the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN from the Lambda's heap memory, proving that del does not erase data.
4. IAM Role Validation: A Python script utilizing the recovered credentials to call sts:GetCallerIdentity was shown, successfully returning the ARN of the "allow nothing" role, confirming the validity of the stolen credentials.
5. ECR Repository Enumeration: The enumerate-iam tool was demonstrated, showing its output confirming the ecr:DescribeRepositories permission for the "allow nothing" role. A subsequent Python script using ecr:DescribeRepositories successfully listed over 1,000 internal Zapier ECR repositories.
6. Direct ECR Image Pulling: A custom Python script that directly invoked the ecr:BatchGetImage and ecr:GetDownloadUrlForLayer APIs was presented. This script successfully downloaded a Zapier container image by fetching its layers from S3 and reassembling them, proving the bypass of standard Docker authentication.
7. MCP API Key Discovery and Usage: The config.json metadata of the light-LLM image was shown, highlighting the hardcoded MCP_API_KEY. The key was then used to call the zapier.check endpoint, which successfully identified the key owner as the co-founder and CTO of Light LLM.
8. Gmail Send Email Proof of Access: Using the identified MCP key, a call to the Gmail:send_email action was demonstrated. An email was successfully sent from the CTO's connected Gmail account directly to the researchers, with the body "Hello from Token Research," showcasing a direct and impactful proof of access.
9. NPM Token Discovery and Permissions: The config.json of another Zapier image was displayed, showing the npm_token argument baked into its history. The token was then validated against the npm registry API, and the response confirmed its write permissions, scope: all, and bypass_2fa: true capabilities.

While the final stages of the attack (npm supply chain takeover and stored XSS leading to full account takeover) were not executed due to ethical considerations, their theoretical feasibility was thoroughly explained and backed by the demonstrated capabilities of the discovered npm token and the architecture of Zapier's frontend package loading.

Defensive Implications

▶ Watch: Listing Zapier's internal ECR repositories with recovered credentials (9:10)

The "Zapocalypse" research provides critical insights and actionable recommendations for organizations, particularly those leveraging serverless functions, container registries, and software supply chains:

1. Assume Sandbox Escape: Defenders must operate under the assumption that an attacker will eventually escape any sandbox environment. Security should not solely rely on the sandbox's integrity. Instead, defensive layers must focus on blast radius containment through strict IAM scope and network egress filtering. This means limiting what an escaped process can access and communicate with, regardless of its perceived isolation.
2. "Delete" is Not Delete: The Python del statement (and similar constructs in other languages) only removes references, not the actual data from memory. Organizations should never store secrets in environment variables or other easily accessible memory locations if they intend to "delete" them later. Instead, use secure secret management solutions that inject secrets just-in-time and clear memory securely, or pass secrets as encrypted parameters.
3. Audit IAM Permissions: IAM role names can be misleading. An "allow nothing" role might still possess dangerous permissions. It is crucial to audit the resolved permissions on every assumed role to verify they precisely match the intended minimal access. Tools like enumerate-iam or AWS IAM Access Analyzer can help identify unintended permissions.
4. Secure Build-Time Secrets: Build-time variables (like Docker ARG) can persist in unexpected ways, such as in image history. Never pass secrets via ARG. Instead, use secret mount features in build tools (e.g., Docker BuildKit's --secret flag) that prevent secrets from being baked into the final image layers or metadata. Implement rigorous scanning of container images for hardcoded credentials in both the file system and metadata.
5. Scope and Verify NPM Tokens: NPM tokens (and similar package manager tokens) should be scoped to the absolute minimum necessary packages and permissions. Avoid granting write access across "all scopes" or bypass_2fa. Utilize tools like npm ci to ensure reproducible builds and verify package integrity. Regularly audit the permissions associated with all package manager tokens.
6. Audit Public Package Consumption: Organizations consuming public packages, especially those loaded directly on authenticated user pages (like design systems), must audit new versions of these packages before deployment. Implement automated checks or manual reviews to detect malicious changes that could introduce XSS or other vulnerabilities.
7. Outbound Monitoring: Implement robust outbound network monitoring and detection. Anomalous outbound traffic, such as large downloads from ECR or suspicious API calls, should trigger alerts. This could have flagged the ECR enumeration and image pulling long before the deeper compromise.

By adopting these defensive strategies, organizations can significantly reduce their attack surface and mitigate the risks posed by complex, multi-stage attack chains like Zapocalypse.

Key Takeaways

• Sandbox escapes are a default assumption: Design defenses with the expectation that attackers will break out of sandboxes. Focus on blast radius containment via strict IAM scope and network egress, not just sandbox integrity.
• "Delete" in Python is not deletion: The del statement only removes references, leaving sensitive data recoverable from memory. Avoid storing secrets in environment variables that are merely "deleted."
• IAM names lie: Always audit the resolved permissions of IAM roles to ensure they match their intended purpose; "allow nothing" roles can still grant dangerous access.
• Build-time arguments persist: Docker ARG values are permanently baked into image history. Use secure secret mount features for build-time secrets and scan images for hardcoded credentials in all layers and metadata.
• Scope NPM tokens tightly: NPM tokens require minimal permissions, scoped to specific packages, and must not bypass 2FA. Implement strict controls and auditing for all package manager tokens.
• Monitor outbound traffic and package integrity: Implement network layer detection for unusual outbound activity (e.g., ECR enumeration). Continuously audit new versions of packages, especially those loaded on authenticated origins, for malicious code.

About the Speaker(s)

Yair Balilti is a Security Researcher at Token Security. Token Security is a company dedicated to helping organizations secure and manage AI agents across their environments. In his free time, Yair enjoys traveling, playing football, and indulging in good burgers.