The GCP Jenga Tower: Hacking Millions of Google's Servers With a Single Package (and more)

The GCP Jenga Tower: Hacking Millions of Google's Servers With a Single Package (and more)

Speakers: Liv Matan, Senior Security Researcher, Tenable

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=nZWpDeY9p6g

Overview

In this compelling Black Hat USA talk, Liv Matan, a Senior Security Researcher at Tenable and recognized as Microsoft's Most Valuable Researcher, unveiled two critical vulnerabilities impacting Google Cloud Platform (GCP): CloudImposer and Confused Function. Matan introduced the "Jenga" concept, illustrating how major cloud providers build services atop one another, inadvertently creating interconnected attack surfaces ripe for privilege escalation and novel vulnerabilities. The talk detailed how a specific instance of dependency confusion within GCP Composer, a managed Apache Airflow service, allowed for code execution on internal Google servers, potentially affecting millions of instances.

Matan's research highlights the amplified risk of supply chain attacks in cloud environments, where a single malicious package can have a blast radius far exceeding traditional on-premise deployments. The CloudImposer vulnerability demonstrated how an attacker could leverage Google's own internal package dependencies to run code on vast swathes of Google's infrastructure. Complementing this, Confused Function exposed a privilege escalation flaw in GCP Cloud Functions, allowing an attacker to gain extensive permissions within a target environment. This presentation serves as a stark reminder of the complex security challenges inherent in hyperscale cloud architectures and the critical need for vigilance in software supply chains.

The significance of this research lies not only in the discovery of specific vulnerabilities but also in the broader conceptual framework of the "Jenga Tower." By understanding how cloud services interlock, security professionals can better anticipate and defend against novel attack vectors that leverage these intricate dependencies. Matan's work underscores the continuous cat-and-mouse game between cloud providers and security researchers, pushing the boundaries of what's considered possible in cloud exploitation.

Background

▶ Watch: Introduction: Hacking millions of Google servers (0:00)

The foundation of Matan's research, particularly the CloudImposer vulnerability, is rooted in a well-established attack technique known as dependency confusion. First publicly documented by Alex Berson, dependency confusion exploits the way package managers resolve dependencies when an application or system relies on both private and public package registries. In essence, if a private package (e.g., an internal company library) shares the same name as a package available on a public registry (like Python's PyPI or Node.js's NPM), a package manager configured to search both sources might prioritize the public, higher-versioned package. An attacker can exploit this by uploading a malicious package with the same name and a higher version to the public registry, thereby tricking the victim's build or deployment process into installing the attacker's code instead of the legitimate private dependency.

This problem is particularly acute in cloud environments, where the scale and interconnectedness of services magnify the potential blast radius of such an attack. As Matan emphasized, a malicious package deployed on an on-premise server might affect only that single server, but the same package deployed to a cloud service could potentially impact millions of instances. This "on steroids" effect transforms what might be a localized issue into a widespread systemic risk. For instance, the Apache Airflow package in Python saw nearly 22 million downloads by June 2024, with almost 26 million in the preceding month. This immense usage highlights the potential for a single compromised dependency to affect a massive number of users and services, including managed cloud offerings like GCP Composer, which is Google's managed version of Apache Airflow.

A critical enabler for these vulnerabilities was Google's own documentation. Matan discovered that Google's official guides for services like GCP Cloud Functions, GCP Cloud App Engine, and GCP Composer explicitly advised users to install private dependencies using the pip install command with the --extra-index-url argument. This argument instructs pip to search additional package indexes beyond the primary one, including the public PyPI registry. This guidance, inadvertently, exposed numerous GCP customers to dependency confusion attacks. Further investigation revealed that Google's guidance itself stemmed from recommendations within the broader Python packaging user guide, specifically regarding hosting simple repositories. This chain of recommendations and misconfigurations created a widespread vulnerability landscape, extending the reach of dependency confusion far beyond individual developer errors. Matan reported this false guidance to both Google and Python, noting that the Python Enhancement Proposal (PEP 708), aimed at mitigating dependency confusion, was still an open issue as of February 2023, indicating the persistent nature of this threat.

Key Findings

▶ Watch: Cloud supply chain attacks: boosted blast radius (2:00)

Liv Matan's research unveiled two primary vulnerabilities, CloudImposer and Confused Function, both underpinned by a novel conceptual framework dubbed "Jenga." The Jenga concept posits that major cloud providers, by building services on top of each other, create intricate dependencies that can lead to unique privilege escalation paths and vulnerabilities. Just like a Jenga tower, removing or manipulating one block can have cascading effects across the entire structure.

The flagship vulnerability, CloudImposer, represents a unique and highly impactful instance of dependency confusion directly affecting Google's internal infrastructure. Matan discovered that GCP Composer, Google's managed Apache Airflow service, uses specific images bundled with PyPI packages during deployment and updates. Crucially, among these pre-installed dependencies was a Google internal package, Google Cloud Data Catalog Lineage Producer Client, which was missing from the public PyPI registry. Because the installation process for GCP Composer deployments utilized the pip install command with the --extra-index-url argument (as per Google's own recommended practices), it would search PyPI for this package. Matan realized that by uploading a malicious package with the identical name and a higher version to PyPI, an attacker could hijack the installation process of this internal Google dependency. This would result in the execution of arbitrary code on the internal Google servers responsible for provisioning and managing GCP Composer instances, potentially affecting "millions of servers," as claimed by Matan.

The second significant finding, Confused Function, is a novel privilege escalation vulnerability within GCP Cloud Functions, Google's serverless execution environment. This vulnerability allows an identity with seemingly limited permissions—specifically, the ability to update or create a function—to escalate its privileges to those of the default Cloud Build service account. The Cloud Build service account (often [PROJECT_NUMBER]@cloudbuild.gserviceaccount.com) is highly privileged, typically possessing extensive permissions across various GCP services, including storage, logging, Artifact Registry, and buckets. Matan demonstrated that by manipulating the deployment process of a Cloud Function, which internally relies on Cloud Build, an attacker could leverage the extra-index-url mechanism (again, often guided by Google's prior documentation) to inject malicious code that runs with the elevated permissions of the Cloud Build service account. This represents a significant lateral movement and privilege escalation path for attackers who might initially only have limited access to a GCP project.

Together, CloudImposer and Confused Function illustrate the profound security implications of interconnected cloud services and the subtle dangers embedded within software supply chains, especially when combined with common package management practices and potentially misleading documentation.

Technical Deep Dive

▶ Watch: Deep dive: Dependency confusion attack technique (2:50)

The technical underpinnings of both CloudImposer and Confused Function revolve around the intricate mechanics of Python package management and the specific configurations employed within Google Cloud Platform. At its core is the dependency confusion attack, which leverages the behavior of package managers like pip.

When a Python project specifies its dependencies, it typically lists them in a requirements.txt file. The pip install command is used to fetch and install these packages. Crucially, the --extra-index-url argument modifies pip's search behavior. Instead of solely relying on a designated private package registry, this argument instructs pip to also search one or more additional public indexes, such as PyPI, the Python Package Index. The vulnerability arises when a package with the same name exists in both a private registry and a public registry. Package managers, by default, often prioritize the package with the highest version number, regardless of its source.

CloudImposer: Hacking Google's Internal Services

The CloudImposer vulnerability specifically targeted GCP Composer, Google's managed Apache Airflow service. When a user deploys or updates a GCP Composer environment, Google provisions an underlying instance using a pre-configured image. This image includes a set of bundled PyPI packages, some of which are internal Google dependencies. Matan's critical discovery was identifying one such internal package, Google Cloud Data Catalog Lineage Producer Client, which was not present on the public PyPI registry.

The attack sequence for CloudImposer is as follows:

1. Identification of a Missing Internal Package: Matan systematically scanned the dependencies of GCP Composer images, looking for packages that were part of Google's internal ecosystem but were absent from PyPI. The Google Cloud Data Catalog Lineage Producer Client was the key finding.
2. Leveraging --extra-index-url: The installation process for GCP Composer deployments and updates, following Google's (now rectified) recommended practices, used pip install with --extra-index-url. This meant that when Google's internal systems attempted to install Google Cloud Data Catalog Lineage Producer Client, they would also query PyPI.
3. Malicious Package Upload: An attacker could then upload a malicious Python package to PyPI with the exact name Google Cloud Data Catalog Lineage Producer Client and a version number higher than any expected internal version.
4. Hijacked Installation: During the provisioning or update of any GCP Composer instance globally, Google's internal build systems, when resolving dependencies, would encounter the attacker's higher-versioned malicious package on PyPI. Due to the dependency confusion logic, pip would preferentially install the malicious package.
5. Code Execution on Google's Infrastructure: This effectively granted the attacker arbitrary code execution on the internal Google servers responsible for managing and deploying GCP Composer environments. Matan's initial observation of "requests... from internal Google servers" confirmed this unprecedented level of access, leading to the chilling realization that code could potentially run on "millions of servers." The speaker noted that Google was not responsible for blocking the malicious package; rather, PyPI itself detected and blocked the activity, suggesting advanced defense mechanisms or a partnership with Google to monitor for such exploits.

Confused Function: Privilege Escalation in GCP Cloud Functions

The second vulnerability, Confused Function, targets GCP Cloud Functions, a serverless platform where code runs in response to events. This vulnerability is a privilege escalation flaw, allowing an attacker with limited initial access to gain significantly higher permissions.

The technical flow for Confused Function is as follows:

1. Initial Access: An attacker needs an identity (user account or service account) with permissions to update or create GCP Cloud Functions within a project. This level of access is often granted for developers managing serverless applications.
2. Cloud Build Integration: When a GCP Cloud Function is deployed, Google Cloud internally uses Cloud Build to containerize and deploy the function's code. This Cloud Build process runs under the context of the default Cloud Build service account (e.g., [PROJECT_NUMBER]@cloudbuild.gserviceaccount.com).
3. Highly Privileged Service Account: The default Cloud Build service account is inherently powerful. It typically possesses a broad range of permissions, including storage.objects.get, logging.logEntries.create, artifactregistry.repositories.downloadArtifacts, cloudbuild.builds.get, and access to various Google Cloud Storage buckets. These permissions are necessary for Cloud Build to fetch source code, build containers, push images to Artifact Registry, and log its activities.
4. Exploiting Dependency Resolution: Similar to CloudImposer, the vulnerability leverages how Cloud Functions deployments (via Cloud Build) resolve dependencies. If a Cloud Function's requirements.txt specifies a dependency that is also an internal, non-public package, and the build process uses --extra-index-url (again, potentially influenced by past Google documentation), an attacker can inject a malicious package.
5. Privilege Escalation: By crafting a malicious package and uploading it to PyPI with a higher version, an attacker can trick the Cloud Build process into installing and executing their code. This malicious code would then run with the extensive permissions of the default Cloud Build service account, effectively escalating the attacker's privileges from merely managing functions to potentially controlling storage, logging, and other critical resources within the GCP project. This represents a significant security boundary bypass, turning a function deployment permission into a broad administrative capability.

Both vulnerabilities underscore the critical importance of secure supply chain practices, precise dependency resolution, and the principle of least privilege, especially in dynamic and interconnected cloud environments. The --extra-index-url argument, while seemingly innocuous, emerged as a central vector for these sophisticated attacks.

Demo / Proof of Concept

▶ Watch: Python's packaging guide caused Google's misguidance (6:10)

While the talk did not feature a live, real-time demonstration of the full exploit chain for CloudImposer, Liv Matan presented compelling evidence of its effectiveness and impact. The presentation began with a striking visual of "requests that you see here are from internal Google servers," indicating that Matan had successfully executed code within Google's internal infrastructure. This initial success led to the realization that code was running on "all of these servers" and could potentially extend to "millions of servers" managing GCP Composer instances.

Matan elaborated on the practical steps taken to achieve this, specifically for the CloudImposer vulnerability. The core of the proof of concept involved:

1. Identifying the Vulnerable Package: As detailed, Matan discovered the internal Google package Google Cloud Data Catalog Lineage Producer Client was a dependency for GCP Composer but was missing from PyPI.
2. Crafting a Malicious Package: A Python package was created with the same name, containing code designed to exfiltrate information or demonstrate control (e.g., making outbound network requests that could be logged by the researcher).
3. Uploading to PyPI: The malicious package was then uploaded to the public PyPI registry with a version number designed to be prioritized by pip over any hypothetical internal version.
4. Observing Execution: Upon the deployment or update of a GCP Composer instance (which would trigger the vulnerable dependency resolution process), Matan observed the malicious code executing on Google's internal infrastructure, as evidenced by the "requests from internal Google servers."

A particularly notable aspect of the demonstration was the swift and unexpected response from PyPI. Matan attempted to remove the malicious package and associated user account from PyPI after the proof of concept. However, instead of a clean deletion, PyPI immediately displayed a message indicating it was "down for maintenance or is having an outage." Matan clarified, "PyPI was not down for maintenance. PyPI actually blocked me, my user and also the package that I just uploaded in a matter of seconds." This suggests either a sophisticated automated defense mechanism within PyPI actively monitoring for dependency confusion exploits or a direct partnership between PyPI and Google to detect and neutralize such threats rapidly. This quick blocking action prevented the malicious package from remaining publicly available and widely exploited, underscoring the vigilance required in the open-source ecosystem.

For the Confused Function vulnerability, while a live demo was not shown, Matan described the attack vector clearly: an identity with create or update permissions for a GCP Cloud Function could deploy a function whose dependencies (maliciously crafted via dependency confusion) would execute with the elevated permissions of the default Cloud Build service account. The speaker showcased a slide detailing the extensive permissions of this service account, emphasizing the significant privilege escalation achieved.

Overall, the presentation provided sufficient detail and evidence to validate the existence and severity of both vulnerabilities, highlighting the critical impact of dependency confusion when applied to cloud supply chains.

Defensive Implications

▶ Watch: CloudImposer technical details: Scanning Composer images (7:40)

The vulnerabilities uncovered by Liv Matan, CloudImposer and Confused Function, carry significant defensive implications for both cloud providers and users of cloud services, particularly within GCP. Addressing these issues requires a multi-layered approach focusing on secure software supply chain practices, vigilant dependency management, and robust identity and access management.

For Cloud Providers like Google, the immediate implication is the need for rigorous scrutiny of internal dependency management. The CloudImposer vulnerability demonstrated a critical flaw in how internal Google services (specifically GCP Composer) resolved their own dependencies. Cloud providers must:

• Implement Strong Internal Dependency Management: Ensure that all internal packages used by managed services are either exclusively hosted in a private, authenticated registry or are explicitly "pinned" to specific versions, preventing any potential override from public registries.
• Proactively Scan for Shadow Dependencies: Continuously scan internal codebases and deployed service configurations for dependencies that might accidentally share names with public packages but are not intended to be public.
• Audit Build and Deployment Processes: Review all build and deployment pipelines for managed services to ensure they do not use --extra-index-url or similar arguments that could inadvertently introduce dependency confusion. Prioritize --index-url for explicit, trusted sources.
• Update Documentation and Best Practices: Google has already taken steps to update its documentation, removing the problematic guidance of using --extra-index-url. Cloud providers must ensure their official documentation consistently promotes secure dependency management practices.
• Collaborate with Public Registries: The swift action by PyPI in blocking Matan's malicious package highlights the importance of collaboration between cloud providers and public package registries to detect and mitigate supply chain attacks.

For GCP Users and other cloud consumers, the defensive implications are equally crucial:

• Avoid --extra-index-url: The most direct defense against dependency confusion is to avoid using the --extra-index-url argument in pip install or equivalent commands for other package managers (NPM, Maven, etc.). Instead, explicitly specify your private registry using --index-url and ensure it is the only source searched for private packages.
• Pin Package Versions: Always pin exact versions of dependencies in requirements.txt (e.g., package==1.2.3) rather than using flexible version specifiers (e.g., package>=1.2.3). This prevents a higher-versioned malicious package from being installed, even if it appears on a public registry. Tools like pip-tools can help manage pinned dependencies.
• Use Private Registries Exclusively: For internal or sensitive packages, host them in a dedicated, authenticated private package registry (like GCP Artifact Registry or GitHub Packages) and configure your build systems to only pull those packages from that private source.
• Implement Principle of Least Privilege for Service Accounts: For vulnerabilities like Confused Function, strictly adhere to the principle of least privilege for all service accounts, especially those involved in deployment processes (like the Cloud Build service account). Grant only the absolute minimum permissions required for a service account to perform its function. Regularly review and audit these permissions.
• Monitor for Anomalous Package Downloads: Implement monitoring solutions that alert on unusual or unexpected package downloads, especially those from public registries that might correspond to internal package names.
• Supply Chain Security Tools: Utilize software supply chain security tools that can scan dependencies for known vulnerabilities, identify potential dependency confusion risks, and verify package integrity.
• Understand Cloud Service Dependencies: Recognize that managed cloud services are complex and may have their own internal dependencies. While users cannot directly control these, understanding the "Jenga" concept helps in anticipating potential attack vectors and advocating for stronger security from cloud providers.
• Stay Informed: Keep up-to-date with security advisories and best practices from cloud providers and security researchers, as the landscape of cloud supply chain attacks is constantly evolving.

By adopting these defensive strategies, organizations can significantly reduce their exposure to sophisticated supply chain attacks like CloudImposer and Confused Function, thereby strengthening their overall cloud security posture.

Key Takeaways

• Cloud Supply Chain Attacks are Amplified: A single malicious package in the cloud can affect millions of servers, far exceeding the blast radius of traditional on-premise supply chain attacks.
• Dependency Confusion Remains a Potent Threat: The pip install --extra-index-url argument, and similar mechanisms in other package managers, can lead to critical vulnerabilities if internal package names conflict with public ones.
• "Jenga" Concept Reveals Interconnected Risks: Cloud providers' practice of building services on top of each other creates complex interdependencies that attackers can leverage for unique privilege escalation and novel vulnerabilities.
• Google's Internal Infrastructure Was Vulnerable: CloudImposer demonstrated that even Google's own managed services (GCP Composer) could be exploited via dependency confusion, leading to code execution on internal Google servers.
• Privilege Escalation in Cloud Functions: Confused Function showed how limited permissions to create/update GCP Cloud Functions could be escalated to the highly privileged default Cloud Build service account within a project.
• Proactive Defense is Crucial: Both cloud providers and users must adopt stringent supply chain security practices, including avoiding --extra-index-url, pinning package versions, and implementing the principle of least privilege, to mitigate these evolving threats.

About the Speaker(s)

Liv Matan is a Senior Security Researcher at Tenable, a leading cybersecurity company. He has been recognized as Microsoft's Most Valuable Researcher, a testament to his significant contributions to vulnerability discovery and security research. Matan specializes in hunting across major cloud providers, uncovering critical vulnerabilities and developing novel attack techniques. His past notable published research includes vulnerabilities named "Emoji Deploy" and "Flow Fixation." In this talk, he introduced "CloudImposer" and "Confused Function," further solidifying his reputation as a prominent figure in cloud security research.