Push-Pull-Pwn: Hacking the Cloud through Container Registry Poisoning

Push-Pull-Pwn: Hacking the Cloud through Container Registry Poisoning

Speakers: Hillai (Security Researcher, Wiz), Nir (Security Researcher, Wiz)

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=kJzfNeQ7E0U

Overview

This talk, "Push-Pull-Pwn: Hacking the Cloud through Container Registry Poisoning," delivered by Hillai and Nir from Wiz Research, uncovers a critical and often under-discussed security risk: the container registry. Positioned as the central hub of modern cloud environments, container registries facilitate the lifecycle of container images, from build to deployment. The speakers demonstrate how these seemingly innocuous components, when misconfigured or vulnerable, can become a single point of failure, bridging otherwise isolated environments and leading to widespread compromise.

The core thesis of the presentation is that container registries frequently serve as the sole connective tissue between disparate environments—such as development, testing, staging, CI/CD, and production. This interconnectedness, if not properly secured, creates a potent attack surface. Through three compelling real-life case studies involving Hugging Face, SAP, and Azure Container Instances, Hillai and Nir illustrate how attackers can leverage vulnerabilities in these registries to achieve broad access, ranging from reading private applications and internal infrastructure to gaining full cluster administrative privileges and even accessing customer data.

The significance of this research lies in its practical demonstration of how a compromise in one part of the software supply chain, specifically the container registry, can cascade into a full-blown service takeover across an entire cloud provider or organization. The talk not only highlights these severe vulnerabilities but also provides actionable defensive strategies, emphasizing the importance of strong authentication, granular access controls, and secure image management practices to mitigate this prominent and lethal threat.

Background

▶ Watch: Introduction: Container registry as a central, vulnerable hub (0:37)

Container registries are fundamental to modern cloud-native architectures, serving as centralized libraries for storing and managing container images. In a typical workflow, a build environment (e.g., a CI/CD pipeline) pushes newly created container images to a registry, while a runtime environment (e.g., a Kubernetes cluster or a serverless function) pulls these images for deployment. This push-and-pull model underpins the agility and scalability of containerized applications.

However, as Hillai and Nir point out, this seemingly straightforward process often introduces significant security challenges. In many organizations, a single, centralized container registry is used across all environments: production, testing, development, CI/CD, staging, and even individual developer workstations. While the intention might be to streamline operations and provide a unified source for images, this practice inadvertently creates a critical chokepoint. Environments that are otherwise designed to be isolated and segmented—to prevent the spread of compromise—become interconnected through this one shared resource.

Prior work in cloud security has extensively focused on vulnerabilities within container images themselves, misconfigurations in Kubernetes clusters, or weaknesses in CI/CD pipelines. However, the container registry, often treated as a passive storage component, has been comparatively under-discussed as an active attack vector capable of bridging these isolated segments. The problem is exacerbated by default configurations that may lack robust authentication or granular authorization, turning the registry into a high-value target for attackers seeking to pivot from a limited initial foothold to broader cloud infrastructure compromise. This talk delves into why this problem exists and how it manifests in real-world scenarios, demonstrating that the registry's role as an "isolation breaker" is a critical blind spot in many cloud security postures.

Key Findings

▶ Watch: Case Study 1: Hacking Hugging Face introduction (2:32)

The talk's key findings revolve around the critical role of container registries as central points of connection within cloud environments and how their compromise can lead to widespread impact. Through three distinct, real-life case studies on Hugging Face, SAP, and Azure Container Instances, Hillai and Nir consistently demonstrated that vulnerabilities or misconfigurations in these registries could be leveraged to escalate privileges, access sensitive data, and even achieve full service takeovers.

In the Hugging Face case, the researchers discovered that arbitrary code execution during the image build process (rather than deployment) granted root privileges within the CI/CD environment. This led to the discovery of an internal, unauthenticated container registry, allowing read and write access to all private applications and the ability to overwrite existing ones. This highlighted how a simple oversight in a build pipeline could expose the entire registry.

The SAP case study showcased a more complex chain of vulnerabilities. Initial code execution within an isolated AI workflow engine was circumvented by bypassing an Istio service mesh firewall using a specific user ID (1337). This internal network access revealed an exposed Helm Tiller server, which, lacking default authentication, provided credentials to SAP's centralized Docker and Artifactory registries. Critically, these credentials were highly privileged (admin@SAP), granting access to SAP's internal infrastructure and, more alarmingly, to customer-specific Docker image pull secrets, enabling a potential spread to customer registries. The ability to install arbitrary Helm packages also led to full Kubernetes cluster admin privileges.

Finally, the Azure Container Instances (ACI) investigation uncovered a critical isolation bypass. Starting with root access in an isolated GPU container, the researchers discovered unauthenticated Kubelet ports on the underlying node. This allowed access to the node's network namespace and, subsequently, to a Kubernetes API server proxy. By exploiting a path traversal vulnerability using Windows-style backslashes (\) in a .NET-based proxy, they bypassed tenant isolation, gaining read/write access to all pods and secrets across the shared cluster. This primitive then led to the compromise of Azure's internal registry, exposing proprietary Microsoft IP and enabling the overwriting of system images. Since ACI for GPU pulled unsigned images by tag, this meant a full service takeover was possible, affecting all tenants.

Across all three cases, the common thread was the container registry's role as a high-impact target. Whether due to unauthenticated access, exposed credentials, or isolation bypasses, compromising the registry consistently provided a pathway to broader control, sensitive data, and cross-tenant impact, underscoring its pivotal position in cloud security.

Technical Deep Dive

▶ Watch: Privilege escalation to root in Hugging Face build environment (3:54)

The technical deep dive of the talk illustrates three distinct attack paths, each culminating in the compromise of a central container registry and subsequent broader impact.

Hugging Face: Unauthenticated Internal Registry

The initial foothold in Hugging Face involved understanding their application hosting service. Users can provide a Dockerfile, and Hugging Face will build and run it. The critical distinction identified was between the CMD command, executed during container deployment, and the RUN command, executed during the container build process. When a Dockerfile with a CMD id was deployed, the user was 1000 Ubuntu, indicating low privileges. However, when the RUN id command was used, the execution occurred in the CI/CD build environment, revealing execution as root. This immediate escalation to root in the build environment provided a powerful primitive.

From this root-privileged build environment, the researchers investigated network connections to identify where the built image was pushed. They discovered a hostname: Docker builder registry. A simple GET request to this registry (/v2/_catalog) unexpectedly returned a 200 OK response, listing all hosted containers, without any authentication. This unauthenticated access was not limited to reading; it also granted write access. This meant an attacker could:

1. Read all private applications hosted on Hugging Face, bypassing standard authentication mechanisms.
2. Overwrite existing applications by pushing malicious code to their respective image tags, potentially affecting all users of those applications.

The vulnerability stemmed from the build environment having direct, unauthenticated network access to the internal container registry, a critical misconfiguration that broke the isolation between the build process and the registry's security boundary.

SAP: Istio Bypass and Helm Tiller Compromise

The SAP case focused on their AI Core service, which allows users to run custom AI training procedures by providing a configuration file that specifies a command to run within a container. Similar to Hugging Face, code execution was a feature, not a bug. Initial attempts to escalate privileges (e.g., setting uid=0 or enabling privileged containers) were blocked by SAP's hardened container configuration. Network scanning was also blocked by what appeared to be a firewall.

Further investigation revealed the "firewall" was actually Istio, a service mesh, configured with IP tables rules. A crucial detail was found in these rules: "take all the traffic going in and out except traffic coming from the user ID 1337." This exemption is a known Istio feature, allowing its internal components to bypass its own traffic redirection rules. By changing their container's user ID to 1337, the researchers successfully bypassed Istio's network restrictions, gaining unrestricted access to SAP's internal network.

Inside the internal network, they discovered an exposed Helm Tiller server. Helm, a package manager for Kubernetes, had an API that, by default, did not enforce authentication. Querying the Tiller server (tiller read) immediately revealed highly sensitive credentials for Docker container registries and Artifactory registries, with usernames like admin@SAP and admin2. These credentials granted full administrative access to SAP's centralized registries.

Beyond registry access, the Tiller server also allowed the tiller install command, enabling the deployment of arbitrary Helm packages. This was leveraged to install a package granting cluster admin privileges within the Kubernetes cluster. The kubectl can-i command confirmed wide-ranging privileges, effectively leading to a full-blown service takeover. Crucially, access to SAP's registries also meant access to customer Docker image pull secrets, enabling potential lateral movement from SAP's infrastructure to their customers' private registries.

Azure Container Instances (ACI): Kubelet and API Server Proxy Bypass

The final case study targeted Azure Container Instances (ACI), specifically its GPU-enabled offering. Here, users could deploy containers with root access, again, as a feature. The challenge was breaking out of the isolated container. The researchers knew that ACI containers run on dedicated pods within dedicated nodes, sharing a Kubernetes cluster with other customer nodes.

Initial attempts to access the Kubernetes API directly from the container were blocked. However, scanning the network for Kubernetes-specific ports revealed two open Kubelet ports: a read-only port and, critically, a read-write port. The Kubelet agent, running on every Kubernetes node, manages pods and containers. The read-write port, usually protected by authentication, was found to be unauthenticated in this ACI configuration. This allowed the researchers to execute commands on any pod on the node and, more importantly, access the networking namespace of the node itself, thus gaining access to the Kubernetes API.

When enumerating the Kubernetes API, responses mentioned a Kestrel server, which is a .NET web server. This was unusual as Kubernetes is not written in .NET. It was discovered that Microsoft had implemented an API server proxy in .NET, sitting between the customer container and the actual Kubernetes API. This proxy was responsible for enforcing tenant isolation.

The researchers attempted a standard path traversal attack (e.g., ../) but it failed. Recognizing Kestrel's .NET origins, they experimented with Windows path separators (\). By switching forward slashes (/) to backslashes (\) in their Kubernetes API requests, they successfully bypassed the API server proxy's isolation, gaining access to all pods and secrets in the shared cluster—including those belonging to other customers and Azure's own infrastructure.

This access revealed that all infrastructure pods pulled images from the Azure registry. Leveraging their newfound primitive, they extracted credentials to this internal registry, gaining read/write access to proprietary Microsoft IP in the form of container images, binaries, and source code. Critically, they could overwrite any system image in the cluster. Since ACI for GPU was observed to pull unsigned images by tag, any overwritten image would be deployed to all ACI for GPU tenants, leading to a full-blown service takeover across the entire service. This vulnerability led Microsoft to retire the ACI for GPU service shortly after remediation.

Demo / Proof of Concept

▶ Watch: Discovery of unauthenticated internal container registry (5:03)

The entire talk functions as a detailed demonstration and proof-of-concept for the "Push-Pull-Pwn" attack methodology, broken down into three distinct real-world scenarios. Each case study meticulously walks through the steps an attacker would take, from initial code execution to full system compromise, highlighting the critical role of the container registry in bridging isolated environments.

For Hugging Face, the demonstration began by showcasing how a user-provided Dockerfile could execute commands with root privileges in the build environment by using the RUN instruction, as opposed to the CMD instruction which ran with lower privileges during deployment. The subsequent PoC involved sniffing network connections from this privileged build environment to identify the internal Docker builder registry. A simple HTTP GET request to this registry's catalog endpoint (/v2/_catalog) successfully returned a 200 OK response with a list of all containers, demonstrating unauthenticated read access. The presentation then explained that this access extended to write permissions, allowing for the overwrite of existing applications or the insertion of malicious code into private applications hosted on Hugging Face.

In the SAP case, the PoC started with legitimate code execution within the SAP AI Core service, initially as a nobody user. The core demonstration here was the Istio bypass. By changing the container's user ID to 1337, the researchers showed how network requests that were previously blocked by Istio's firewall rules suddenly succeeded, granting unrestricted access to SAP's internal network. This internal access then led to the discovery of an unauthenticated Helm Tiller server. The PoC involved issuing a tiller read command, which immediately dumped highly privileged credentials for SAP's centralized Docker and Artifactory registries. Further, the ability to execute tiller install was demonstrated by explaining how it could be used to install a package granting Kubernetes cluster admin privileges, confirmed by kubectl can-i output. The final impact shown was the ability to access customer-specific Docker image pull secrets, demonstrating the potential for lateral movement to customer environments.

The Azure Container Instances (ACI) PoC began with obtaining root access inside a user's GPU container. The critical step was the discovery of unauthenticated Kubelet read-write ports on the underlying Kubernetes node through a network scan. This allowed for execution of commands on other pods and, crucially, access to the node's network namespace. The subsequent demonstration involved interacting with the Kubernetes API server proxy. While a standard path traversal (../) failed, the PoC successfully bypassed the proxy's tenant isolation by using Windows path separators (\) in API requests. This granted full read/write access to all pods and secrets in the shared cluster. The ultimate impact demonstrated was the extraction of credentials to Azure's internal registry, which contained proprietary Microsoft IP, and the ability to overwrite system images. The fact that ACI for GPU pulled unsigned images by tag meant this PoC could lead to a complete service takeover across all tenants.

Each of these scenarios served as a compelling, step-by-step proof of concept, illustrating the severity and practicality of container registry poisoning attacks.

Defensive Implications

▶ Watch: Impact: Full read/write access to Hugging Face applications (5:59)

The findings from "Push-Pull-Pwn" carry significant defensive implications for organizations operating in cloud environments. The researchers highlighted that many of these critical vulnerabilities could have been mitigated by adhering to fundamental security principles.

Firstly, the talk strongly emphasizes the importance of image signing and pulling images by digest rather than by tag. In the Azure ACI case, the ability to achieve a full service takeover hinged on the fact that ACI for GPU pulled unsigned images by tag. If all three case studies had enforced image signing, where only cryptographically verified images are accepted, and pulled images by their immutable digest (e.g., sha256:…) instead of mutable tags (e.g., latest), the attacks involving image overwriting would have been completely mitigated. This is a robust defense-in-depth principle that prevents an attacker from injecting malicious code into the supply chain by replacing legitimate images with compromised ones.

Secondly, scoping credentials per task boundary is critical. This means moving away from a single, shared "master password" or highly privileged set of credentials for the container registry. Instead:

• Production and Development Environments should only possess pull-only credentials. These credentials should be further scoped to access only the specific images absolutely required by that environment.
• Build Environments should primarily have push-only credentials. These credentials should be restricted to pushing new images to designated repositories and should ideally not have pull access to other environments' images, preventing a build environment compromise from leading to broader registry enumeration or image tampering.

This segmentation ensures that a compromise in one environment (e.g., a dev workstation or a CI/CD pipeline) does not automatically grant wide-ranging access to the entire registry.

To implement such fine-grained access control, organizations must leverage Role-Based Access Control (RBAC) features offered by flagship container registries. The speakers specifically mentioned:

• Harbor projects: For granular permissions within the Harbor registry.
• ACR scope maps: For Azure Container Registry, allowing precise definition of permissions.
• ECR policies: For Amazon Elastic Container Registry, enabling detailed access policies.
• Most other container registries offer similar capabilities to define who can push, pull, delete, or manage images, and for which repositories.

These RBAC features enable organizations to enforce the principle of least privilege, ensuring that each service, user, or environment only has the minimum necessary permissions to perform its intended function. This significantly reduces the blast radius of any potential compromise.

Finally, the talk implicitly underscores the importance of network segmentation and authentication for internal services. The unauthenticated internal registry in Hugging Face, the Istio bypass in SAP, and the unauthenticated Kubelet ports in Azure all point to weaknesses in internal network hygiene and default security configurations. Internal services, even those presumed to be protected by network boundaries, should always require strong authentication and authorization.

The researchers concluded by acknowledging the close collaboration with the security teams of Hugging Face, SAP, and MSRC (Microsoft Security Response Center) in remediating these issues. Notably, Microsoft decided to retire the ACI for GPU service shortly after the research, underscoring the severity of the findings.

Key Takeaways

• Container Registries are Centralized Attack Surfaces: Registries act as the single point connecting otherwise isolated cloud environments (dev, test, prod, CI/CD), making them high-value targets.
• Default Configurations Can Be Lethal: Unauthenticated access to internal registries (Hugging Face) or unauthenticated internal services (SAP's Helm Tiller, Azure's Kubelet) can grant attackers broad control.
• Bypassing Isolation is Achievable: Sophisticated techniques like the Istio user ID bypass (SAP) or API server proxy path traversal using Windows separators (Azure) can break tenant isolation.
• Enforce Image Signing and Pull by Digest: Always sign container images and configure environments to pull images using their immutable digest, not mutable tags, to prevent supply chain poisoning.
• Implement Granular RBAC for Registry Access: Scope credentials strictly: pull-only for runtime environments (prod/dev), push-only for build environments. Utilize registry-specific RBAC features (e.g., ACR scope maps, ECR policies).
• Internal Services Require Authentication: Even internal network services should enforce strong authentication and authorization, as network segmentation alone is insufficient defense against determined attackers.

About the Speaker(s)

Hillai Ben Sasson and Nir Ohana are security researchers based in Tel Aviv, Israel, and are both integral members of the Wiz Research team. Their work focuses on cutting-edge cloud security research, identifying novel attack vectors and vulnerabilities within cloud environments. Their expertise lies in uncovering critical weaknesses in cloud infrastructure and services, as demonstrated by their detailed analysis and exploitation of container registry vulnerabilities across major cloud platforms and popular services.