The Good, the Bad, and the Ugly: Hacking 3 CSPs with 1 Vulnerability

The Good, the Bad, and the Ugly: Hacking 3 CSPs with 1 Vulnerability

Speakers: Hillai Ben-Sasson; Andres Riancho

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=TkDsnzlPJAg

Overview

Andres Riancho and Hillai Ben-Sasson, security researchers at Wiz, presented research demonstrating how a single critical container escape vulnerability in NVIDIA Container Toolkit was used to compromise more than 10 different cloud service providers. The talk detailed three representative case studies -- Azure Container Instances, Replicate, and DigitalOcean (Paperspace) -- that illustrate the full spectrum of security outcomes: from no cross-tenant access (Azure), to limited data exposure with active detection (Replicate), to full service compromise with access to all tenants' models, secrets, and source code (DigitalOcean/Paperspace). The vulnerability required only the ability to control the container image provided to the service, making exploitation trivial across any GPU-accelerated cloud offering.

Background

▶ Watch: NVIDIA Container Toolkit: used in 90% of AI cloud services, critical containe... (2:00)

NVIDIA Container Toolkit is the runtime component that configures containers to access NVIDIA GPUs. It is ubiquitous in AI/ML cloud services -- the researchers estimate that 90% of AI services running on the cloud use NVIDIA Container Toolkit. The vulnerability discovered by the Wiz research team is a critical container escape that allows an attacker to mount the host file system inside the container, enabling read and write access to host files and interaction with host Unix sockets. The only prerequisite is controlling the container image (specifically, the Dockerfile) provided to the service.

This was described as a "dream vulnerability" for the research team. The exploit was extremely reliable, had no risk of crashing target services if it failed, and the affected component was deployed across virtually every AI cloud offering. This created a unique natural experiment: the same zero-day vulnerability across many different service providers, revealing how each vendor's security architecture and isolation controls performed against a real threat.

Key Findings

▶ Watch: Azure Container Instances: no cross-tenant access despite full host compromise (6:00)

Azure Container Instances (The Good). After successful exploitation and host file system access, the researchers found approximately 20 container images on the host, kubelet credentials, and host processes -- but no evidence of other tenants. The kubelet credentials were used to connect to the Kubernetes API (via a docker-sock-based container created in the host's network namespace), but the API returned only a single node. After five days of network scanning, DNS resolution, and traffic capture, the team concluded there was no cross-tenant access. The environment appeared to be purpose-built for the single tenant. Microsoft's isolation design held up completely.

Replicate (The Bad). Replicate is an AI SaaS platform where users upload models for community inference. The exploit required adaptation to Replicate's Cog format (a container packaging for ML models), which took approximately two to four hours. Post-exploitation revealed over 500 container images on the host, successful kubelet credential recovery and Kubernetes API access (no network restrictions), and host processes attributable to other tenants. Most critically, the researchers discovered a Redis task queue URL with embedded credentials in a host process's command-line arguments. This Redis queue carried all user prompts and model predictions for both public and private models, meaning the researchers could have read, modified, or deleted inference data for any model on the platform. Before they could explore the Kubernetes cluster further, Replicate's blue team detected their activity, disabled their account, and contacted them within five minutes. The researchers stopped and disclosed their findings.

DigitalOcean / Paperspace (The Ugly). Paperspace Deployments, a DigitalOcean service, required no exploit customization. Post-exploitation revealed 500+ container images, successful kubelet credential access, and unrestricted Kubernetes API connectivity. The critical difference: kubectl get nodes returned a cluster with 700+ nodes. Using the kubelet credentials, the researchers retrieved pod information including image names and image pull secrets for other tenants' private registries. This was repeatable for every pod in the cluster, constituting a full service compromise: access to all tenants' models, source code, secrets (Kubernetes secrets, environment variables, configuration files), and infrastructure. All three providers patched the underlying NVIDIA Container Toolkit vulnerability within less than two days of notification.

Technical Deep Dive

▶ Watch: Replicate detects the researchers and disables their account (14:00)

The NVIDIA Container Toolkit vulnerability enables host file system mounting from within a container. The attacker crafts a Dockerfile that triggers the vulnerability at container startup, causing the host's root filesystem to become accessible within the container. From this position, the standard post-exploitation methodology proceeds:

1. Host filesystem analysis: Enumerate container images via overlay filesystem inspection to gauge multi-tenancy density
2. Credential recovery: Locate kubelet credentials on the host filesystem
3. Kubernetes API access: Attempt to reach the kube-apiserver -- success depends on network policies
4. Network namespace escape: If direct API access is blocked (as in Azure), create a new container via docker.sock in the host's network namespace to inherit the host's network connectivity
5. Cluster enumeration: Use kubelet credentials to query the API for nodes, pods, secrets, and other cluster resources
6. Cross-tenant data access: Extract secrets, environment variables, image pull credentials, and data from other tenants' workloads

The divergent outcomes across the three providers trace directly to their security architecture decisions:

• Network policies: Azure blocked container-to-API-server connectivity; Replicate and Paperspace did not
• Cluster isolation: Azure dedicated infrastructure per tenant (single node); Replicate and Paperspace shared large multi-tenant clusters
• Kubelet permission scoping: Paperspace's kubelet credentials had excessive permissions that allowed reading secrets and pod specifications across the entire cluster
• Runtime detection: Replicate had active monitoring that detected the researchers' post-exploitation activity; Azure and Paperspace did not flag it

The researchers recommend three specific Kubernetes hardening measures: user namespaces (running pods with different UIDs than the host to limit container escape impact), network policies restricting pod-to-API-server connectivity, and principle of least privilege for all cluster identities including kubelets.

Demo / Proof of Concept

▶ Watch: DigitalOcean/Paperspace: full service compromise with 700+ node cluster access (16:00)

No live demo was performed during the presentation. The research was presented through slides detailing each exploitation step and its outcomes. The Wiz team has published a detailed technical blog post on the vulnerability at wiz.io/blog, with additional research content scheduled for release in the following months. The researchers referenced Wiz's PEACH framework for guidance on building secure multi-tenant Kubernetes architectures, and recommended considering microVMs (such as Firecracker) for stronger tenant isolation than containers alone can provide.

Defensive Implications

▶ Watch: Q&A: containers aren't strong isolation, consider microVMs like Firecracker (22:00)

For organizations building cloud services on Kubernetes with multi-tenant workloads:

• Implement user namespaces: Pods should run with different user IDs than the host to reduce the impact of container escapes
• Restrict pod-to-API-server network access: Application containers almost never need to talk to the Kubernetes API; only sidecars and system components should have this connectivity
• Apply least privilege to kubelet credentials: Kubelets should only have permissions for the nodes they manage, not cluster-wide access
• Consider microVMs (Firecracker): Containers alone do not provide strong enough isolation for multi-tenant workloads handling sensitive data
• Implement runtime detection: Replicate's ability to detect the researchers' activity and respond within five minutes demonstrates the value of monitoring even when preventive controls fail
• Separate tenants onto different nodes or clusters: Shared compute infrastructure creates lateral movement opportunities that stronger isolation models prevent

For organizations consuming cloud AI services:

• There is limited ability to assess a provider's internal isolation architecture; diversity of providers and periodic security assessments are the primary risk management tools
• The same provider may have vastly different security postures across different service offerings -- one offering may be well-isolated while another is not
• Patching velocity is a positive signal: all three providers in this study patched within two days

Key Takeaways

• A single NVIDIA Container Toolkit container escape vulnerability affected 90%+ of AI cloud services, creating a natural experiment in provider security architecture
• Azure Container Instances demonstrated best-in-class isolation with no cross-tenant access despite full host compromise
• Replicate had cross-tenant data exposure via a shared Redis queue but detected the researchers within five minutes -- demonstrating that detection matters even when prevention fails
• DigitalOcean/Paperspace suffered full service compromise: 700+ node cluster with access to all tenants' models, secrets, and source code
• The three critical differentiators were network policies, cluster isolation per tenant, and least-privilege kubelet permissions
• All providers patched within two days, but architectural weaknesses amplified the impact far beyond the initial container escape

About the Speaker(s)

Andres Riancho and Hillai Ben-Sasson are security researchers at Wiz specializing in cloud security research. They focus on identifying vulnerabilities in cloud service providers and widely-deployed open-source software including Kubernetes and NVIDIA Container Toolkit. Ben-Sasson was unable to attend in person; Riancho presented solo. Their research blog is at wiz.io/blog.