Breaking Out of The AI Cage: Pwning AI Providers with NVIDIA Vulnerabilities

Breaking Out of The AI Cage: Pwning AI Providers with NVIDIA Vulnerabilities

Speakers: Hilai Ben Sasson and Andres Liancho, Security Researchers, Wiz Research

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=5RH0StmV7Eo

Reading Time: ~9 minutes

Type: Briefing

TL;DR

Wiz Research discovered a critical TOCTOU (time-of-check time-of-use) vulnerability in NVIDIA Container Toolkit that allows a malicious container image to mount the host filesystem — effectively breaking out of container isolation. The flaw, now patched as CVE-2024-0132, affected virtually every major cloud and SaaS provider running GPU workloads, enabling attackers with the ability to upload a container image to achieve full host filesystem access, cross-tenant data exposure, and lateral movement through internal cloud infrastructure.

Introduction

AI infrastructure has a single common denominator: NVIDIA GPUs. Every AI vendor, every cloud provider, every training and inference pipeline runs on the same hardware, managed by the same software stack. This homogeneity creates a security asymmetry that Wiz researchers Hilai Ben Sasson and Andres Liancho spent the past year exploiting — with results that affected the entire AI cloud ecosystem simultaneously.

Their target was not a specific cloud platform or LLM service, but rather the foundational layer that all of them depend on: NVIDIA Container Toolkit, the open-source runtime library responsible for giving GPU containers access to host GPU resources. Finding one vulnerability in this library meant finding a vulnerability in every service built on top of it — and that turned out to be exactly what happened.

What is NVIDIA Container Toolkit?

▶ Watch: Target Introduction (04:00)

NVIDIA Container Toolkit is a container runtime layer that sits between standard container runtimes (like Docker/runc) and the host GPU. When a container is launched with GPU support, the toolkit modifies the OCI spec, mounts necessary NVIDIA libraries and drivers into the container, and initializes GPU access before the container's entry point executes.

The key action is bind mounting: NVIDIA Container Toolkit mounts specific shared objects from the host's /usr/lib tree into the container's filesystem at /usr/lib. Among the paths involved is a "compat" directory inside the container, from which files are mounted into /usr/lib. The Wiz researchers identified that these container-to-container mounts — where both source and destination paths are inside the container — were the primitive that would eventually yield a full container escape.

The TOCTOU Vulnerability

▶ Watch: Vulnerability Discovery (12:01)

NVIDIA Container Toolkit includes a security check (do_path_resolve) that validates all symlinks before performing mounts, ensuring they don't resolve outside the container filesystem. A naive approach of placing a symlink pointing to / in the compat directory is blocked by this check.

The vulnerability arises from a classic time-of-check / time-of-use race condition in the mount sequence:

1. The toolkit lists files in the container's compat/ directory
2. It applies do_path_resolve to each file (the time-of-check)
3. It then mounts libnvidia6 — which itself modifies the filesystem structure
4. It then mounts libnvidia7 (the time-of-use)

The exploit uses a carefully crafted Dockerfile that creates both a libnvidia6/ and a libnvidia7/ entry as directories (not symlinks, so they pass the security check). A symlink inside usr/lib pointing back to the compat/ directory is also created. When the first mount executes, it causes the compat directory to be overridden. By the time the second mount (libnvidia7) executes, the security assertions made during the check phase no longer hold — and the mount resolves to the host filesystem.

"Your image is the exploit," Liancho noted. A single docker run with the malicious image is all that is required; no race condition timing, no memory addresses, no crashes. 100% reliability.

From within the resulting container, the host filesystem is accessible via a predictable path. Reading /etc/shadow from the host, accessing other containers' filesystems, and interacting with Unix sockets — including the Docker daemon socket at /var/run/docker.sock — all become possible immediately.

Real-World Case Study 1: Replicate

▶ Watch: Replicate Attack (24:03)

Replicate is a widely-used AI model hosting service used by organizations including OpenAI, Salesforce, ByteDance, xAI, and Google. It allows users to upload custom AI models packaged in Cog format — essentially a Docker container with specific Cog runtime code.

The Wiz researchers packaged their NVIDIA Container Toolkit exploit into a Cog model designed to act as a shell: prompts submitted to the model's inference endpoint were executed as arbitrary commands on the underlying host. After exploiting the toolkit to escape the container, they discovered a Redis database credential in the host's process list.

This Redis instance was previously known to the Wiz team from earlier Replicate research: it serves as the centralized task queue routing every prompt submitted by every user across every model on the platform. Access to this Redis instance meant access to all prompts, predictions, and the ability to redirect inference requests — a cross-tenant impact affecting not just their own model but all Replicate customers.

Replicate's security team detected the anomalous activity and reached out, at which point the researchers reported the full scope and halted testing. Wiz praised the vendor's detection and response capabilities.

Real-World Case Study 2: DigitalOcean

▶ Watch: DigitalOcean Attack (28:03)

DigitalOcean's GPU Droplets service allows customers to deploy GPU-enabled virtual machines. After exploiting the NVIDIA Container Toolkit vulnerability on a DigitalOcean GPU droplet, the researchers found Kubernetes credentials (/etc/kubernetes kubelets) on the underlying host — accessible through the exposed filesystem via the NVIDIA library path.

Using those credentials, they enumerated the cluster and found over 700 nodes — immediately establishing that this was a multi-tenant shared environment. Testing exclusively against their own secondary accounts, they confirmed that the kubelet credentials provided access to:

• All AI models running in the cluster
• All source code deployed to the environment
• Customer secrets — third-party credentials (AWS keys, API tokens, etc.) that customers had configured in their pods

The impact extends beyond DigitalOcean's own infrastructure: customer secrets exfiltrated from the cluster could allow an attacker to pivot into DigitalOcean customers' external environments, turning a GPU vulnerability into a supply chain attack on the customers of cloud AI services.

Responsible Disclosure and Patch Status

▶ Watch: Remediation Guidance (32:03)

All issues were reported to their respective vendors before public disclosure. NVIDIA assigned CVE-2024-0132 and fixed the vulnerability in NVIDIA Container Toolkit version 1.16.2. Operators running version 1.16.2 or later are not affected. Replicate and DigitalOcean both remediated their specific environment configurations.

The researchers also hinted at a second critical container escape vulnerability discovered in NVIDIA Container Toolkit months after this one, which has not yet been publicly disclosed.

Notable Quotes

"When we found one vulnerability in it, we were able to hack into pretty much every single cloud provider, SaaS provider — every sort of service that offers any AI cloud services was vulnerable to our attack." — Hilai Ben Sasson (00:00)

"Your image is the exploit." — Andres Liancho (16:02)

"This was a dream vulnerability. It was a single vulnerability affecting the entire cloud ecosystem. It was reliable to exploit — no memory addresses, no potential crashes, buffer overflows. If you have NVIDIA Container Toolkit running on the system, our exploit was going to work one hundred percent." — Hilai Ben Sasson (20:03)

"Containers and containerization should never act as a sole security barrier when separating between environments." — Hilai Ben Sasson (34:03)

Key Takeaways

• Patch NVIDIA Container Toolkit immediately. Organizations running GPU workloads must upgrade to version 1.16.2 or later. Any unpatched deployment is vulnerable to full container escape by any user who can supply a container image.
• Shared infrastructure multiplies blast radius. A vulnerability in a foundational library doesn't just affect one product; it simultaneously undermines every service built on that library. AI infrastructure's GPU monoculture makes this especially dangerous.
• Containers are not a security boundary. The attack required no special privileges, no race condition timing, and no memory corruption — just a crafted Dockerfile. Defense-in-depth must include virtualization-based isolation (like gVisor) where strong tenant separation is required.
• Validate shared AI infrastructure supply chains. AI services built on third-party inference runtimes inherit the security posture of those runtimes. Operators must treat foundational libraries like NVIDIA Container Toolkit as part of their attack surface and monitor for updates.
• The AI infrastructure attack surface is largely unexplored. NVIDIA publishes nearly 600 open-source repositories. This research examined one. The researchers have already found a second critical vulnerability in the same toolkit. The broader AI infrastructure layer warrants systematic security review.

No slides PDF was listed for this briefing. CVE-2024-0132 is patched in NVIDIA Container Toolkit v1.16.2.