CUDA de Grâce: Owning AI Cloud Infrastructure with GPU exploits

CUDA de Grâce: Owning AI Cloud Infrastructure with GPU exploits

Speakers: Valentina Palmiotti (IBM X-Force Offensive Research (XOR)); Samuel Lovejoy (IBM X-Force Offensive Research (XOR))

Conference: Hexacon

YouTube: https://www.youtube.com/watch?v=Lvz2_ZHj3lo

Overview

In an era defined by the explosive growth of Artificial Intelligence (AI) and Machine Learning (ML), the underlying compute infrastructure, particularly Graphics Processing Units (GPUs), has become a critical, yet often overlooked, security frontier. This talk, "CUDA de Grâce," delivered by Valentina Palmiotti and Samuel Lovejoy of IBM X-Force Offensive Research (XOR), sheds light on a significant vulnerability in NVIDIA's proprietary CUDA driver stack. Their research demonstrates how a sophisticated kernel exploit targeting NVIDIA GPU drivers can lead to a complete compromise of AI cloud infrastructure, enabling container escape and cross-tenant attacks on multi-tenant serverless compute nodes.

The IBM X-Force Offensive Research team embarked on this project with several motivations: the undeniable expansion of AI and its reliance on GPUs, the introduction of a new AI category in the prestigious Pwn2Own competition, and the notable lack of deep binary exploitation research targeting NVIDIA's products despite their pervasive use. This presentation, marking the team's inaugural public appearance, not only details a complex race condition vulnerability and its exploitation but also exposes a fundamental security weakness in how many cloud providers isolate AI workloads. By achieving root access on a host system from within a compromised AI container, Palmiotti and Lovejoy underscore the severe implications for data confidentiality and integrity in shared AI cloud environments.

Background

▶ Watch: Introduction and motivation for GPU research (0:00)

The foundation of NVIDIA's AI/ML ecosystem is its CUDA parallel computing platform, which provides the compilers, libraries, and APIs necessary for GPU programming across languages like C, C++, Python, and Fortran. To simplify the deployment of GPU-accelerated workloads, NVIDIA developed the NVIDIA Container Toolkit. This toolkit seamlessly integrates with various container runtimes (Docker, CRIO, containerd, Podman) to ensure that containerized AI/ML applications have full access to GPU devices, drivers, and CUDA libraries. Beneath this container layer lies the host kernel and the NVIDIA GPU drivers, which directly interface with the GPU hardware.

The threat scenario investigated by the researchers, and particularly relevant for competitions like Pwn2Own, involves an AI/ML application compromise within a container. The objective is to escape this malicious container and achieve execution on the host system. Escaping via the kernel offers significant advantages: it bypasses the specific container runtime used and, crucially, circumvents container hardening measures. While containers can be configured with lower privileges, seccomp filters, or reduced capabilities to restrict access to notoriously buggy kernel subsystems (like netfilters or io_uring), GPU access is inherently required for AI/ML workloads. This means the NVIDIA GPU drivers are always reachable by a malicious container, making them an attractive target. This approach also avoids the limitation of prior research, such as that by Wiz, which required control over the container image itself. However, kernel exploits are notoriously complex, often unreliable, and demand substantial development effort to target multiple kernel versions and defeat various mitigations.

Initial code reconnaissance of the NVIDIA drivers revealed several red flags that suggested potential vulnerabilities. The researchers observed numerous entry points for user-mode interaction with the kernel, complex interactions across multiple subsystems that could lead to logic flaws or invalid state transitions, and, most critically, the use of mixed locking types and intricate concurrency mechanisms. Such complexity often introduces opportunities for developers to make mistakes, leading to race conditions or time-of-check, time-of-use (TOCTOU) bugs. Given a tight four-week deadline for Pwn2Own registration, the team swiftly set up a fuzzing campaign using syzkaller. Their setup involved a dedicated laptop with a GPU, utilizing GPU passthrough with VFIO on a KVM virtual machine for fuzzing, and Azure cloud instances with dedicated GPUs for crash analysis and exploit development.

To facilitate fuzzing, NVIDIA drivers were compiled with instrumentation (KUB, KSAN, KTSAN) against a custom Linux kernel. Syzkaller's "resources" and "pseudo syscalls" features were instrumental in modeling the drivers' stateful behavior and complex multi-step interactions. Despite these efforts, challenges arose: limited hardware (one GPU per fuzzing VM), the inherent statefulness of GPU drivers which syzkaller's resource model doesn't explicitly encode order for, and parasitic coverage issues with KCOV. Parasitic coverage, a known weakness where large loops or recursion overflow KCOV's buffer, led to syzkaller favoring these unproductive paths and wasting execution time. Due to the time constraints, these fuzzing challenges meant that several bugs were initially found manually while writing syzkaller definitions, which were then adapted to be found by the fuzzer. Ultimately, the Pwn2Own registration deadline was missed, but the team pivoted to focus on exploit development and proving impact, tailoring their exploit for Azure serverless compute.

Key Findings

▶ Watch: Pwn2Own threat model: container escape via kernel (4:00)

The central discovery of this research is a critical race condition vulnerability residing within the Nvidia_unlocked_ioctl function, specifically when handling the Nvidia_escape_attach_gpus_to_fd ioctl. This vulnerability results in a highly potent double free primitive within the NVIDIA GPU driver stack. The Nvidia_escape_attach_gpus_to_fd ioctl is designed to allow a file descriptor to hold references to GPU identifiers, passed as an array of 32-bit integers.

The vulnerability stems from a time-of-check, time-of-use (TOCTOU) race window. The ioctl first checks if the input array has a non-zero size, is a multiple of four bytes, and crucially, if no GPUs are already attached to the file descriptor. If this check passes, the driver allocates a buffer of a user-controlled size (arg_size) on the heap (using nvk_malloc) and copies user-supplied data into the nvlfp->attached_gpus variable. The problem arises because nvlfp (NVIDIA Linux File Private) is a heap-allocated structure per file descriptor, and Nvidia_unlocked_ioctl operates without a lock during this critical phase. This creates a tight race window: after the initial check passes but before the nvlfp->attached_gpus variable is fully assigned, a second concurrent call to the same ioctl can interfere.

In the failure case of the ioctl (e.g., if Nvidia_dev_get fails for any GPU ID), the driver iterates backward, releasing any references already grabbed, calls nvk_free on the attached_gpus variable, and resets the count of attached GPUs to zero. Critically, it does not reset the nvlfp->attached_gpus pointer back to NULL. This omission is the root cause of the double free: if a racing thread's initial nvk_free occurs, and then the first thread proceeds to free the same pointer after its own allocation, a double free occurs. The attacker gains control over the size of the double-freed object, ranging from 4 bytes to 16 kilobytes, making it an incredibly powerful primitive for heap manipulation. While a more complex reference count manipulation could also be achieved, the double free was chosen for its reliability.

To overcome kernel mitigations like CONFIG_SLAB_FREE_LIST_HARDENED, which would typically cause a kernel panic on a double free of the same pointer, the researchers leveraged the behavior of large kallocs. By allocating objects larger than 8192 bytes (specifically, within the 8KB to 16KB range), the kernel's kalloc function directly invokes the page allocator. Unlike kmalloc's virtual memory allocations, physical memory allocations from the page allocator are reference counted. A double free of such a page causes its reference count to drop from 1 to 0 on the first free, and then to -1 on the second. Instead of panicking, the kernel merely issues a warning message and continues execution, making the race condition safe to retry repeatedly until successful. This crucial insight transformed an otherwise unreliable primitive into a viable exploitation vector.

Technical Deep Dive

▶ Watch: Key advantages of kernel-level container escape (4:20)

The exploitation chain built upon the double-free primitive is a testament to sophisticated kernel exploitation techniques. The core challenge was reliably triggering and controlling the race condition. The researchers designed a multi-threaded attack involving four CPUs: a "fast CPU" to trigger the first free, a "slow CPU" for the second free, and two "spray CPUs" to allocate memory in the race window.

1. Race Condition Trigger: Both the fast and slow CPUs simultaneously invoke the vulnerable Nvidia_escape_attach_gpus_to_fd ioctl.
2. First Free: The fast CPU is engineered to exit the ioctl's internal loop early, performing its nvk_free on the nvlfp->attached_gpus variable.
3. Heap Spray and Reclaim: Upon the fast CPU returning to user space, the spray CPUs immediately begin allocating physical memory pages. The goal is to reclaim the memory freed by the fast CPU as an order zero page (4KB). This is critical because the target aggressor object, a pipe buffer, fits this size.
4. Second Free: The slow CPU eventually performs its nvk_free, resulting in the double free of the page allocator's memory.

Initial attempts at this race had a success rate below 1%. Two key issues were identified and addressed:

• Per-CPU Free List Contention: The order 2 page (8KB-16KB) freed by the NVIDIA driver could get stuck in the per-CPU free list of the allocating CPU. If this happened, no amount of spraying would reclaim it. The fix involved evicting/flushing the per-CPU free list on the spray CPUs before starting the spray, ensuring the page would be available in global zones for reclamation.
• Tight Race Window: On Azure instances, the race window was extremely narrow (around 20 microseconds). To extend this, a well-known technique by Project Zero's Jann Horn was employed: using numerous hardware timers (via timerfd_create) on the slow CPU. These timers fire during the race, significantly slowing down the slow CPU's execution and extending the race window to approximately 38 milliseconds – a 2000x increase in reliability.

With a controlled double free of a physical page, the next step was to establish a powerful arbitrary read/write primitive. The target object for reclamation was a pipe buffer.

• Pipe Buffer as Dangling Reference: The Linux pipe buffer data structure is highly suitable for kernel exploitation because its first field (struct page *page) holds a physical reference, and user space can easily read and write to this physical reference using read/write syscalls on the pipe. By spraying pipes, one of them ends up with its backing struct page having a reference count of zero due to the double free.
• Physical Memory Type Confusion: The double-freed page is then reclaimed as a kmalloc-192 slab. This creates a type confusion where the pipe buffer's page pointer now points to the same physical address as the kmalloc-192 slab, which is essentially an order zero page (4KB) filled with 192-byte objects.
• KASLR Bypass: To bypass Kernel Address Space Layout Randomization (KASLR), the researchers "moved" the double-freed pipe inside the kmalloc-192 slab itself. This slab was pre-filled with other pipe buffers during the heap spraying phase. By corrupting one of the adjacent pipe buffers on this slab using the double-freed pipe's out-of-bounds read/write capabilities, they could make the corrupted pipe point to the start of the kmalloc-192 slab. This effectively provided a known base address within the slab.
• Arbitrary Read/Write: With this setup, the corrupted pipe could then be used to re-point the original double-freed pipe's page pointer to any arbitrary address in kernel memory. Subsequent read and write syscalls on the double-freed pipe then translate into arbitrary kernel memory read/write operations.

The final objective, container escape and local privilege escalation, was achieved by corrupting the task_struct of the attacker's process. The task_struct holds all privilege and sandboxing information for a process. By copying the relevant privilege fields from the init process's task_struct (e.g., systemd) to the attacker's process, full root privileges and container escape are gained. This technique works across various Linux distributions and cloud environments.

A notable aspect of this exploit is its ability to bypass mitigation v4, a set of hardened kernel configurations often found in Kernel CTFs, specifically CONFIG_SLAB_VIRTUAL. CONFIG_SLAB_VIRTUAL prevents type confusion by assigning unique page table entries (PTEs) to memory from different slab caches, preventing memory from kmalloc-192 from overlapping with kmalloc-256, etc. However, this mitigation does not apply to large kallocs (those over 8192 bytes) because they directly interact with the page allocator. Since the double free primitive inherently relies on large kallocs to avoid kernel panics, the exploit effectively sidesteps CONFIG_SLAB_VIRTUAL. While this mitigation still introduces minor inconveniences for writing to kernel heap objects, the ability to read/write arbitrary physical memory and target low-entropy kernel image sections ensures that local privilege escalation and container escape remain fully achievable.

Demo / Proof of Concept

▶ Watch: Fuzzing campaign setup with syzkaller (7:00)

The talk included a compelling live demonstration of the exploit against an Azure serverless compute node. The scenario began with the researchers queuing a malicious training job within the Azure AI/ML workspace panel. This job contained code designed to obtain a reverse shell. The job was configured to run on a serverless GPU node, utilizing a standard container environment (TensorFlow was chosen, but the specific environment was irrelevant to the exploit).

Upon successful execution of the malicious training job, a reverse shell was obtained, confirming initial code execution within the container. A quick ps command showed the process was confined within the container's namespace. The NVIDIA GPU exploit was then executed from this containerized shell. Within moments, a second reverse shell materialized, this time granting root access directly onto the host compute node. The demonstration validated the full container escape and privilege escalation by showing that the attacker could now view all processes running on the host, including co-located training jobs initiated by other Azure users. This visually confirmed the ability to breach tenant isolation in a multi-tenant cloud environment.

Defensive Implications

▶ Watch: Compiling drivers for instrumentation and fuzzing stateful drivers (7:50)

The research presented in "CUDA de Grâce" carries profound defensive implications, particularly for cloud providers offering AI/ML as a service. The core takeaway is that OS-level containerization (namespaces and cgroups) provides an insufficient security boundary for multi-tenant AI workloads when a shared kernel driver, such as the NVIDIA GPU driver, is vulnerable.

1. Shared Kernel, Shared Risk: In cloud environments where multiple customers' AI workloads run on the same compute node, the underlying shared kernel and its drivers become a single point of failure. A single kernel vulnerability, like the double free discovered, can transform a container compromise into a cloud-wide breach.
2. Cross-Tenant Attacks: A successful container escape allows an attacker to gain kernel-level control over the host. From this privileged position, the attacker can:

• Steal sensitive data from co-located workloads, including proprietary model weights, training datasets, user data, and operational secrets like credentials.
• Manipulate models or data, potentially leading to supply chain attacks or data poisoning.

1. Inadequate Isolation in AI Cloud: Many GPU cloud providers, especially newer or less mature ones, rely on containerization for efficiency, faster startup times, and better GPU utilization. However, this convenience often comes at the cost of robust security isolation compared to stronger boundaries like hypervisors. Even mature cloud providers like Google Cloud Platform (GCP), which employs GVisor for enhanced container sandboxing, remain vulnerable because GVisor does not emulate NVIDIA GPU drivers, leaving them directly exposed to the host kernel and its vulnerabilities.
2. Critical Attack Surface: NVIDIA GPU drivers represent a massive and under-researched attack surface. Given the increasing reliance on GPUs for AI, security auditing and vulnerability research into these drivers are paramount.
3. Recommendations for Defenders:

• Stronger Isolation: Cloud providers should prioritize stronger isolation mechanisms, such as hardware virtualization (hypervisors), for multi-tenant AI workloads, rather than solely relying on OS-level containerization.
• Driver Auditing and Patching: Organizations deploying GPU-accelerated infrastructure must maintain rigorous patching policies for GPU drivers and encourage independent security audits of these complex codebases.
• Defense-in-Depth: Implement additional layers of security beyond containerization, such as network segmentation, strict access controls, and runtime monitoring, to detect and mitigate potential host compromises.
• Assume Breach: Design infrastructure with the assumption that a container escape is possible, and implement controls to limit the blast radius of a compromised host.

The research clearly demonstrates that the "explosion of GPU providers" has outpaced the maturity of their security architectures, making cross-tenant attacks a tangible and high-impact threat.

Key Takeaways

• NVIDIA GPU drivers are a critical and under-researched attack surface, especially given the rapid expansion of AI and its reliance on GPU cloud infrastructure.
• A sophisticated race condition in Nvidia_unlocked_ioctl leads to a powerful double-free vulnerability, allowing arbitrary kernel memory corruption.
• The exploit cleverly bypasses significant kernel mitigations, including CONFIG_SLAB_FREE_LIST_HARDENED (by leveraging page allocator's ref-counting behavior) and CONFIG_SLAB_VIRTUAL (by utilizing large kallocs).
• Advanced techniques like hardware timers for race window extension, physical memory type confusion with pipe buffers, and kmalloc-192 slab shaping were used to achieve a reliable arbitrary read/write primitive.
• Containerization, while efficient, provides insufficient security isolation for multi-tenant AI workloads when a shared kernel driver is vulnerable, enabling full container escape and cross-tenant attacks.
• Cloud providers offering AI/ML as a service must adopt stronger isolation (e.g., hypervisor-based) and rigorously audit critical kernel drivers to prevent cloud-wide breaches.

About the Speaker(s)

Valentina Palmiotti and Samuel Lovejoy are integral members of the IBM X-Force Offensive Research (XOR) team. This group specializes in vulnerability research and exploit development, focusing on discovering and weaponizing security flaws in critical software and hardware. "CUDA de Grâce" marked their inaugural public presentation as a team, showcasing their expertise in complex kernel exploitation and their commitment to advancing security research in emerging areas like AI infrastructure. Their work highlights the importance of proactive offensive security research in identifying and mitigating risks before they are exploited in the wild.