Tile-Based Deferred Rooting: When Your GPU Starts Rendering To Kernel Code Space!

Tile-Based Deferred Rooting: When Your GPU Starts Rendering To Kernel Code Space!

Speakers: Xingyu Jin (Security Researcher, Google), Martijn Bogaard (Senior Security Analyst, Riscure)

Conference: OffensiveCon

YouTube: https://www.youtube.com/watch?v=KGbGU_MK9hY

Overview

This talk, presented by Xingyu Jin and Martijn Bogaard, delves into a fascinating and unconventional GPU hardware vulnerability discovered during the development of the Google Pixel 10. The researchers, part of the Android threat team, uncovered a critical flaw in the Imagination Technologies PowerVR GPU (DXT generation) that allowed an untrusted application to overwrite Linux kernel code and achieve root privileges. The presentation highlights the intricate process of black-box reverse engineering GPU hardware behavior and the creative exploitation techniques required to leverage such a primitive against a modern Android kernel.

The significance of this research extends beyond a single device. It sheds light on the often-opaque world of GPU security, demonstrating that even low-level hardware design choices can introduce severe vulnerabilities. The collaboration between Google's Android threat team and Imagination Technologies led to a timely remediation of these issues ahead of the Pixel 10's release, underscoring the importance of proactive security research in the hardware supply chain. This talk serves as a compelling case study for offensive security engineers, revealing the depth of persistence and ingenuity required to exploit complex hardware-software interactions.

The vulnerability, ultimately tracked as CVE-2023-28509, is particularly notable for its unique nature. Unlike typical software bugs, this exploit manipulates GPU hardware directly, causing it to write arbitrary data to kernel memory. The journey from identifying a suspicious hardware behavior to developing a stable root exploit involved a multi-stage process of understanding undocumented GPU operations, crafting precise memory layouts, bypassing kernel mitigations like ASLR and SELinux, and ultimately demonstrating a full privilege escalation from an unprivileged Android application.

Background

▶ Watch: Introduction: Pixel 10 GPU & kernel root exploit (0:00)

To understand the vulnerability, it's essential to grasp the fundamental architecture of the PowerVR GPU driver stack and its rendering pipeline. An application interacts with the GPU through a user mode driver (e.g., OpenGL, OpenCL, Vulkan). This user mode stack communicates with the kernel driver via ioctl calls, dispatching commands to a central component called the bridge server. The bridge server contains various "bridges" for managing core GPU functionalities such as memory, cache, tracing, debugging, and hardware-specific operations like 3D rendering and computations. These bridges expose commands that user mode applications can invoke.

The 3D rendering process on the PowerVR GPU begins with creating a device memory context and subsequently allocating device memory heaps. These heaps represent specific ranges of GPU virtual memory, hardcoded in the driver for distinct purposes, such as a "general heap" for metadata, a "shadow code heap," or a "texture heap." Memory is mapped into these GPU virtual address spaces using PMRs (Physical Memory Resources), which unify various memory blocks. User mode applications have significant, though restricted, control over the GPU virtual memory layout, allowing them to place shader code, textures, geometry data, and GPU control structures.

A critical component in the rendering pipeline is the free list. The PowerVR GPU employs a tile-based deferred rendering (TBDR) architecture, which processes geometry data into "tiles" and stores it temporarily in a parameter buffer in system memory. The free list is the minimal unit for constructing these parameter buffers. Each free list manages a pool of kernel pages for storing geometry data. When a free list is created, it's associated with a free list state buffer, a special page containing metadata for managing these kernel pages. This state buffer is shared with the GPU hardware and is protected by the MMU, preventing normal user-mode access. It also undergoes extensive security checks to prevent tampering or aliasing.

Free list creation involves submitting a protected GPU memory handle and configuration parameters. These parameters, often determined by graphics libraries like Vulkan or OpenGL, dictate properties such as the maximum number of pages, initial allocated pages, and the "grow free list pages" value, which specifies how many new pages are allocated when an out-of-memory event occurs. An example given in the talk illustrates that if 50 kernel pages are requested, 40 might be for normal usage and 10 for a "ready page list" cache, used by the GPU firmware while the driver allocates more pages.

The free list state buffer contains vital metadata, including the free list base address (pointing to the memory containing the page frame numbers (PFNs)) and the stack pointer register. The stack pointer register indicates the first page to be consumed by the GPU hardware's parameter manager unit. Other fields include page status and MMU status, typically cleared to zero. The GPU firmware performs checks and sets up these registers, but crucially, the initial vulnerability stems from insufficient sanitization of the configuration parameters that influence these values.

Key Findings

▶ Watch: GPU virtual address space and device memory heaps (3:40)

The core vulnerability, CVE-2023-28509, was identified as a critical input sanitization failure in the Imagination Technologies GPU driver. Back in 2015, the driver did not adequately sanitize the input free list configuration arguments, specifically allowing an "extremely large growth parameter." This seemingly innocuous configuration value had a profound impact on the GPU's internal state.

The immediate consequence of an oversized growth parameter was a corrupted stack pointer register within the free list state buffer. Instead of pointing to a valid location within the protected free list memory, the stack pointer would end up with an "off-the-rail" value. This corruption caused the GPU hardware, when trying to find the first physical page to use, to access memory regions outside the protected GPU memory. Crucially, these out-of-bounds regions were often unprotected by the MMU, making them accessible to user-mode control.

The researchers theorized that a corrupted stack pointer could redirect the GPU hardware to access, for example, the GPU general heap area, which is user-controlled. By crafting fake page frame numbers (PFNs) in this accessible out-of-bounds memory, they could potentially ask the GPU hardware to write geometry data to arbitrary physical pages. Initial attempts to validate this theory involved providing a crafted 0xA0A0A0A0 PFN and expecting an immediate crash, but this didn't happen reliably. Screens might blink, but the system often recovered, indicating some GPU recovery mechanisms. After persistent experimentation, a first kernel panic was observed, albeit at a slightly different address than expected (0x81A0A0A0 instead of 0xA0A0A0A0), confirming the issue.

The next challenge was to understand what data the GPU hardware was writing to these arbitrary pages. This required a black-box approach, as no driver or firmware code controlled these writes; it was purely a hardware operation by the parameter manager unit. Through careful experimentation with different memory layouts and PFN arrangements in the crafted out-of-bounds stack:

• Placing two pages in the correct stack growth direction revealed that the first page often contained repeated 0x3C0x01 patterns, suspected to be proprietary flags or part of a page table. The second page contained geometry data.
• Setting both PFNs to the same physical page resulted in a mix of geometry data and the mysterious 0x3C0x01 patterns.
• Crafting three different PFNs showed the GPU writing 0x000008FD and 0x000008FD entries to the top page, suggesting the GPU was indeed constructing a page table.

However, this initial write primitive was highly constrained. Many bits were uncontrollable, and the resulting 4-byte writes would often form invalid ARM instructions (0x00000001 is undefined). This limited primitive made direct kernel code patching for a Control Flow Integrity (CFI)-protected system incredibly difficult, leading the researchers to temporarily abandon this line of attack before revisiting it with renewed determination.

Technical Deep Dive

▶ Watch: User mode control over GPU virtual memory layout (6:00)

The journey to a stable exploit involved several breakthroughs in understanding and refining the GPU's write primitive. Initially, the team observed that a specific memory layout (BBBBCCCD followed by a PFN) resulted in the GPU writing BBBBBBBBCCCCCCCCDDDDDDDD shifted by 12 bits and adding 0x01. This provided a new potential primitive, but it still suffered from uncontrolled bits, particularly the lowest 0x001 and the upper bits. The 0x0001 in the lowest bits was problematic because, in ARM instruction encoding, if the top 24 bits are zero and the lowest bit is one, it often results in an undefined instruction, making direct code injection difficult.

A crucial observation came from analyzing crash logs, which consistently printed the kernel page table base register address. This sparked the idea that the 0x001 pattern might be useful for manipulating page tables. Consulting the ARM reference manual revealed that block descriptors (used in Level 1 or Level 2 page tables) have their lowest two bits set to 01. This aligned perfectly with the 0x001 constraint of their primitive, provided they could also tolerate the 10 bits above being zero (which turned out to be permissive enough for low attributes). The goal became to overwrite a predictable kernel page table entry to remap kernel code to a controlled memory region. However, ASLR made finding predictable page table entries challenging, and the "writing blind" nature of the primitive complicated matters.

After further persistent experimentation, a more powerful primitive was discovered by manipulating the memory layout before the fake stack. By unmapping the memory segment directly preceding their crafted stack, they observed an "underflow" behavior where the GPU started dereferencing memory 128 bytes before the overflown stack. This allowed for a more controlled write. The final, most potent primitive was found by brute-forcing negative offsets, yielding an almost arbitrary 32-bit write at the beginning of any physical page. The only remaining uncontrollable bits were the lowest four bits (D1), which could be effectively managed. This "weird but powerful" primitive was now suitable for directly patching the Android Generic Kernel Image (GKI).

With this refined arbitrary write primitive, the next steps were to bypass kernel mitigations:

1. ASLR Bypass: The Pixel 10 at the time of discovery did not implement physical memory randomization, meaning kernel functions had predictable physical addresses. The researchers identified approximately 4,000 kernel functions starting at the beginning of a page. They targeted the schedule function, which is large and frequently calls printf. By patching the very first instruction of schedule (which points to a format string address via the X22 register), they could redirect X22 to point to a different, crafted format string. When an unprivileged application read /proc/self/sched, the schedule function would then print kernel string addresses, effectively leaking kernel ASLR offsets.

1. Arbitrary Write (Refined): To get a more general arbitrary write, they looked for syscalls that receive user input and produce output, ideally spanning multiple pages. The set_timer syscall was identified as a suitable target. By patching a single instruction within set_timer, they could turn it into a perfect arbitrary write primitive. This involved making the copy_from_user instruction (which normally copies user-supplied data to a kernel buffer) use a user-controlled pointer as the destination in kernel space. A key challenge was the Privileged Access Never (PAN) mitigation; if other processes called the patched set_timer with a user-space pointer, it would trigger a PANIC and reboot the device. The solution was to ensure no other process crashed during the brief exploitation window.

1. Arbitrary Read: Obtaining an arbitrary read was crucial for full exploitation. Socket objects are often targeted due to their many ioctl handlers. The first step was to leak the heap address of a created socket object. While /proc/net/udp could print this, the kernel_pointer_restrict mitigation usually zeroed out or hashed these addresses. By using the GPU write primitive to patch kernel_pointer_restrict to 0 in the kernel data section, and then overwriting the no_hash_pointer flag (also in kernel read-only data, which could be overwritten by GPU geometry data), they could force the kernel to print the full, unhashed heap address. An alternative, more stable approach involved overwriting the zip_hash key (also in the kernel data section) to 0. This made the hash function effectively useless, allowing them to brute-force the likely range of socket heap addresses without collisions. Once the socket object's heap address was known, they abused the io_control_v4 function (within /proc/net/udp output) by patching an instruction to make it print arbitrary kernel memory, thus achieving an arbitrary read primitive.

With ASLR bypass, arbitrary read, and arbitrary write, the exploit could then override credentials, disable SELinux, and flush the as_cache to gain full root privileges.

Demo / Proof of Concept

▶ Watch: Preparing a rendering job and the 'kick' command (6:40)

The live demonstration showcased the culmination of this complex research, illustrating the full privilege escalation chain on a target device. The exploit was orchestrated through two Android applications:

1. "Todd" Application: This initial application was responsible for leveraging the GPU hardware vulnerability. Its primary function was to patch specific instructions within the Android kernel. This included the initial schedule function patch for ASLR bypass and the set_timer syscall patch for the arbitrary write primitive. The speaker emphasized the "back and forth" nature of this phase, hinting at the precise timing and sequence required for the GPU operations.
2. Second Application: Once the critical kernel instructions were patched by the "Todd" app, the second application took over. This app performed the subsequent stages of the exploit:

• Brute-forcing hash: Likely referring to the process of finding the socket object heap address after nullifying the zip_hash key.
• Overwriting credentials: Elevating the privileges of the unprivileged application to root.
• Disabling SELinux: Bypassing the security enhancements that would otherwise restrict root's actions.
• Flushing as_cache: Ensuring that the kernel's internal caches reflected the new privilege state.

The demonstration successfully showed the transition from an unprivileged application ID to having root privileges. A key visual cue was the ability to open privileged drivers like /dev/aoc, which would normally be inaccessible to a standard user. The speaker also provided a live demonstration of the ASLR bypass, showing how reading /proc/self/sched (which typically prints scheduling information) now printed a sequence of kernel string addresses, confirming the successful format string vulnerability exploitation.

A humorous but critical point highlighted during the demo was the instability introduced by the set_timer syscall patch. The speaker cautioned that if any other process on the device crashed during the brief exploitation window, it would trigger a kernel panic and reboot the device due to the Privileged Access Never (PAN) mitigation. This anecdote underscored the fragility and precision required for such low-level exploits, even for a successful proof-of-concept.

Defensive Implications

▶ Watch: Offensive view: Tiling and deferred rendering pipeline (8:00)

The discovery and remediation of CVE-2023-28509 offer several crucial defensive implications for hardware manufacturers, operating system developers, and security researchers:

1. Rigorous Input Sanitization for Hardware Configuration: The root cause of this vulnerability was insufficient sanitization of GPU configuration parameters, specifically the "grow free list pages" value. This emphasizes the need for extremely strict validation of all user-supplied inputs that influence hardware control structures, even if they appear benign. Hardware-level interfaces are fertile ground for subtle bugs with severe consequences.
2. Comprehensive MMU Protection and Isolation: While the MMU protected the free list state buffer, the corrupted stack pointer allowed the GPU hardware to bypass this protection by accessing out-of-bounds memory. This highlights that MMU protections are only as strong as the integrity of the control structures they rely on. Robust isolation between trusted (kernel/firmware) and untrusted (user) memory regions must be maintained through all stages of hardware operation.
3. Black-Box Hardware Auditing: The talk vividly demonstrates the difficulty of understanding and exploiting GPU hardware behavior without source code or detailed specifications. Manufacturers should consider providing more transparency or tools for security researchers to audit hardware-level interactions, or invest in extensive internal black-box testing.
4. Strengthening Kernel Mitigations:

• Physical Memory Randomization: The absence of physical memory randomization on the Pixel 10 significantly aided the ASLR bypass. Implementing robust physical memory randomization across all devices is a critical step to complicate similar hardware-level exploits.
• Control Flow Integrity (CFI): While Android employs CFI, the primitive was eventually refined to allow patching of specific instructions, bypassing CFI by altering direct execution paths. This necessitates continuous research into CFI's robustness against novel primitive types.
• Privileged Access Never (PAN): PAN was effective in preventing accidental kernel memory access from user-controlled pointers but introduced an instability challenge for the exploit. Continued hardening of PAN and other memory access controls is important.

1. Careful Review of procfs and Syscalls: The exploit heavily relied on abusing procfs entries (e.g., /proc/self/sched, /proc/net/udp) and specific syscalls (set_timer) for information leaks and write primitives. These interfaces, often designed for debugging or system information, require thorough security review to prevent their misuse. Specifically, functions that print kernel addresses or data, especially those influenced by configuration flags like kernel_pointer_restrict or no_hash_pointer, need stringent controls.
2. Unconventional Threat Modeling: The "unconventional" nature of this vulnerability, stemming from GPU hardware behavior rather than a typical software bug, underscores the need for security teams to broaden their threat models. Focusing solely on common software vulnerabilities might leave critical hardware-software interaction points exposed.
3. Supply Chain Security and Collaboration: The successful pre-release remediation of this vulnerability on the Pixel 10 due to collaboration between Google and Imagination Technologies highlights the immense value of strong security partnerships across the hardware supply chain. Early detection and responsible disclosure are paramount.

Key Takeaways

• Unconventional GPU Vulnerability: The exploit leveraged a unique GPU hardware flaw (CVE-2023-28509) where improper input sanitization of free list configuration parameters led to a corrupted stack pointer register, causing the GPU to write to out-of-bounds kernel memory.
• Black-Box Reverse Engineering: Discovering the precise nature of the write primitive required extensive black-box analysis, observing undocumented GPU hardware behavior through trial and error, and meticulously analyzing memory dumps.
• Progressive Primitive Refinement: The exploit chain evolved from a highly constrained write with many uncontrollable bits to an almost arbitrary 32-bit write at the beginning of any physical page, achieved through clever memory unmapping and offset manipulation.
• Multi-Stage Kernel Exploit: A full root exploit was built by combining several techniques: ASLR bypass via a format string vulnerability in the schedule function (abusing /proc/self/sched), an arbitrary write primitive by patching the set_timer syscall, and an arbitrary read primitive by manipulating kernel_pointer_restrict and abusing /proc/net/udp entries.
• Mitigation Bypasses: The exploit demonstrated bypasses for ASLR (due to lack of physical memory randomization), and SELinux, highlighting that even modern kernel defenses can be circumvented by sufficiently powerful primitives.
• Importance of Collaboration: The vulnerability was fixed ahead of the Pixel 10 launch through close collaboration between Google's Android threat team and Imagination Technologies, emphasizing the critical role of security partnerships in the hardware ecosystem.

About the Speaker(s)

Xingyu Jin is a Security Researcher at Google. His work focuses on vulnerability research and exploitation, particularly in complex systems like GPUs and Android.

Martijn Bogaard is a Senior Security Analyst at Riscure. He is also part of the Android threat team at Google, where he conducts vulnerability research for the Pixel devices and the broader Android ecosystem. His expertise lies in uncovering and addressing security issues in critical hardware and software components.