Build a Fake Phone, Find Real Bugs: Qualcomm GPU Emulation and Fuzzing with LibAFL QEMU

Build a Fake Phone, Find Real Bugs: Qualcomm GPU Emulation and Fuzzing with LibAFL QEMU

Speakers: Romain Malmain

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=kRzF2LXSLgk

Overview

This talk, presented by Romain Malmain at 39C3, delves into the intricate process of building a comprehensive emulated Android phone environment, specifically targeting the Qualcomm GPU kernel driver for security research. Malmain, a PhD student at EURECOM who undertook this work during a three-month internship at Qualcomm, highlights the journey from traditional, resource-intensive on-device fuzzing to a more scalable and efficient emulation-based approach. The core objective was to create a virtual testing ground that accurately mimics the complex interactions between the Android kernel and Qualcomm's proprietary GPU hardware, enabling more effective vulnerability discovery.

The significance of this research lies in its potential to revolutionize security testing for embedded systems, particularly within the Android ecosystem. By demonstrating the feasibility of emulating highly complex, vendor-specific hardware components like GPUs, the project paves the way for faster iteration cycles, lower costs, and enhanced debugging capabilities in fuzzing efforts. This shift from physical device farms to software-defined emulation environments represents a critical advancement in proactive vulnerability assessment for a platform that underpins billions of devices worldwide.

The talk not only details the technical hurdles encountered and overcome during the emulation development but also showcases tangible results, including the discovery of a real-world vulnerability in the Qualcomm GPU driver itself. This outcome underscores the dual benefit of such an endeavor: not only does it establish a robust fuzzing platform, but the very act of meticulously understanding and reimplementing hardware interfaces can expose underlying design flaws and security weaknesses.

Background

▶ Watch: Talk Goal: GPU Fuzzing with Emulated Android Phone (0:00)

The Android ecosystem presents an enormous and ever-growing attack surface, characterized by its deep complexity. With an estimated two million lines of code (and likely more), it encompasses multiple layers, from the kernel and C libraries to user applications, bootloaders, boot ROMs, baseband components, and trusted execution environments. This intricate interplay of components creates a fertile ground for vulnerabilities, as evidenced by the consistent reporting of security issues across various chip manufacturers.

Fuzzing is a widely adopted technique for discovering software bugs by feeding programs with large amounts of semi-random data. The fundamental fuzzing loop involves a fuzzer generating an input, a target function executing that input, and the fuzzer receiving coverage feedback (e.g., which code paths were executed) to guide subsequent mutations of the input. This iterative process aims to explore as many program paths as possible, maximizing the chances of triggering crashes or unexpected behavior.

Traditionally, security testing for Android devices often relies on on-device fuzzing. This involves setting up "fuzzing farms" of physical phones, where targets are deployed and continuously fed inputs. While effective, this approach suffers from significant limitations:

• Scalability and Cost: Each fuzzing unit requires a dedicated physical device, leading to substantial hardware costs, especially for large-scale operations or frequent hardware updates.
• Debugging Challenges: When a crash occurs, debugging on a physical device is often cumbersome, requiring reboots and manual intervention, which significantly slows down the bug discovery and analysis workflow.
• Statefulness: The inherent statefulness and side effects of real hardware can introduce non-determinism, making bug reproduction and root cause analysis difficult.

To overcome these challenges, the concept of emulation emerged as a promising alternative. Emulation aims to replicate the behavior of a physical device entirely in software, allowing for faster execution, easier debugging, and greater scalability. However, emulating complex hardware like modern smartphones, with their myriad of devices (Wi-Fi, baseband, GPU, etc.) and intricate concurrency mechanisms, is a monumental task.

Fortunately, prior work has laid some groundwork. Projects like Goldfish and its successor, Cuttlefish, have focused on enabling Android virtualization and emulation. Cuttlefish, in particular, is designed to work with a generic emulator like QEMU without requiring extensive forks or patches. It acts as a driver for QEMU, managing disk layers and generating the complex command-line arguments needed to configure a virtual machine. Cuttlefish leverages the arm virt board, a simple virtual board for AArch64 systems, providing a basic set of components like a Cortex-A57 CPU, RAM, UART, and an RTC device. This arm virt board defines a specific address space that the Android VM uses to interact with its virtualized components. This existing infrastructure provides a starting point for more specialized hardware emulation.

Key Findings

▶ Watch: Fuzzing 101: How Fuzzers Work (2:40)

The central achievement of this research is the successful emulation and integration of Qualcomm's proprietary GPU kernel driver within a custom QEMU-based Android environment. This represents a significant leap from theoretical possibilities to practical implementation, demonstrating that highly complex, vendor-specific hardware can be effectively virtualized for security purposes.

A particularly noteworthy finding, and one that underscores the value of the emulation process itself, was the discovery of a real-world vulnerability in the Qualcomm GPU driver. This CVE, slated for public disclosure in January, was identified not primarily through automated fuzzing, but during the meticulous reverse engineering and implementation of the GPU's interaction mechanisms within QEMU. The act of understanding the kernel driver's expectations and mirroring them in the emulator exposed discrepancies and logic flaws in the original code. This highlights a crucial insight: deep technical understanding required for emulation can serve as a powerful manual audit, yielding critical bugs even before fuzzing commences.

Furthermore, the project successfully integrated this emulated Snapdragon environment with LibAFL QEMU, a custom fork of QEMU optimized for fuzzing. This integration allows for the entire fuzzing loop—from input generation to execution and coverage feedback—to occur within a single process, dramatically increasing efficiency and reducing overhead compared to traditional multi-process fuzzing setups. The ability to run the Qualcomm GPU driver in an emulated, fuzzer-aware environment provides a scalable and reproducible platform for continuous security testing, a significant improvement over cumbersome on-device methods.

Technical Deep Dive

▶ Watch: Challenges of On-Device Fuzzing: Scalability and Debugging (4:05)

The ambitious goal of this project was twofold: first, to get the Qualcomm GPU kernel driver running correctly within a QEMU-based Android virtual machine, and second, to integrate this setup with LibAFL QEMU for efficient fuzzing. This journey involved overcoming numerous technical challenges, primarily related to bridging the gap between a generic virtual board and specific, proprietary hardware.

The foundational challenge stemmed from the need to reconcile the arm virt board (the generic QEMU virtual platform used by Cuttlefish) with the specific hardware expectations of a Snapdragon device. The Device Tree (DTB) plays a crucial role here. A DTB is a binary blob that describes the hardware present on a system, including devices, their memory ranges, interrupt lines, and interconnections. The arm virt board has its own DTB, while the Qualcomm kernel expects a DTB reflecting the Snapdragon's architecture. A direct collision of address spaces and device configurations would occur.

To resolve this, the approach involved a clever merging strategy. The arm virt DTB, which defines the generic QEMU environment, was combined with the much larger and more complex Snapdragon DTB. This wasn't a simple concatenation; it required relocating addresses recursively within the Snapdragon DTB to fit into free address space provided by the QEMU Android board. The boot process then involves QEMU copying the combined DTB into the VM's RAM. The bootloader, aware of the DTB's location (e.g., at address 0x04...), parses it and passes its address to the kernel via register X0, allowing the Linux kernel to also parse the hardware description and configure itself accordingly.

Getting the Android kernel, specifically one compiled for Snapdragon, to boot in this custom QEMU environment necessitated several kernel configuration tweaks. These included:

• Disabling the hypervisor where it wasn't required for GPU operation.
• Adjusting KSLR (Kernel Address Space Layout Randomization) settings, primarily for debugging purposes.
• Removing unnecessary code and SELinux init patches from the Android init process, as these were largely irrelevant for security testing in a controlled environment.

The kernel build process itself proved to be a significant hurdle. Android's build system is notorious for its complexity, often involving four interleaved and sometimes incompatible build systems. Each kernel build could take up to four hours, making iterative debugging and development a time-consuming affair, requiring powerful machines and considerable patience.

The core of the technical deep dive involved implementing the necessary Snapdragon devices within QEMU. The goal was not 100% feature completeness for every device, but rather to implement just enough functionality to satisfy the GPU kernel driver's expectations and allow it to initialize and operate. A generic sysfs abstraction was created in QEMU to automatically populate device information (like MMIO addresses and interrupt lines) directly from the DTB, saving significant development time.

During the initial VM boot attempts, common problems included:

• Infinite loops: Devices expecting specific return values that the emulator wasn't providing.
• Incompatible devices: Mismatches between the emulated hardware and the DTB configuration.
• Kernel module deferral (the "deferral mess"): This was identified as the most challenging problem. Kernel modules often have dependencies, and if a required device or module isn't ready, the kernel will return -EPROBE_DEFER, indicating it should be rescheduled later. The difficulty arises because these deferrals can be silent or cascade through a dependency chain, making it hard to pinpoint the root cause of an uninitialized module. Effectively, understanding and meticulously implementing the dependency graph of the critical devices was key.

By filtering the complex Snapdragon device tree, the critical components for GPU emulation were identified: special memory, RPMH (power management), the GPU itself, and GPU power levels (which connect to RPMH). The speaker implemented approximately 10,000 lines of code for these devices:

• RPMH (Remote Power Management Host): This device handles power management requests. It operates via an asynchronous protocol using interrupts. The guest kernel writes a message, triggers the RPMH device, which processes it, and then an interrupt signals completion. The asynchronous nature made debugging particularly challenging.
• SMMU (System Memory Management Unit) / IOMMU: This component is critical for DMA (Direct Memory Access) operations, mapping the GPU's device virtual addresses (DVAs) to the VM's physical address space. The speaker simplified the kernel's SMMU code, replacing it with a basic IOMMU implementation that sets up mappings based on physical addresses, DVAs, page sizes, counts, permissions, and VMIDs, followed by a "commit" operation to activate these mappings.
• HFI (Host-Firmware Interface): This is the core communication protocol between the kernel module and the emulated GPU, relying heavily on DMA. It's both asynchronous and bidirectional, involving a shared Q-table and two queues: one for sending messages and another for acknowledgments. Correct implementation of this DMA mechanism was paramount for any GPU interaction.

The final virtual machine setup, while visually complex, represents a functional emulation of these critical components, allowing the GPU kernel driver to load and execute its internal tests, as demonstrated by the dmesg output.

Finally, integrating with LibAFL QEMU involved specific optimizations. kcov, Linux's kernel coverage tool, was modified because its default output (listing instruction pointers) wasn't directly compatible with LibAFL's expected coverage format. It was adapted to provide the more standard block-level coverage, which also proved faster. Additionally, to improve performance, virtual memory allocations in the kernel were replaced with physical memory allocations, enabling faster memcpy operations and reducing memory fragmentation overhead during fuzzing.

Demo / Proof of Concept

▶ Watch: Cuttlefish: Android Virtualization without QEMU Forks (7:00)

The talk included a compelling demonstration of the emulated environment's functionality, specifically showcasing the successful boot of the custom Android kernel and the initialization of the Qualcomm GPU kernel driver. The speaker presented a snippet of the dmesg output from the running virtual machine.

This dmesg log clearly illustrated the sequential boot process of Linux, with various init lines indicating the loading and execution of kernel modules. Crucially, after the initial system modules, the log showed the KGSL (Kernel Graphics Subsystem Layer), which corresponds to the Qualcomm GPU kernel driver, actively performing its internal tests. The speaker noted that "even if there are some errors like it's most likely working," indicating that the driver was interacting with the emulated hardware as expected, despite potential minor discrepancies that might appear as errors in a real-world scenario.

This visual proof point served as validation that the extensive work in emulating the necessary devices (RPMH, SMMU, HFI, etc.) and configuring the kernel had paid off. The ability to observe the KGSL driver's activity within the emulated environment confirmed that a critical, proprietary component was now running in a software-defined sandbox, ready for further security analysis and fuzzing.

Defensive Implications

▶ Watch: Cuttlefish and QEMU Architecture for Android VM (7:36)

The work presented by Romain Malmain carries significant defensive implications for securing complex embedded systems, particularly those powered by Qualcomm chipsets within the Android ecosystem.

Firstly, the most direct implication is the enhancement of vulnerability discovery capabilities. By moving from on-device fuzzing to a scalable and reproducible emulation environment, defenders can significantly accelerate the identification of bugs in critical kernel drivers like the GPU. The ability to run fuzzers faster, debug crashes more efficiently (without slow reboots), and control the environment precisely means vulnerabilities can be found and patched before they are exploited in the wild. This proactive approach is crucial for reducing the attack surface of billions of Android devices.

Secondly, the very process of developing the emulation itself acts as a potent security audit. As demonstrated by the discovery of a CVE (to be published in January) during the implementation phase, the deep dive required to reverse engineer hardware interactions and reimplement device behavior in QEMU inherently exposes logic flaws, undocumented features, and potential security weaknesses in the original kernel driver code. This "emulation-driven security audit" complements traditional code review and fuzzing by forcing a meticulous understanding of complex interfaces, which can reveal subtle but critical vulnerabilities. Defenders can leverage this methodology by investing in internal emulation efforts for their proprietary hardware interfaces.

Thirdly, the open-source nature of the QEMU device implementations (excluding the specific fuzzing harness, which remains proprietary for Qualcomm) fosters community-driven security research. By making the virtual board and device models available, other security researchers, independent developers, and even other manufacturers can build upon this work. This collaborative approach can lead to broader scrutiny of hardware-software interfaces, potentially uncovering more vulnerabilities across the industry and raising the overall security bar.

Finally, the technical optimizations presented, such as the modified kcov for efficient kernel coverage feedback and the use of physical memory allocation for performance, provide a blueprint for building more effective and performant fuzzing infrastructure. Defenders can adopt these techniques to improve the efficiency of their own emulation-based fuzzing setups, ensuring that their testing efforts are not bottlenecked by slow execution or inadequate coverage collection. This work provides a concrete example of how deep technical understanding of emulation can translate directly into practical improvements for defensive security operations.

Key Takeaways

• Emulation for Android Security is Feasible and Scalable: Building a full emulated Android phone for security testing, particularly for complex components like Qualcomm GPUs, is achievable using existing frameworks like Cuttlefish and QEMU, offering a superior alternative to costly and slow on-device fuzzing farms.
• Device Trees are Central to Hardware Emulation: The Device Tree Binary (DTB) is fundamental for describing hardware to the kernel. Overcoming address space collisions between generic virtual boards and specific hardware requires meticulous DTB merging and address relocation.
• Kernel Module Deferral is a Major Challenge: The -EPROBE_DEFER mechanism, which signals device dependencies, presents a significant hurdle during emulation development. Understanding and resolving these implicit dependencies is critical for successful kernel boot and driver initialization.
• Critical Devices for GPU Emulation: Key components like RPMH (power management), SMMU (IOMMU for address mapping), and HFI (DMA-based host-firmware communication) are essential for getting the GPU kernel driver to operate correctly in an emulated environment, often requiring thousands of lines of custom QEMU code.
• Emulation Development Itself Uncovers Bugs: The rigorous process of reverse engineering and implementing device behavior in QEMU can expose real-world vulnerabilities in the target software, as evidenced by the discovery of a CVE in the Qualcomm GPU driver during this project.
• LibAFL QEMU Enhances Fuzzing Efficiency: Integrating the emulated environment with LibAFL QEMU allows for in-process fuzzing, significantly improving execution speed and reducing overhead compared to traditional multi-process fuzzing setups.
• Performance Optimizations are Crucial for Fuzzing: Customizing kernel coverage collection (e.g., modifying kcov output) and optimizing memory allocation (e.g., using physical memory for faster operations) are vital steps to maximize fuzzing throughput and effectiveness in emulated environments.

About the Speaker(s)

The primary speaker for this talk was Romain Malmain. At the time of the presentation, Romain was a PhD student at EURECOM, an engineering school and research center in the south of France. The work presented in the talk was conducted during a three-month internship at Qualcomm over the summer, where he focused on GPU emulation and fuzzing. He received technical assistance and supervision from Scott Bauer during his internship, with whom he also collaborated extensively to make the research public and open source.