SyzBridge: Bridging the Gap in Exploitability Assessment of Linux Kernel Bugs in the Linux Ecosystem

SyzBridge: Bridging the Gap in Exploitability Assessment of Linux Kernel Bugs in the Linux Ecosystem

Speakers: Xiaochen Zou

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=YD-pwB64Cs

Overview

Continuous fuzzing, exemplified by platforms like syzbot, has become an indispensable component of the Linux kernel development ecosystem, successfully unearthing thousands of bugs. However, a significant and puzzling disparity exists: despite numerous bugs being classified as high-risk by advanced exploitability assessment tools such as SyzScope and KOOBE, only a minuscule fraction ever translate into real-world exploits against production Linux distributions like Ubuntu or Fedora. This talk, presented by Xiaochen Zou, critically examines this "exploitability gap," challenging the prevailing assumption that Proof-of-Concepts (PoCs) developed for upstream, development-oriented kernels will seamlessly apply to the diverse landscape of downstream production kernels.

The core motivation behind SyzBridge is to understand why upstream PoCs frequently fail on downstream kernels and to develop an automated solution to bridge this crucial gap. The research reveals that environmental differences, such as varying kernel configurations, module loading practices, background processes, and privilege requirements, are the primary culprits preventing bug reproduction. By systematically identifying and adapting PoCs to these downstream nuances, SyzBridge aims to make exploitability assessments more accurate, relevant, and actionable for real-world Linux kernel security, ultimately enabling better protection for end-users.

This work not only quantifies the extent of the problem but also provides a practical, automated framework to overcome it. SyzBridge significantly enhances the ability to reproduce bugs and assess their true exploitability on production systems, thereby transforming how the security community perceives and addresses Linux kernel vulnerabilities.

Background

▶ Watch: Introduction and motivation for SyzBridge (0:00)

The Linux kernel ecosystem is characterized by a multi-layered structure that significantly impacts bug propagation and exploitability. At its core is Linux mainline, the upstream development branch where new features and bug fixes are initially integrated. From mainline, stable or long-term-support (LTS) branches fork, prioritizing stability and maintenance. These stable/LTS branches then serve as the foundation for downstream Linux distributions (distros) such as Ubuntu, Fedora, Debian, and SUSE, which are deployed in production environments. Crucially, downstream kernels often diverge from their upstream counterparts. This divergence stems from various factors, including support for specific user-space features, integration of out-of-tree drivers, enabling or disabling kernel features via CONFIG options, and selective cherry-picking of security patches. This inherent diversity is a central tenet of SyzBridge's investigation.

Syzbot, a Google-operated continuous kernel fuzzing platform, plays a pivotal role in bug discovery. It compiles and fuzzes the latest mainline or LTS kernels daily, typically focusing on mainline to detect bugs early in the development cycle. Syzbot employs common kernel CONFIG options and various sanitizers like KASAN (Kernel Address SANitizer) and KMSAN (Kernel Memory SANitizer) to enhance bug detection. A key operational detail is that syzbot always executes test cases as the root user, which enables it to uncover bugs that might require elevated privileges, but also masks the real-world privilege requirements for unprivileged users.

Bug exploitability is often contingent on the presence of high-risk primitives, which allow an attacker to gain control over system resources or execution flow. Tools like SyzScope and KOOBE have been developed to identify and elevate low-risk bugs to high-risk classifications. Common high-risk primitives include memory write primitives (e.g., use-after-free write, out-of-bounds write), memory free primitives (e.g., arbitrary free, double-free), and control flow hijacking. SyzScope, for instance, classified 183 out of 1,173 fuzzer-exposed bugs as high-risk, and KOOBE successfully generated 6 new exploits. Despite these advancements, the number of publicly reported real-world Linux kernel exploits remains remarkably low, with syzbot itself reporting only 5 exploitable kernel vulnerabilities since 2017. This stark discrepancy between the theoretical high-risk bugs and practical exploitation forms the "exploitability gap" that SyzBridge seeks to bridge.

Previous research has explored cross-version bug and exploit assessment. Static analysis methods attempt to detect recurring bugs across different kernel versions based on code similarity, but they do not provide concrete PoCs. Dynamic approaches like VulScope focus on migrating user-space PoCs across versions, while AEM (Automated Exploit Migration) concentrates on adapting kernel exploits. However, AEM assumes the bug already triggers on the target kernel and focuses on adapting post-triggering exploit logic. SyzBridge's work is orthogonal and foundational, focusing squarely on the prerequisite of triggering the bug itself on diverse downstream kernels, ensuring that the underlying vulnerability is reproducible and accessible in real-world production environments before exploit development can even begin.

Key Findings

▶ Watch: Linux kernel ecosystem divergence and Syzbot context (2:00)

To comprehensively understand the reasons behind the failure of upstream PoCs on downstream kernels, an extensive exploratory experiment was conducted. This study utilized a dataset of 230 KASAN bugs discovered by syzbot on Linux upstream between January 1st, 2020, and December 31st, 2022, all of which had C-language PoCs. Additionally, 5 publicly known CVEs originating from syzbot within the same timeframe were included. The downstream targets comprised 43 distinct major releases of four popular distributions: Ubuntu, Fedora, Debian, and SUSE, covering versions supported during the bug discovery period (as detailed in Table I of the original paper). This resulted in an extensive dataset of 8,032 unique bug/distro pairs.

For the experiment, each distro kernel was compiled with CONFIG_KASAN and other debug features enabled to ensure bug detection. A dedicated virtual machine was launched for each bug/distro pair. PoCs were initially executed as the root user, and if successful, re-executed as a default, unprivileged user. Each PoC was allotted a 600-second timeout.

The results of this exploratory experiment were highly illuminating:

• Root-level Triggerability: Only a small fraction of the 230 upstream bugs were triggerable by the root user on downstream distros: 55 for Ubuntu, 58 for Fedora, 33 for Debian, and 29 for SUSE. This translates to an average of only 19.1% of bugs.
• Unprivileged Triggerability: The situation was even more dire for unprivileged users, with only 2 bugs for Ubuntu, 3 for Fedora, 2 for Debian, and 1 for SUSE being triggerable. This averages to a mere 0.9% of bugs.
• CVEs: Among the 5 known CVEs, 4 were triggerable by root, but none by an unprivileged user.

These statistics unequivocally demonstrated that the vast majority of fuzzer-exposed upstream bugs do not directly impact downstream kernels in their default configuration, especially for unprivileged users.

A subsequent manual investigation of 50 KASAN bugs that failed to reproduce or only reproduced with root privileges provided deeper insights into the root causes. After filtering out 120 pairs where the buggy commit was simply absent in the downstream kernel (true negatives), 80 bug/distro pairs with the vulnerable code present were analyzed. Only 18 of these reproduced with root, and none with unprivileged users. The manual analysis categorized failures into four primary types (summarized in Table II):

1. R1: Necessary Logic Missing (41 out of 62 failed pairs): The buggy commit was present, but a critical function or code path involved in the bug was not compiled into the downstream kernel. This made direct adaptation impossible. SyzBridge deliberately excludes these true negatives from its scope of adaptation.
2. R2: Code Context Change (1 out of 62 failed pairs): Minor differences in kernel code, such as an extra check, required small PoC adaptations. These rare cases were deemed outside the primary scope of SyzBridge, which focuses on more prevalent issues.
3. R3: Environment Requirements Unsatisfied (20 out of 62 failed pairs): Failures not due to core logic changes but environmental discrepancies in the downstream OS. This category was further broken down:

• R3-1: Preparation Steps Failing (3 pairs): PoCs contained debugging-specific devices (e.g., /dev/raw-gadget) that are absent in production kernels.
• R3-2: Distro Background Noises (13 pairs): Daemon processes and services in downstream OSes occupied resources (e.g., loop devices) or interfered with race conditions, which upstream kernels (minimal environments) easily won.
• R3-3: Necessary Kernel Modules Not Loaded (10 pairs): Modules required by the bug were compiled but not loaded by default. These could be loaded, but the PoC didn't do so.

1. R4: Privilege Requirement (18 pairs successful with root, but not unprivileged): PoCs failed due to kernel security checks, specifically uid/gid checks and capability checks (e.g., CAP_NET_ADMIN). Syzbot runs as root, bypassing these, but unprivileged users cannot.

These detailed findings formed the empirical foundation for the design and implementation of the SyzBridge system, directly informing its automated adaptation strategies.

Technical Deep Dive

▶ Watch: The exploitability gap: Few real-world kernel exploits (3:00)

SyzBridge is an automated, end-to-end system designed to address the challenges identified in the exploratory experiment. It takes an upstream PoC, evaluates its triggerability on downstream kernels, and automatically applies necessary adaptations. Implemented with approximately 8,000 lines of Python code, SyzBridge comprises an upstream bug crawler, virtual machine management, PoC handlers, and a suite of plugins, each targeting a specific failure reason.

SyzBridge's core functionality is encapsulated in its adaptation plugins:

A. Environment Adaptation for Failed Preparation Steps (Addressing R3-1)

Upstream syzbot PoCs often include preparation steps for kernel functionalities (e.g., USB, WiFi) that rely on debugging interfaces or specific device configurations not present in production kernels. SyzBridge tackles this by performing a minimization process on these steps. It iteratively disables existing preparation steps and identifies the minimal set absolutely necessary for the bug to trigger, ensuring the PoC doesn't fail due to a missing debug device or an unavailable resource.

B. Environment Adaptation for Background Noise (Addressing R3-2)

Downstream operating systems are characterized by a richer environment, including numerous daemon processes and services that can occupy critical resources or interfere with race conditions.

• Resource Occupation: If a PoC requires a resource (e.g., a loop device) that is currently in use, SyzBridge attempts to force the kernel to release it (e.g., by unmounting) before executing the PoC.
• Race Conditions: SyzBridge treats every bug as a potential race condition, especially given the increased contention in production environments. It adapts PoCs to launch multiple processes (typically 6, mirroring syzbot's approach for race bugs) to increase the probability of a collision. Additionally, PoCs are adapted to run in a continuous loop with a 600-second timeout, significantly improving the chances of winning a race condition that might be masked by background noise.

C. Environment Adaptation for Module Loading (Addressing R3-3)

Downstream kernels frequently compile many functionalities as loadable kernel modules, which are not always loaded by default. SyzBridge intelligently identifies and loads only the necessary modules, ideally without requiring root privileges.

1. Pinpointing Explicitly Dependent Modules (Depexp): These are modules whose functions appear in the PoC's execution trace on the upstream kernel but are missing when executed downstream. SyzBridge collects function-level kernel execution traces using ftrace for the PoC on both upstream and downstream kernels. It then maps these functions to their respective kernel modules by analyzing source file names from the kernel's Makefile. Only loadable add-on modules are considered for automatic loading; modules that are built-in or not compiled are discarded.
2. Pinpointing Implicitly Dependent Modules (Depimp): These modules do not appear in the PoC's direct execution trace but are crucial for the bug to trigger due to indirect dependencies. A prime example is the xfrm_user module in the motivating example, which initializes a global linked list. If xfrm_user isn't loaded, a function reading from that list will fail, even if the reading function itself is present. SyzBridge employs static analysis using a kernel dependency analysis tool (referred to as "tool 42" in the paper) to identify cross-module dependencies on global variables. It specifically focuses on global variables that are read in one module and written in the module_init functions of another, helping to identify modules that must be loaded early for the bug to manifest. Three common global linked lists related to character, block, and network device drivers were specifically targeted.
3. Module Minimization: After identifying both Depexp and Depimp modules, SyzBridge performs a minimization similar to syscall minimization. It iteratively adds identified modules one by one, verifying if the bug triggers, to find the minimal set of modules truly required for successful bug reproduction.

D. Privilege Adaptation for Module Loading (Addressing R4-1 for modules)

While standard module loading via modprobe requires the CAP_SYS_MODULE capability (typically root privileges), Linux kernels offer less conspicuous ways for unprivileged users to trigger module loading through specific syscalls (e.g., socket()). SyzBridge systematically discovers these unprivileged loadable modules by identifying request_module() invocations within the kernel. It developed a guided fuzzer, built upon syzkaller, to generate unprivileged test cases that successfully reach request_module(). This fuzzer leverages call graph analysis (MLTA 54) to resolve indirect calls and incorporates feedback on both code coverage and the actual module name loaded (by instrumenting request_module()). This fuzzer successfully identified hundreds of unprivileged loadable modules across various distros (Table III), and SyzBridge builds a database of PoCs to load these.

E. Privilege Adaptation for Kernel Security Checks (Addressing R4-1 for other checks)

Modern Linux kernels support user namespaces (25), which allow unprivileged users to create a namespace and gain elevated privileges within that namespace, provided kernel.unprivileged_userns_clone is enabled. SyzBridge instruments the upstream kernel to precisely pinpoint security checks (e.g., uid/gid or capability checks). If a check is against init_user_ns (the initial user namespace), it cannot be bypassed by an unprivileged user. Otherwise, SyzBridge applies namespace adaptation to elevate privileges within the newly created user namespace, allowing the PoC to pass the kernel security checks and trigger the bug. This effectively transforms many privilege-gated bugs into unprivileged ones in environments where user namespaces are permitted.

Demo / Proof of Concept

▶ Watch: Exploratory experiment setup for bug analysis (4:00)

SyzBridge's effectiveness was not merely theoretical; its capabilities were rigorously validated through practical demonstrations and end-to-end exploit development. The "End-to-End Exploitation" phase of the evaluation served as a powerful proof of concept, sampling 5 bugs from a list of newly identified high-risk vulnerabilities that were previously not publicly known to be exploitable. For all five, the researchers successfully developed end-to-end exploits. Crucially, one of these exploits achieved privilege escalation against the latest Ubuntu kernel, leading to the assignment of a new CVE. For the other four bugs, which had already been fixed in later kernel versions, simplified exploits (with kernel defenses like KASLR, SMAP, and SMEP disabled for demonstration purposes) were developed to serve as a proof-of-exploitability. This practical validation underscores SyzBridge's ability to identify truly exploitable vulnerabilities in real-world environments.

Two specific case studies further illustrate how SyzBridge applies its adaptations:

1. Case Study: bd699d3 (Figure 3 in the paper) - Implicit Module Dependency

This bug involved a vulnerability in tc_new_tfilter() within the kernel core, triggered after passing two checks. When the upstream PoC was initially run on Ubuntu-20.04.2 as root, it failed the second check, which involved _tcf_qdisc_find() attempting to retrieve a network device from a global net_device linked list. SyzBridge's module loading adaptation diagnosed the root cause: upstream kernels often statically link many modules, meaning ipip_init_net() (which registers the necessary network device) is called during boot. In contrast, Ubuntu uses ipip as a loadable module, and it's not loaded by default. Without the ipip module, the network device isn't registered, causing _tcf_qdisc_find() to fail. SyzBridge successfully identified ipip as an implicitly dependent missing module (Depimp). It then leveraged its unprivileged module loading technique to load the ipip module, allowing the PoC to bypass the check and successfully trigger the bug. This case highlights SyzBridge's advanced capability in resolving complex, indirect module dependencies.

1. Case Study: e67f2fc (Figure 4 in the paper) - Race Condition due to Background Noise

This bug was a use-after-free race condition occurring between rtnetlink_rcv_msg() and tun_detach(). The vulnerable object was destroyed by _tun_detach() (line 16), but another kernel thread might still be using it via free_netdev() (line 9). For the bug to trigger, the thread executing tun_detach() needed to be scheduled immediately after a lock release (line 5) and before free_netdev() executed. On the upstream kernel, the PoC often won this race easily. However, on downstream distros, with their increased background noise and system processes, the same PoC consistently failed to win the race within the 600-second timeout. SyzBridge's environment adaptation for background noise addressed this directly. By treating the bug as a potential race condition, it adapted the PoC to run in a continuous loop. This seemingly simple adaptation allowed the race condition to be triggered successfully in less than a second on downstream kernels, vividly demonstrating how environmental factors, rather than core logic changes, can prevent bug reproduction and how SyzBridge effectively overcomes them.

These demonstrations collectively illustrate SyzBridge's practical utility in transforming theoretically high-risk bugs into reproducible and exploitable vulnerabilities on real-world production systems.

Defensive Implications

▶ Watch: Striking results: Low downstream bug triggerability (4:40)

SyzBridge's findings and capabilities carry significant implications for Linux kernel defenders, offering crucial insights into the real-world impact of vulnerabilities and guiding more effective defensive strategies.

Firstly, the most fundamental implication is the critical need to test vulnerabilities against actual downstream distribution kernels, not just upstream development versions. The exploratory experiment unequivocally demonstrated that only 19.1% of upstream bugs were root-triggerable and a mere 0.9% unprivileged-triggerable on downstream systems. This highlights a dangerous disconnect: relying solely on upstream PoCs or exploitability assessments can lead to a false sense of security regarding production systems. Defenders must integrate testing on representative distro kernels into their vulnerability assessment pipelines to gain an accurate understanding of their exposure.

Secondly, the identified failure categories (R3-1, R3-2, R3-3, R4) provide actionable intelligence for hardening.

• Environment Configuration (R3-1, R3-2): Downstream kernel configurations, particularly regarding debug devices and the presence of background services (like Ubuntu's snap occupying loop devices), can significantly affect bug triggerability. Defenders should be aware that a minimalist upstream testing environment might inadvertently mask real-world race conditions or resource contention issues. Implementing more robust, production-like testing environments, potentially mirroring common user installations, is essential.
• Module Loading Practices (R3-3): The common practice of compiling many functionalities as loadable modules in downstream kernels, but not loading them by default, is a double-edged sword. While it reduces the attack surface, SyzBridge demonstrates that attackers can often identify and load these necessary modules. Defenders should review their module loading policies, especially for potentially vulnerable modules, and consider whether some modules should be built-in or entirely removed if not strictly necessary. The discovery of hundreds of unprivileged loadable modules by SyzBridge's fuzzer is particularly concerning, as it reveals a hidden attack surface for unprivileged users to load modules that might expose new bugs or facilitate exploitation. Regular auditing of request_module() invocations and the syscalls that trigger them is advised.
• Privilege Requirements and User Namespaces (R4): SyzBridge's success in leveraging user namespaces to bypass uid/gid and capability checks for unprivileged users is a stark reminder of the evolving privilege model in Linux. While user namespaces offer legitimate benefits, their default enablement (via kernel.unprivileged_userns_clone) means that many bugs previously considered "root-only" can become exploitable by unprivileged users within a namespace. Defenders should carefully consider their kernel.unprivileged_userns_clone policy. If user namespaces are enabled, it's crucial to understand that they can significantly expand the attack surface, transforming many local privilege escalation (LPE) vulnerabilities from theoretical to practical.

Thirdly, the research underscores the slow propagation of upstream patches to downstream kernels. The observation that 38 of the 53 high-risk bugs identified by SyzBridge did not even require adaptation suggests that many vulnerabilities linger in production kernels for extended periods. This highlights a systemic issue that distro maintainers grapple with. Defenders should advocate for and support efforts to accelerate patch integration, potentially leveraging automated patch correctness assessment and adaptation tools. End-users and system administrators should prioritize keeping their systems updated, as even old, "fixed" upstream bugs can remain exploitable in downstream distributions for months or years.

Finally, SyzBridge's integration with tools like SyzScope to identify 53 high-risk bugs on downstream distributions, leading to new CVEs and validated exploits, provides a strong argument for adopting such integrated pipelines. Defenders can use this information to prioritize patching and mitigation efforts, focusing on vulnerabilities that have been validated as triggerable and exploitable in their specific production environments, rather than being overwhelmed by a flood of upstream bug reports that may not directly impact them. This shift from upstream-centric to downstream-relevant exploitability assessment is paramount for effective kernel security.

Key Takeaways

• Significant Exploitability Gap: Thousands of high-risk Linux kernel bugs found by syzbot rarely translate into real-world exploits on production (downstream) distributions due to environmental and privilege differences.
• Upstream PoCs are Insufficient: Only an average of 19.1% of upstream PoCs reproduce bugs as root, and a mere 0.9% as unprivileged users on downstream kernels, highlighting the ineffectiveness of direct application.
• SyzBridge Bridges the Gap: SyzBridge is an automated system that intelligently adapts upstream PoCs to satisfy downstream kernel requirements, including environment adjustments (background noise, preparation steps, module loading) and sophisticated privilege adaptations (unprivileged module loading, user namespaces).
• Dramatic Improvement in Triggerability: SyzBridge successfully increased root-triggerable bugs by an average of 61% and unprivileged-triggerable bugs by an astonishing 1300% on downstream distros.
• Identified New High-Risk Bugs: Integrated with SyzScope, SyzBridge identified 53 high-risk, downstream-applicable bugs (including 9 control flow hijacks, 16 use-after-free writes, 11 out-of-bounds writes, and 7 double-free bugs), a substantial increase from the 5 previously known syzbot-originated exploits.
• Practical Validation and New CVE: The research validated these findings by developing end-to-end exploits for 5 additional bugs, one of which led to the assignment of a new CVE against the latest Ubuntu kernel, proving SyzBridge's practical impact.

About the Speaker(s)

Xiaochen Zou is the speaker for this detailed technical presentation on SyzBridge. While specific biographical details beyond their name are not provided in the transcript, the depth and rigor of the research presented indicate expertise in kernel security, fuzzing, exploitability assessment, and operating system internals. Their work on SyzBridge demonstrates a strong background in identifying practical challenges in vulnerability analysis and developing sophisticated automated solutions to address them within complex software ecosystems like the Linux kernel.