MOAT: Towards Safe BPF Kernel Extension

MOAT: Towards Safe BPF Kernel Extension

Speakers: Hongyi Lu; Shuai Wang; Yechang Wu; Wanning He; Fengwei Zhang

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=__2WUqcTJjg

Overview

The talk "MOAT: Towards Safe BPF Kernel Extension" by Hongyi Lu and colleagues from SASC and HK addresses a critical security challenge within the rapidly expanding extended Berkeley Packet Filter (eBPF) ecosystem. eBPF, a powerful kernel virtual machine that allows user-space programs to extend kernel functionality, has become ubiquitous in modern Linux systems for tasks ranging from networking and tracing to security. Despite its advantages, including performance comparable to kernel modules and a static verifier designed to prevent kernel crashes, eBPF has been plagued by a significant number of kernel vulnerabilities, with over 20 privilege escalation flaws reported in its subsystem.

MOAT (Memory-safe eBPF with Hardware-Assisted Protection) proposes a novel solution to these security concerns by leveraging hardware features, specifically Intel Memory Protection Keys (MPK), to enforce strong isolation between eBPF programs and the Linux kernel. The research identifies the inherent limitations of static analysis in the eBPF verifier and the challenges posed by eBPF's rapid development as primary sources of these vulnerabilities. By integrating hardware-backed isolation with targeted software mitigations for domain-specific issues like limited MPK tags and BPF API protection, MOAT aims to provide a robust and performant defense against malicious or buggy eBPF programs.

This work is particularly significant because it tackles a fundamental problem in kernel security: safely extending kernel functionality without introducing new attack surfaces. As eBPF continues to grow in adoption and complexity, ensuring its security is paramount for the stability and integrity of the entire operating system. MOAT demonstrates that a hybrid approach combining mature hardware security features with intelligent software design can effectively mitigate a broad range of eBPF-related kernel exploits with minimal performance overhead, making it a viable path forward for enhancing kernel extension safety.

Background

▶ Watch: Introduction to MOAT and safe BPF kernel extension (0:00)

The extended Berkeley Packet Filter (eBPF) has evolved dramatically since its introduction to Linux in 2014, transforming from a mere network packet filter into a full-fledged kernel virtual machine. This architecture enables user-space programs to inject custom logic directly into the kernel, extending its functionality in an event-driven manner. Its applications span various Linux subsystems, including high-performance networking, system tracing, and security monitoring.

One of eBPF's primary appeals, especially when compared to traditional kernel modules, lies in its performance and safety guarantees. eBPF programs benefit from JIT (Just-In-Time) compilation, executing as native machine code with performance equivalent to kernel modules. Crucially, eBPF programs interact with the kernel through a stable set of helpers APIs, promoting portability across different kernel versions, unlike kernel modules that often require recompilation for each kernel update. The most significant safety feature is the verifier, a static analysis component deployed within the kernel that checks eBPF programs for safety before execution. It aims to reject any "badly written" eBPF program that could potentially crash the kernel or exploit vulnerabilities, saving developers significant debugging time compared to unstable kernel modules.

Despite these advantages, eBPF's security model, which relies almost exclusively on the static checks of its verifier, has proven to be insufficient for such a critical kernel component. Over the years, more than 20 kernel privilege escalation vulnerabilities, specifically arbitrary read and write primitives, have surfaced within the BPF subsystem. The core reasons for these vulnerabilities are twofold:

1. Inherent Difficulty of Static Analysis: Balancing the soundness (never flagging a safe program as unsafe) and completeness (never missing an unsafe program) of a verifier is a notoriously hard problem. This challenge is compounded by the verifier's deployment within the kernel, where it must operate under strict performance and resource constraints.
2. Rapid Development and Complexity: eBPF is a rapidly evolving technology. Each new feature introduced into the eBPF framework necessitates corresponding updates and additions to the verifier's logic. This rapid pace, often involving different developers for eBPF features and verifier updates, creates opportunities for inconsistencies, omissions, and outright bugs in the verifier's implementation. These discrepancies can lead to the verifier making incorrect static deductions about program behavior, creating exploitable gaps.

Recognizing these systemic issues, MOAT proposes to augment the eBPF security model by introducing hardware-assisted isolation. The work specifically focuses on leveraging hardware features like Intel Memory Protection Keys (MPK) to isolate BPF programs from the Linux kernel, thereby providing a more robust layer of defense that complements and extends the existing verifier's capabilities. Furthermore, MOAT addresses specific challenges associated with integrating MPK, such as the limited number of available MPK tags and the need to protect the BPF helper APIs, which remain a potential attack surface.

Key Findings

▶ Watch: The critical security flaw of eBPF and its causes (2:09)

MOAT's research culminates in several key findings that collectively demonstrate a viable and effective strategy for securing eBPF kernel extensions:

• Hardware-Assisted Isolation is Feasible: The project successfully demonstrates that hardware features like Intel Memory Protection Keys (MPK) can be effectively utilized to establish strong isolation boundaries between eBPF programs and the sensitive Linux kernel memory. This hardware-backed approach provides a robust defense against memory corruption vulnerabilities that bypass the eBPF verifier.
• Novel MPK Domain Design: MOAT overcomes the inherent limitation of MPK's 16 available tags by introducing a three-domain design: a kernel domain, a BPF domain (initially grouping all BPF programs), and a shared domain. This design efficiently utilizes the limited MPK resources to protect the kernel from eBPF programs, and vice-versa for shared structures.
• Fine-Grained Per-Program Isolation: Beyond the initial three-domain model, MOAT further refines isolation by giving each individual eBPF program its own address space. This ensures that even if one eBPF program is compromised, it cannot directly interfere with other eBPF programs or critical kernel components without triggering a page fault.
• Efficient TLB Overhead Mitigation: The overhead associated with frequent TLB (Translation Lookaside Buffer) flushes during context switches between multiple eBPF programs and the kernel is significantly reduced. MOAT achieves this by maintaining MPK-protected kernel memory as a constant mapping across all address spaces and by utilizing process context IDs (PCID) to avoid unnecessary flushes.
• Robust BPF Helper API Protection: Recognizing that BPF helpers constitute a remaining attack surface, MOAT introduces two layers of defense. Firstly, it identifies 44 "critical objects" within the kernel that are susceptible to exploitation via helpers and isolates them with an additional MPK tag, restricting helper access. Secondly, it implements runtime parameter validation for helper arguments, comparing actual runtime values against the verifier's "sound bonds" to catch discrepancies caused by flawed static deductions.
• Comprehensive CVE Mitigation: A thorough security evaluation confirms that MOAT is capable of mitigating all 26 known BPF kernel privilege escalation vulnerabilities identified within its scope. The detailed analysis of a representative CVE demonstrates how MOAT's MPK-based isolation prevents exploitative memory accesses that would otherwise bypass the verifier.
• Minimal Performance Overhead: MOAT achieves its security goals with remarkably low performance overhead across various mainstream eBPF use cases. This includes less than 2% overhead for network filtering with five concurrent eBPF programs, an average of 6% (max 13%) for system profiling with 11 eBPF programs, and only 3% for system call auditing. Even when running over 100 eBPF programs in parallel, MOAT's overhead remains below 10%, indicating its practicality for real-world deployments.

Technical Deep Dive

▶ Watch: Explanation of Intel MPK: The core hardware feature (3:57)

MOAT's core innovation lies in its multi-layered approach to eBPF program isolation and protection, primarily built upon Intel Memory Protection Keys (MPK) and complemented by intelligent software design.

Intel Memory Protection Keys (MPK)

At its foundation, MOAT leverages Intel MPK, a hardware feature introduced in Intel processors. MPK adds a 4-bit tag to each page table entry, providing a total of 16 distinct keys (from 0 to F). These keys allow for the dynamic toggling of page permissions (read/write access) for entire groups of pages at once, simply by manipulating a specific register (PKRU). This mechanism is significantly more efficient than modifying individual page table entries, making it suitable for frequent permission changes during execution flow.

Three-Domain Isolation Design

A primary challenge with MPK is its limited number of tags (16). While insufficient for isolating hundreds of individual eBPF programs directly, MOAT smartly utilizes these tags to establish fundamental isolation between the kernel and all eBPF programs collectively. It defines three distinct protection domains:

1. Kernel Domain: This domain encompasses all the code and data belonging to the Linux kernel itself. It is assigned a specific MPK tag.
2. BPF Domain: This domain contains all loaded eBPF programs. Initially, all eBPF programs share this single domain and its associated MPK tag.
3. Shared Domain: This domain is allocated for essential kernel structures, such as descriptor tables used for low-level routines like interrupts, which both the kernel and eBPF programs might legitimately need to access. To prevent tampering, this domain is typically set to read-only for eBPF programs.

When an eBPF program is executing, MOAT enables the BPF domain's permissions while simultaneously disabling the kernel domain's permissions. This prevents any unauthorized memory access from the eBPF program to the sensitive kernel memory, immediately triggering an MPK violation if an attempt is made. This initial design effectively protects the kernel from malicious eBPF programs but still leaves all eBPF programs vulnerable to each other within the shared BPF domain.

Per-BPF Program Address Space Isolation and TLB Optimization

To address the risk of malicious eBPF programs attacking benign ones, MOAT extends its isolation model by assigning each individual eBPF program its own dedicated address space. This means that if an eBPF program attempts to read from or write to the memory belonging to another eBPF program, it will result in a page fault, effectively containing the malicious activity.

However, frequent switching between numerous address spaces can introduce significant performance overhead due to TLB (Translation Lookaside Buffer) flushes. The TLB caches recent virtual-to-physical address translations, and a flush invalidates these cached entries, forcing expensive page table walks. MOAT employs two strategies to mitigate this overhead:

1. Constant Kernel Mapping: Thanks to the three-domain design, the MPK-protected kernel memory (kernel domain) can be kept as a constant mapping across all eBPF program address spaces. This means the kernel's page table entries are not flushed when switching between different eBPF program contexts, significantly reducing TLB pressure, especially since the kernel's memory footprint is much larger than that of individual eBPF programs.
2. Process Context ID (PCID): MOAT utilizes Process Context IDs (PCIDs), a feature that allows the TLB to distinguish between entries from different address spaces without requiring a full flush. By tracking the page table entries used by eBPF programs with PCIDs, MOAT avoids unnecessary TLB flushes when switching between an eBPF program and the kernel, or between different eBPF programs, as long as the PCID matches a valid TLB entry.

BPF Helper API Protection

Even with robust memory isolation, eBPF programs can still interact with the kernel through BPF helpers – a set of predefined kernel functions. These helpers represent a potential attack surface if they can be manipulated to access or corrupt critical kernel data. MOAT addresses this with a two-pronged approach:

1. Isolation of Critical Kernel Objects: By meticulously analyzing previous BPF CVEs, the MOAT team identified 44 specific kernel objects that were frequently exploited or could be exploited via BPF helpers. These are designated as "critical objects." MOAT allocates an additional dedicated MPK tag for these 44 objects, creating an "extra domain." This allows MOAT to enforce fine-grained permissions, specifically preventing helpers from reading or writing to these critical objects, as such access is generally not legitimate for normal helper operation. Other non-critical kernel objects remain accessible to helpers to ensure normal functionality.
2. Runtime Parameter Validation for Helpers: The eBPF verifier is known to sometimes make incorrect static deductions about variable values, which can lead to vulnerabilities even if it provides "sound bonds" (valid argument ranges). For instance, the verifier might deduce a variable's range incorrectly but still correctly identify the maximum allowed value for a given helper argument. MOAT exploits this observation by performing runtime validation of helper arguments. It directly compares the actual runtime value of an argument against the "sound bonds" (the statically deduced valid argument ranges) provided by the verifier. If the runtime value falls outside these sound bounds, it indicates an attempt to exploit a verifier bug (e.g., due to incorrect static truncation or range deduction), and MOAT intervenes to prevent the operation. This runtime check acts as a crucial safety net, catching exploits that leverage verifier misinterpretations.

Demo / Proof of Concept

▶ Watch: MOAT's three-domain isolation: Kernel, BPF, and Shared (4:30)

The efficacy of MOAT was rigorously evaluated through both security validation and performance benchmarking, demonstrating its practical viability.

For security evaluation, MOAT was tested against all 26 known BPF-related CVEs that fall within its scope. The results confirmed that MOAT is capable of effectively mitigating every one of these vulnerabilities. The talk provided a detailed example of a specific CVE (though not explicitly numbered in the transcript) to illustrate MOAT's defense mechanism. In this particular exploit scenario, an attacker manipulates register R5 with a controlled address and R6 with a large constant. Through clever code, the attacker tricks the verifier into making a wrong static deduction: the verifier believes R5, after a 32-bit truncation, becomes a constant '1'. This misdeduction allows R5 to be used as a legitimate offset in a memory access operation. However, R5 is, in fact, an arbitrary value controlled by the attacker, leading to an arbitrary read/write primitive. MOAT's defense is straightforward and powerful: by strictly confining all eBPF programs to their designated isolated domains using MPK, any attempt by the attacker to access unauthorized memory (i.e., outside its allocated BPF domain) results in an immediate MPK violation. This hardware-enforced boundary prevents the exploit from succeeding regardless of the verifier's static analysis error.

The performance evaluation was equally critical, as any security solution for a performance-sensitive component like eBPF must introduce minimal overhead. MOAT demonstrated excellent performance across various mainstream eBPF use cases:

• Network Filtering: In a scenario involving network packet filtering, MOAT exhibited a throughput loss of less than 2%, even when five different eBPF programs were running concurrently. This minimal impact highlights its suitability for high-performance networking applications.
• System Profiling: When 11 eBPF programs were loaded to profile various kernel metrics, and Unixbench was run with these programs attached, the maximum overhead observed was under 13%, with an average overhead of only 6%. This demonstrates MOAT's efficiency in complex system monitoring and profiling environments.
• System Call Auditing: For system call auditing, where MOAT was used to audit all system calls of an nginx server, the performance loss was a mere 3%. This makes MOAT a practical choice for security auditing without significantly impacting application performance.
• Scalability Comparison: MOAT was also benchmarked against sandBPF, a previous software-based BPF isolation solution. In a challenging test with over 100 eBPF programs running in parallel, MOAT's overhead remained less than 10%. This competitive performance, especially when compared to software-only solutions, underscores the efficiency gained from leveraging hardware features.

These results collectively demonstrate that MOAT not only provides robust security by mitigating known eBPF vulnerabilities but also does so with an overhead that is acceptable for real-world production environments.

Defensive Implications

▶ Watch: MOAT's security evaluation results and CVE mitigation (9:58)

MOAT's research offers profound defensive implications for various stakeholders involved in kernel development, system administration, and security engineering:

• For Kernel Developers and eBPF Maintainers: MOAT highlights the inherent limitations of relying solely on static analysis (like the eBPF verifier) for securing complex, rapidly evolving kernel extensions. It strongly suggests that incorporating hardware-assisted security features, such as Intel MPK, is a necessary evolution for ensuring the long-term safety of eBPF. This implies a shift towards designing kernel subsystems with hardware isolation in mind, potentially influencing future CPU architecture designs and kernel interfaces. The work also emphasizes the importance of providing "sound bonds" from the verifier, even if static deductions are flawed, as these can be leveraged for runtime validation.
• For System Administrators and Security Engineers:
• Awareness of eBPF Risks: Even with the verifier, eBPF remains a significant attack surface in the kernel. Administrators should be aware that vulnerabilities can and do exist, leading to potential privilege escalation.
• Leverage Hardware Features: If systems are equipped with hardware features like Intel MPK, MOAT demonstrates their potential for enhancing kernel security. While MOAT itself is a research prototype, its principles could inform the development of more secure eBPF runtime environments or be integrated into future kernel versions.
• Layered Security Approach: The success of MOAT's hybrid hardware-software approach underscores the value of layered security. Relying on a single defense mechanism (like a static verifier) is insufficient. Combining static analysis with dynamic, hardware-enforced isolation provides a much stronger defense in depth.
• Runtime Validation as Complement: The concept of using runtime checks to validate parameters against verifier-provided "sound bonds" is a powerful technique. This can be generalized to other kernel interfaces where static analysis might be fallible, providing a dynamic safety net against subtle logic bugs.
• Performance vs. Security Trade-off: MOAT's low overhead figures are critical. They demonstrate that robust hardware-backed security for eBPF does not necessarily incur prohibitive performance costs, making such solutions practical for high-performance production environments. This should encourage adoption of similar secure execution environments.
• For Security Researchers: MOAT provides a new framework and set of techniques for analyzing and mitigating kernel vulnerabilities. It opens avenues for exploring other hardware features (e.g., ARM MTE, RISC-V PMP) for similar isolation purposes and for developing more sophisticated runtime validation techniques that integrate deeply with static analysis results.

In essence, MOAT advocates for a paradigm shift in how we secure kernel extensions, moving beyond purely software-based static analysis to embrace a more robust, hardware-assisted, and multi-layered defense strategy.

Key Takeaways

• eBPF's Rapid Growth Poses Security Challenges: While eBPF is a powerful and thriving ecosystem, its rapid development and the inherent difficulty of static analysis in its verifier have led to numerous kernel vulnerabilities.
• Hardware Features Enhance Kernel Extension Security: Leveraging hardware features like Intel Memory Protection Keys (MPK) can significantly bolster the security of eBPF programs by enforcing strong, efficient isolation from the Linux kernel.
• Hybrid Hardware-Software Solutions are Effective: MOAT successfully combines MPK for memory isolation with targeted software mitigations (e.g., address space isolation, TLB optimizations, critical object protection, runtime parameter validation) to provide comprehensive security for eBPF programs and their API interactions.
• Runtime Validation Complements Static Analysis: MOAT's approach of validating BPF helper arguments at runtime against the verifier's "sound bonds" effectively addresses vulnerabilities arising from incorrect static deductions, acting as a crucial safety net.
• Practicality Through Low Performance Overhead: Despite introducing robust security, MOAT demonstrates remarkably low performance overhead (typically <10% across various use cases), making its hardware-assisted protection practical for real-world eBPF deployments.
• Comprehensive Mitigation of Known Exploits: MOAT has been proven to mitigate all 26 known BPF-related kernel privilege escalation vulnerabilities within its scope, showcasing its effectiveness against real-world threats.

About the Speaker(s)

The talk "MOAT: Towards Safe BPF Kernel Extension" was presented by Hongyi Lu, representing SASC. This research is the result of a collaborative effort, jointly conducted by SASC and HK. The co-authors contributing to this significant work include Shuai Wang, Yechang Wu, Wanning He, and Fengwei Zhang. While specific titles were not detailed in the transcript, their collective contribution highlights expertise in systems security, kernel development, and hardware-assisted protection mechanisms.