IsolatOS: Detecting Double Fetch Bugs in COTS RTOS by Re-enabling Kernel Isolation

IsolatOS: Detecting Double Fetch Bugs in COTS RTOS by Re-enabling Kernel Isolation

Speakers: Yingjie Cao

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=uzXVHGqn1Zk

Overview

Real-time operating systems (RTOS) are the invisible backbone of cyber-physical systems -- from automotive ECUs and aerospace systems to power plants and medical devices. With over 2.2 billion embedded devices relying on RTOS, the security of these systems is critical. This talk presents IsolatOS, a novel approach to detecting double fetch vulnerabilities in commercial off-the-shelf (COTS) RTOS by strategically re-enabling hardware kernel isolation features (SMAP on x86, PAN on ARM) that vendors intentionally disable for performance.

The results are striking: IsolatOS discovered 43 vulnerabilities across QNX, VxWorks, and seL4, with 39 assigned CVEs, including a 19-year-old vulnerability in QNX that allows local privilege escalation and affects production vehicles from major manufacturers. The approach is 173 times faster than state-of-the-art emulation-based detection while achieving near-zero false positive rates -- compared to QEMU-TCG's 87.7% false positive rate. This is the first work to leverage hardware kernel isolation mechanisms for double fetch detection.

Background

▶ Watch: RTOS as the backbone of cyber-physical systems and double fetch threat (0:00)

A double fetch vulnerability occurs when the kernel reads the same user-space memory location multiple times without ensuring data consistency between fetches. The time window between the first fetch (typically validation) and the second fetch (actual use) creates a race condition: if an attacker modifies the data between these fetches via a concurrent malicious thread, the kernel blindly trusts its initial validation and consumes maliciously modified data.

The security impact is severe: approximately 46% of exploitable double fetch bugs lead to privilege escalation, while around 40% result in information leaks. A classic example is a double fetch bug successfully exploited on a Tesla vehicle during Pwn2Own, earning a $100,000 bounty. Critical impacts have also been demonstrated on aviation systems running VxWorks, such as Boeing 787 systems.

RTOS environments make double fetch detection particularly challenging because of three characteristics: priority-based preemptive scheduling (up to 256 priority levels where the kernel is fully preemptable at nearly any point), direct pointer dereferencing for speed instead of explicit wrapper calls, and heavy reliance on shared memory for IPC with limited synchronization. These characteristics mean legitimate concurrent accesses by different threads can appear identical to actual double fetch bugs when monitored through traditional time-window heuristics.

Existing detection methods are inadequate: static analysis requires source code (unavailable for proprietary RTOS), and dynamic emulation-based methods (QEMU-TCG) incur 30-80x slowdown while generating an 87.7% false positive rate due to inability to handle preemption-induced noise.

Key Findings

▶ Watch: Double fetch mechanics and 46% privilege escalation rate (4:00)

IsolatOS exploits a crucial operational reality: COTS RTOS vendors intentionally disable hardware kernel isolation features (Intel SMAP via CR4 bit 21, ARM PAN) to maximize performance. These features are natively designed to identify cross-boundary memory accesses -- exactly what double fetch detection requires. By strategically re-enabling these features, IsolatOS can trap cross-boundary accesses with minimal overhead.

The system call lifecycle tracking innovation eliminates preemption noise: by assigning a unique pairing ID to each system call's entry and exit, IsolatOS can distinguish whether two fetches of the same address belong to the same system call lifecycle (true double fetch) or were interleaved by preemption (false positive). This is why IsolatOS achieves near-zero false positive rates compared to QEMU-TCG's 87.7%.

The key vulnerability discoveries include:

• 37 vulnerabilities in QNX 6.6 and 7.0, including 6 leading to local privilege escalation
• A 19-year-old vulnerability in QNX that allows local privilege escalation via arbitrary write, affecting production vehicles from major manufacturers
• The exploit for the 19-year-old bug achieves a 76% success rate within a critical time window of just 125 CPU cycles
• Additional vulnerabilities in VxWorks and seL4

Technical Deep Dive

▶ Watch: System call lifecycle tracking eliminates false positives (8:00)

IsolatOS operates through a blend of static binary rewriting and dynamic execution monitoring:

Static instrumentation: The system scans RTOS binaries for kernel entry points by identifying instructions like WRMSR (write to model-specific register). At each entry point, code is injected to set the hardware isolation bit (CR4 bit 21 for SMAP) and assign a unique system call pairing ID. This instrumentation requires no source code access.

Dynamic monitoring: When the kernel attempts to access user-space memory, the hardware isolation triggers a page fault. A custom fault handler intercepts this fault, logs the CPU context (target memory address, instruction pointer, current system call ID, and CPU identifier for multi-core systems), and must then recover execution.

Fault recovery paradox: Re-enabling hardware isolation means legitimate kernel cross-boundary accesses will trigger faults. To complete the faulting instruction, isolation must be temporarily disabled, but this creates a monitoring gap. IsolatOS solves this with a specific instruction recovery mechanism: temporarily disable isolation, execute exactly the faulting instruction, and immediately re-enable isolation before continuing. A JTAG/GDB debugging channel ensures architecture-independent recovery.

Double fetch determination: Access logs containing address, instruction pointer, and system call ID are grouped by system call ID. If an address is accessed twice or more within the same ID group, it is flagged as a true double fetch. This lifecycle-based tracking completely eliminates preemption-induced false positives.

The approach was integrated into three distinct RTOS architectures: QNX (as a board support package modification, 8 hours porting effort), VxWorks (as a custom kernel driver, 2 work days), and seL4 (via source code modification, 2 work days).

Demo / Proof of Concept

▶ Watch: Fault recovery paradox solved with instruction-level isolation toggling (10:00)

The 19-year-old QNX vulnerability is the most compelling finding. The bug allows local privilege escalation via arbitrary write through a double fetch in a system call handler. The exploit achieves a 76% success rate within a critical window of just 125 CPU cycles -- demonstrating that RTOS preemptive scheduling creates highly exploitable race conditions. This vulnerability affects vehicles from major automotive manufacturers running QNX, highlighting the real-world impact on safety-critical systems.

Performance comparisons demonstrate IsolatOS's advantage dramatically: on a log scale chart, QEMU-TCG imposes approximately 80x average overhead, while IsolatOS incurs an average overhead of just 45% -- making it 173 times faster in effective throughput. The false positive analysis is equally stark: QEMU-TCG's 87.7% false positive rate breaks down into 80% temporal false positives and 18.7% preemption noise, all of which IsolatOS eliminates.

Defensive Implications

▶ Watch: 43 vulnerabilities found: 19-year-old QNX bug in production vehicles (14:00)

For organizations deploying RTOS in safety-critical applications:

1. QNX users should immediately assess exposure to the 37 discovered vulnerabilities, particularly the 6 privilege escalation bugs. The 19-year-old vulnerability demonstrates that deeply embedded bugs can persist in production for decades.

1. RTOS vendors should stop disabling SMAP/PAN by default. While the performance cost is real, the security benefit of detecting cross-boundary access violations is significant, especially in safety-critical deployments.

1. Automotive manufacturers using QNX (over 250 million vehicles as of 2022 for BlackBerry QNX) should verify that patches for the discovered CVEs have been applied throughout their vehicle fleets. The 76% exploit success rate within 125 CPU cycles makes this a practical, not theoretical, threat.

1. Security testing programs for RTOS should adopt IsolatOS or equivalent approaches. The 173x speed improvement over emulation makes dynamic double fetch detection practical as part of CI/CD or pre-deployment testing.

Future work includes extending the approach to trusted execution environments, hypervisor security, and additional architectures, as well as exploring automated patch generation and compiler-level detection.

Key Takeaways

• IsolatOS is the first tool to leverage hardware kernel isolation (SMAP/PAN) for double fetch detection in COTS RTOS, achieving 173x faster analysis than emulation
• 43 vulnerabilities discovered across QNX, VxWorks, and seL4, with 39 CVEs assigned
• A 19-year-old QNX vulnerability enables local privilege escalation in production vehicles with 76% exploit success rate in 125 CPU cycles
• System call lifecycle tracking with unique pairing IDs eliminates preemption-induced false positives entirely (vs. 87.7% false positive rate with QEMU-TCG)
• RTOS vendors intentionally disable hardware kernel isolation for performance, creating an exploitable gap
• Cross-platform portability demonstrated with 8 hours to 2 work days porting effort across QNX, VxWorks, and seL4

About the Speaker(s)

The paper was presented by a surrogate speaker on behalf of Yingjie Cao and collaborators, who could not attend due to visa issues. The presenter delivered a detailed and well-structured presentation of the technical approach and results, noting that contact information for the authors is available in the paper for follow-up questions.