Of Boot Vectors and Double Glitches: Bypassing RP2350's Secure Boot

Of Boot Vectors and Double Glitches: Bypassing RP2350's Secure Boot

Speakers: stacksmashing; nsr

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=V5KvW4elzXU

Overview

In an engaging and highly technical presentation at 39C3, security researchers Marius Muench (NNSA) and Thomas Roth (stacksmashing) unveiled their findings from the Raspberry Pi RP2350 hacking challenge. Their talk, "Of Boot Vectors and Double Glitches: Bypassing RP2350's Secure Boot," detailed two distinct and sophisticated fault injection attacks that successfully circumvented the secure boot mechanisms of the RP2350 microcontroller. This chip, known for its low cost ($1) and extensive security features like TrustZone-M, glitch detectors, a redundancy coprocessor (RCP), secure boot, and One-Time Programmable (OTP) memory, was designed with a "security through transparency" philosophy, openly releasing its datasheet and boot ROM source code.

The motivation behind this research was the RP2350 hacking challenge, initiated by Raspberry Pi itself at DEF CON, offering a $10,000 bounty to anyone who could extract a hidden 16-byte secret from the chip's securely locked OTP memory. This challenge explicitly permitted fault injection attacks, making it a fertile ground for hardware security exploration. Muench and Roth, along with other researchers, demonstrated that even a robust, multi-layered security architecture, often conceptualized as a "Swiss cheese model" of overlapping defenses, can be penetrated with precise and innovative fault injection techniques. Their work underscores the critical importance of scrutinizing every aspect of a chip's boot process and the inherent vulnerabilities that can arise even with extensive hardware mitigations.

Background

▶ Watch: Introduction to RP2350 and talk agenda (0:00)

The RP2350 is a remarkable microcontroller developed by Raspberry Pi, featuring a dual-core architecture with a Cortex-M33 core and a RISC-V Hazard 3 core, both capable of running at 150 MHz. Beyond its processing capabilities, the chip integrates a suite of advanced security features typically found in much more expensive hardware. These include TrustZone-M for secure and non-secure execution environments, active glitch detectors to prevent tampering, a custom-designed Redundancy Coprocessor (RCP), secure boot to ensure only signed firmware runs, and One-Time Programmable (OTP) memory for immutable configuration. The RP2350's affordability at just $1 makes its security posture particularly noteworthy, as it aims to bring advanced protection to a wider range of embedded applications.

A key focus of the research was fault injection, specifically voltage fault injection or "glitching." This technique involves momentarily dropping the chip's power supply for a few nanoseconds at a precisely calculated moment to induce a computational error in a single instruction. Historically a lab-intensive attack, fault injection has become more accessible, with even the RP2350's predecessor, the RP2040, being widely used as a fault injection tool due to its precise I/O timing. The speakers highlighted its use in Nintendo Switch mod chips as an example of its real-world application.

Raspberry Pi's commitment to "security through transparency" was a cornerstone of the hacking challenge. They published an extensive 1,357-page datasheet for the RP2350, openly released the source code and binary artifacts for the boot ROM, and made the Hazard 3 RISC-V core design open source on GitHub. This unprecedented level of transparency greatly aided researchers by providing detailed insights into the chip's internal workings and security mechanisms, facilitating the discovery of vulnerabilities.

The RP2350's security architecture is built upon five primary countermeasures, forming a "Swiss cheese model" where multiple layers of defense are intended to compensate for individual weaknesses:

1. One-Time Programmable (OTP) Memory: This antifuse memory can be written to only once and stores critical configurations, including core enablement (ARM/RISC-V), debug access status, glitch detector strength, and, crucially, whether secure boot is enabled. Once secure boot is enabled, it cannot be disabled. The OTP also stores hashes of cryptographic keys for firmware signing and supports locking pages to prevent further writes or even reads of sensitive data. The secret for the hacking challenge was stored here, marked as non-readable.

1. Redundancy Coprocessor (RCP): A custom-designed coprocessor by Raspberry Pi, the RCP offers several hardware-enforced security features:

• Hardware-enforced stack canaries: Protecting against stack-based buffer overflows without software implementation.
• Randomized delays: Introducing unpredictable timing to critical operations, making precise fault injection attacks more difficult to synchronize.
• Sequence count checking: Verifying that code executes in its intended order, complicating code reuse attacks.
• In-hardware integer validation: Checking register values for consistency, designed to detect bit-flips caused by glitching (e.g., forcing values to all zeros or ones).
• Panic functionality: Immediately stalling execution and rebooting the chip if any security violation is detected.

1. Glitch Detectors: Unique in the commercial microcontroller space, these hardware components are designed to detect voltage fault injection attempts. They consist of two D flip-flops clocked by the system, with a slight, configurable delay between them. The idea is that a fault would likely affect only one flip-flop, causing a mismatch in their outputs, triggering a chip reset. The sensitivity of these detectors is configurable via OTP.

1. Hardened Boot ROM: The core bootloader code stored in read-only memory is specifically designed and implemented with fault injection resilience in mind, aiming to make critical security checks robust against glitches.

1. Secure Boot: The overarching mechanism ensures that only cryptographically signed firmware can execute on the chip. This verification occurs during boot from ROM, OTP, or flash.

The RP2350 hacking challenge was launched at DEF CON, where Raspberry Pi provided special "playground" boards with exposed voltage rails and debug points, encouraging researchers to find vulnerabilities. The objective was straightforward: read out 16 bytes of secret data from the OTP memory, which was locked and marked as non-readable, while secure boot was enabled. This represented a substantial bounty of $625 per byte, igniting a multi-month hacking effort. The challenge results, announced in January, revealed five distinct attacks, two of which were developed by Muench and Roth, highlighting the effectiveness of the bounty program in uncovering subtle hardware vulnerabilities.

Key Findings

▶ Watch: Overview of five discovered RP2350 vulnerabilities (11:00)

The RP2350 hacking challenge ultimately yielded five distinct methods to bypass its secure boot and extract the hidden secret, underscoring the complexity of securing modern microcontrollers against physical attacks. The speakers focused on two of these, which they personally discovered:

1. Faulting a Vector Boot: This attack exploited a subtle vulnerability in the secure boot process, specifically within the handling of reboot types and the interaction between the non-secure and secure execution states. By precisely glitching a conditional branch instruction in the boot ROM, the attackers could force the chip to execute arbitrary code from a user-controlled program counter and stack pointer, completely bypassing secure boot validation. A critical enabler for this attack was the observation that the USB bootloader, running at a lower clock speed (48 MHz compared to the normal 150 MHz), effectively weakened the chip's integrated glitch detectors, making successful fault injection more feasible.

1. One-Time Programmable Memory Double Glitch: This more advanced attack targeted the OTP lockdown mechanism itself. The boot ROM contains a function responsible for setting the OTP lock states (read-write, read-only, inaccessible). This function was heavily hardened with RCP checks and randomized delays. The vulnerability lay in a pair of logical shift instructions (LSL, LSR) intended to clear upper bits of the lock value. By inducing two precisely timed electromagnetic faults (a "double glitch") to skip the second shift instruction, the attackers could prevent the OTP from being fully locked down, allowing the secret to be read out via picotool. The use of Electromagnetic Fault Injection (EMFI) was crucial here, offering greater precision and localization to bypass the glitch detectors and overcome the challenges posed by random delays.

Beyond their own discoveries, the speakers briefly highlighted the other three successful attacks reported to Raspberry Pi:

• Attacking the OTP Power-On State Machine: Discovered by Eden Cullen, this involved power-faulting the OTP's separate power rail during hardware bring-up, leading to a misconfiguration that enabled the RISC-V core with debug access.
• Laser Fault Injection: Kevin Curdysis developed an "insanely cool" setup using a 3D-printed laser fault injection rig to physically Dremel off the back of the chip and blast it with infrared lasers, achieving a security bypass.
• Focused Ion Beam (FIB) Extraction: Andrew Sonnenberg and his team at IOActive used advanced microscopy techniques, specifically FIB and voltage contrast imaging, to literally "read out" the bit patterns stored in the antifuse memory of the OTP, albeit at a significant cost ($50,000 in microscope time to win $10,000).

These findings collectively demonstrate that despite sophisticated hardware security features and a transparent design process, microcontrollers remain vulnerable to determined and creative physical attacks. The rapid response from Raspberry Pi, fixing these issues in new chip revisions and openly publishing errata, highlights the positive impact of such challenges and the "security through transparency" model.

Technical Deep Dive

▶ Watch: Attack 1: Glitching OTP memory to read locked pages (12:30)

The RP2350's boot process is a meticulously documented, multi-stage sequence, starting with hardware power-on and transitioning to software execution from the boot ROM. The speakers' first attack, "Faulting a Vector Boot," centered on exploiting a critical window during this intricate boot flow. Early in the boot ROM's execution, the chip checks a power manager vector. Marius Muench noted with concern that this vector could theoretically allow booting from a user-defined program counter (PC) and stack pointer (SP), potentially bypassing secure boot validation if written to early enough. Raspberry Pi initially dismissed this as not being an issue, but further investigation proved otherwise.

The standard boot sequence attempts to load firmware from ROM, then OTP, and finally flash. In all these cases, if secure boot is enabled, the firmware's cryptographic signature is rigorously verified. If none of these methods succeed, or if the boot_sel button is pressed, the chip enters a USB bootloader or UART bootloader mode. Critically, the USB bootloader, which presents a significant attack surface due to its complex USB stack, runs in the non-secure execution state of TrustZone-M. This is a good design choice, as any bugs within the USB bootloader would not immediately grant access to the secure world or the protected OTP secret.

The USB bootloader exposes various APIs for firmware developers, including flash operations, OTP accesses, and reboot functionalities. picotool, a command-line utility, interacts with the chip via this USB bootloader, enabling firmware flashing and chip resets. The RP2350 supports multiple reboot types, but one in particular, PCSP (reboot to a specific program counter and stack pointer), is explicitly "not allowed in the ARM non-secure variant." This restriction is crucial: a non-secure entity (like the USB bootloader) should not be able to dictate the exact entry point of the secure boot process. Under the hood, this PCSP reboot type utilizes the special boot vectors Muench initially identified.

The reset command flow is designed to prevent this PCSP abuse. When a reboot command is received via the non-secure USB bootloader, execution transitions to the secure world reboot entry via secure call gateways. The first action in the secure world is argument sanitization, which includes checking the requested reboot type. This check is hardened, split across multiple instructions to resist single-point glitches. If a non-secure request for PCSP reboot is detected, it should fail here. Following sanitization, the code jumps to a shared reset code (the boot ROM API reset), which dispatches various reboot types. As an additional countermeasure, this shared code performs another verification, ensuring that the boot type value, stored in two different ARM registers, consistently matches. Only after passing these multiple, hardened checks is the requested boot type executed.

The vulnerability for the Vector Boot attack lay in the assembly of the shared reset code. Specifically, a comparison instruction (CMP R5, #0xD) checks if the boot type in register R5 is 0xD, which internally represents the PCSP reboot type. If R5 is not 0xD, a conditional branch (BNE) jumps to code handling other reboot types. Otherwise, execution falls through to the do_reboot_type_PCSP branch. The attack strategy was to glitch either the CMP instruction or the BNE (Branch if Not Equal) instruction. A successful glitch on either would cause the chip to fall through to the do_reboot_type_PCSP branch, even if the PCSP boot type was not legitimately requested from the non-secure world. This would allow an attacker to specify an arbitrary program counter and stack pointer, effectively hijacking the boot process.

A critical challenge was bypassing the glitch detectors. While effective at the RP2350's normal 150 MHz operating frequency, the USB bootloader operates at a lower 48 MHz clock speed for better USB synchronization. Muench's profiling revealed that at this lower clock rate, the window for glitch detection widened, making it possible to inject faults that went undetected. The attack strategy involved loading unsigned firmware into RAM, sending a USB reset command with the desired PC/SP parameters (even though the PCSP type was disallowed), and then glitching the critical CMP or BNE instruction. If successful, the chip would reboot into the attacker's unsigned firmware, which could then read the OTP secret.

The second attack, the OTP Double Glitch, targeted the mechanism that locks down the OTP memory. During early boot, the boot ROM needs read access to the OTP. However, as the boot progresses, the OTP pages containing sensitive data are locked down to read-only or inaccessible states. This lock progression is well-documented in the datasheet. There are distinct lock states for the secure world, non-secure world, and the pico bootloader. The relevant function for securing OTP pages is sotp_advanced_bootloader_to_secure_value in the boot ROM.

This function is heavily protected. It includes RCP interactions at its beginning and end, performing stack canary checks and, more importantly, introducing random delays between 0 and 127 clock cycles. These random delays are a significant countermeasure against fault injection, as they make it extremely difficult to precisely time a glitch to hit a specific instruction. The core vulnerability was found in two seemingly innocuous instructions: a Logical Shift Left (LSL) followed by a Logical Shift Right (LSR). These shifts were intended to clear any unexpected upper bits in the 32-bit lock value. If an attacker could glitch and skip the LSR instruction after the LSL, the upper bits of the lock value would remain set. Crucially, the hardware would still accept this modified lock value, effectively keeping the OTP unlocked.

The function's hardening meant a single glitch was insufficient. An RCP comparison at the end of the function checked if multiple internal operations returned consistent values, necessitating a "double glitch" to bypass these checks. Given the random delays and the need for two precise glitches, traditional voltage fault injection was deemed unreliable. Thomas Roth turned to Electromagnetic Fault Injection (EMFI). This technique uses a high-voltage pulse generator and a small coil to generate a localized electromagnetic field that collapses into the chip, inducing faults. EMFI offers precise spatial localization, allowing attackers to target specific areas of the chip. Initially, Roth observed that EMFI only affected the bottom-left corner of the chip (where the internal voltage regulator resides), effectively performing voltage fault injection via EMFI. However, by switching to a counter-clockwise polarized coil, he achieved 100% bypasses of the glitch detectors in other target areas.

The random delays still posed a challenge for EMFI's single trigger. Roth discovered that the RCP delays themselves caused measurable dips in the chip's power consumption, which could be used as a reliable trigger point for EMFI. While the random delays meant precise timing was still difficult, a statistical approach (trying the middle value of the delay range) eventually yielded success. The attack strategy for the OTP double glitch involved setting the boot type to boot select mode, bypassing the glitch detectors using EMFI, performing two precisely timed EMFI glitches on the LSL/LSR instructions, and then using picotool to read the now-unlocked OTP secret. This successfully extracted the "coffee" secret, demonstrating a complete bypass of the OTP lockdown.

Demo / Proof of Concept

▶ Watch: Advanced double glitch to disable hardware detectors (20:30)

The presentation detailed the practical challenges and eventual successes in demonstrating these attacks, highlighting the iterative and often frustrating nature of hardware security research.

For the Vector Boot attack, Marius Muench's initial attempts at live glitching proved difficult. Despite having a solid theoretical understanding and profiling the chip's glitch detector behavior at different clock speeds (confirming vulnerability at 48 MHz), achieving a reliable, live voltage glitch on the specific CMP or BNE instruction in the boot ROM was elusive. Muench's setup became increasingly complex, incorporating custom trigger hardware, a logic analyzer to monitor USB communication, and meticulous wiring. He faced several practical hurdles, including the impact of poor quality jumper wires on glitch effectiveness, undersampling issues with his logic analyzer, and even the physical reset button falling off his development board due to frequent power cycling.

To verify the attack's feasibility, Muench resorted to a controlled environment using a debugger (GDB) on a fully controlled chip. By setting a breakpoint at the critical instruction, manually skipping it, and having the attack code in place, he confirmed that the logical flow would indeed lead to execution of his arbitrary code. Although a reliable physical glitch remained elusive for him, Raspberry Pi, upon reviewing his detailed write-up and evidence, acknowledged the theoretical exploitability of the "false branch" and accepted it as a valid entry in the challenge, validating his findings and "keeping his sanity."

The OTP Double Glitch attack saw a more direct, albeit still challenging, proof of concept thanks to Thomas Roth's expertise with EMFI. His setup involved a sophisticated CNC machine to precisely position an EMFI coil over the RP2350 chip. The EMFI system comprised a ChipSHOWER high-voltage pulse generator connected to custom coils. Initial EMFI attempts, however, were misleading, as the electromagnetic pulses primarily affected the chip's internal voltage regulator, essentially causing voltage fault injection via EMFI. This highlighted the importance of understanding the physical interaction of the fault injection mechanism with the target hardware.

The breakthrough came with using a counter-clockwise polarized EMFI coil, which allowed Roth to achieve 100% bypasses on the glitch detectors in specific, non-voltage-regulator areas of the chip. To address the random delays introduced by the RCP, Roth ingeniously used the power dips caused by these delays as a reliable trigger for his single EMFI pulse. While the double glitch still required statistical attempts due to the randomness, the improved localization and detector bypass of EMFI significantly increased the success rate. Eventually, after numerous attempts, the EMFI setup successfully performed the double glitch, prevented the OTP from locking down, and allowed the researchers to read out the secret, which was humorously displayed as "coffee" in the OTP memory. This demonstrated a complete, end-to-end exploitation of the double glitch vulnerability, serving as a robust proof of concept for the attack.

Defensive Implications

▶ Watch: Lessons learned and Raspberry Pi's security response (24:00)

The findings from the RP2350 hacking challenge, particularly the two attacks detailed by Muench and Roth, offer crucial lessons for hardware designers and security practitioners aiming to build robust embedded systems. The speakers themselves summarized key defensive implications:

1. Comprehensive Fault Models: It is imperative to include every reboot path and all interactions with external IP (Intellectual Property) in fault models. The Vector Boot attack demonstrated that even seemingly restricted boot types or subtle logic flaws in shared boot ROM code can create critical bypasses if not thoroughly hardened against fault injection.
2. Hardware Mitigations Can Fail: The "Swiss cheese model" of layered defenses is valuable, but it's not foolproof. The RP2350 featured multiple sophisticated hardware mitigations—glitch detectors, RCP with randomized delays and checks, hardened boot ROM—yet determined attackers, given enough attempts and the right tools, found bypasses. This underscores that while mitigations increase the cost and complexity of attacks, they do not eliminate them.
3. Complexity Breeds Vulnerability: Complicated boot paths, especially those involving transitions between secure and non-secure execution states or multiple stages of argument sanitization, are inherently difficult to secure. Each check and transition point introduces potential attack surfaces, and a single missed hardening opportunity can compromise the entire chain.
4. Clock Domain Analysis: The Vector Boot attack highlighted that clock domain differences can weaken countermeasures. Glitch detectors tuned for a high operating frequency (150 MHz) were less effective when the target code (USB bootloader) ran at a significantly lower frequency (48 MHz). Defenders must consider the effectiveness of fault detection mechanisms across all operational clock speeds and contexts.
5. Localized Fault Injection: EMFI's ability to precisely target specific physical areas of the chip, potentially bypassing physically localized glitch detectors, demonstrates the need for distributed and redundant fault detection mechanisms across the entire silicon die, rather than relying on a few central points.
6. Redundancy and Consistency Checks: While the RP2350 employed multiple checks (e.g., storing boot type in two registers, RCP comparisons), these can still be bypassed. The double glitch attack showed that even multi-instruction checks, if not perfectly designed, can be subverted by skipping specific operations (like the LSR after LSL). Designers should consider the atomic nature of security-critical operations and ensure that redundancy checks are truly independent and resilient to single or even double faults.
7. Transparency as a Security Asset: Raspberry Pi's "security through transparency" model, including open-source boot ROM and detailed datasheets, was instrumental in allowing researchers to identify these vulnerabilities. This approach fosters a community of security researchers who can contribute to finding and fixing flaws, leading to more robust designs. Indeed, all discovered vulnerabilities were quickly addressed in new chip revisions, with Raspberry Pi publishing open errata. This proactive stance significantly enhances the long-term security of their products.
8. The Value of Bug Bounties: Chip bounties, like the $10,000 challenge, motivate researchers to invest significant effort and creativity into finding vulnerabilities that might otherwise remain undiscovered, ultimately leading to more secure hardware.

In summary, defenders must adopt a holistic view of hardware security, anticipating sophisticated physical attacks, thoroughly reviewing all boot code paths, understanding the limitations of their countermeasures in various operating conditions, and embracing transparency to leverage external security expertise.

Key Takeaways

• The RP2350 is a low-cost microcontroller featuring advanced hardware security, including TrustZone-M, glitch detectors, a Redundancy Coprocessor (RCP), secure boot, and One-Time Programmable (OTP) memory, designed with a multi-layered "Swiss cheese model" of defense.
• The Raspberry Pi hacking challenge (offering a $10,000 bounty) demonstrated that even robust hardware security can be bypassed through sophisticated fault injection techniques, underscoring the importance of deep, independent security research.
• The Vector Boot attack exploited a timing vulnerability where the USB bootloader's lower clock speed (48 MHz) weakened glitch detection, allowing a precise voltage glitch on a CMP or BNE instruction in the secure boot path to force execution of attacker-controlled code from an arbitrary program counter and stack pointer.
• The OTP Double Glitch attack bypassed OTP lockdown by targeting a hardened function with two precisely timed Electromagnetic Fault Injection (EMFI) glitches on LSL/LSR instructions, preventing the OTP from fully locking down, even in the presence of randomized RCP delays and active glitch detectors.
• Transparency (open-source boot ROM, detailed datasheets) played a critical role in enabling researchers to understand the chip's inner workings and discover these vulnerabilities, leading to rapid fixes in new chip revisions and improved security.
• Hardware security research is an arduous, iterative process, often involving significant debugging challenges (e.g., faulty wires, logic analyzer settings, physical component failures) before theoretical exploits can be turned into practical demonstrations.

About the Speaker(s)

Marius Muench (NNSA) is an Assistant Professor at the University of Birmingham. His research primarily focuses on the security of embedded systems, delving into areas such as binary exploitation, micro-architectural exploits, and other low-level security challenges. The work presented in this talk was conducted as a side project, showcasing his dedication to hardware security beyond his academic duties.

Thomas Roth (stacksmashing) is a founder at hackstreet.io, a company he co-founded with LiveOverflow. He is a prolific security researcher known for his work in hacking various devices, often with a particular interest in the cheapest microcontrollers available. He has a reputation for trying to run Doom on as many devices as possible. Roth has previously spoken on topics like TrustZone-M, demonstrating his expertise in ARM-based embedded security.