The Angry Path to Zen: AMD Zen Microcode Tools and Insights

The Angry Path to Zen: AMD Zen Microcode Tools and Insights

Speakers: Benjamin Kollenda

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=GrBSH2N5-lc

Overview

Benjamin Kollenda’s talk, "The Angry Path to Zen: AMD Zen Microcode Tools and Insights," presents a deep dive into the opaque world of AMD Zen microcode. Building on previous work on K8/K10 architectures and drawing inspiration from contemporary research like Google's Zenbleed and related tooling, Kollenda introduces a powerful open-source framework for reverse engineering, analyzing, and experimenting with AMD Zen CPUs at their most fundamental level. The talk elucidates the intricate mechanisms by which complex x64 instructions are translated into internal micro-operations and how microcode updates can fundamentally alter CPU behavior.

The significance of this research lies in its contribution to understanding the core execution engine of modern AMD processors. By developing tools like Angry and zenutils, Kollenda and his team have demystified a critical component of CPU operation, which is often considered a black box. This work provides researchers and security professionals with unprecedented access to the microarchitectural layer, enabling a more thorough analysis of potential vulnerabilities, performance characteristics, and the underlying logic that governs CPU functionality.

Ultimately, this presentation is a testament to the power of reverse engineering in uncovering hidden complexities within proprietary hardware. It demonstrates how a combination of software tooling and, in some cases, physical analysis, can peel back layers of abstraction to reveal the true nature of a system. For anyone interested in CPU security, low-level system internals, or the art of reverse engineering, this talk offers profound insights and practical methodologies for exploring the "angry path" to understanding the Zen microarchitecture.

Background

▶ Watch: Introduction, speaker background, and talk overview (0:00)

The journey into AMD Zen microcode is rooted in a long-standing challenge in computer architecture: the undocumented and often proprietary nature of CPU internal execution. Modern x64 processors, especially those from AMD and Intel, translate complex, variable-length x64 instructions into simpler, fixed-length internal micro-operations (micro-ops or μops). This translation occurs via a microcode decoder, which acts as a virtual machine within the CPU. Crucially, this microcode is not static; it can be updated post-fabrication through microcode updates, often delivered via BIOS/UEFI or operating system loaders. These updates can fix bugs, patch security vulnerabilities, or even introduce new functionalities.

Prior work by Benjamin Kollenda on AMD K8 and K10 microcode, presented at previous C3 conferences, laid some groundwork, but the Zen architecture presented new challenges. Concurrently, Google's Project Zero released Zentool and disclosed vulnerabilities like Zenbleed (CVE-2023-20593), highlighting the critical security implications of microcode. These developments underscored the need for robust, open-source tools to explore Zen microcode. The problem that Kollenda's team aimed to address was the lack of a controlled, low-noise environment to precisely test microcode and observe its effects, particularly given the shift away from easily accessible serial ports on modern mainboards.

The goal was to build a system that could:

1. Control execution: Precisely dictate every instruction executed on the CPU.
2. Isolate testing: Run tests on individual CPU cores without interference from the operating system or other cores.
3. Recover from failures: Quickly reset and re-test, minimizing system reboots when microcode experiments inevitably cause crashes (page faults, general protection faults).
4. Expandability: A framework that could be easily extended to support new architectures and testing methodologies.
5. Ease of use: Simple to build, deploy, and script for automated testing.

This formed the impetus for Angry and zenutils, designed to provide unprecedented visibility and control over AMD Zen microcode, moving beyond reliance on proprietary tools or the limited insights offered by public microcode updates.

Key Findings

▶ Watch: Introducing 'Angry' UEFI application for CPU control (3:40)

The research presented by Benjamin Kollenda yielded several critical findings, significantly advancing the understanding and analysis of AMD Zen microcode:

• Development of Angry and zenutils: The talk's primary contribution is the creation and open-sourcing of two powerful tools:
• Angry: A UEFI-based testing framework that runs directly on bare metal, providing a highly controlled, low-noise environment for microcode execution and analysis. It enables asynchronous testing on multiple CPU cores and robust error recovery.
• zenutils: A custom assembler, disassembler, and macro assembler for AMD Zen microcode. This tool allows researchers to write, compile, and decompile microcode instructions, providing the necessary interface to interact with the CPU's internal micro-engine.
• Microcode Update Mechanics Confirmed: The research validated that microcode updates function by replacing existing instructions in the CPU's Microcode ROM with new content supplied in the update. This replacement is governed by match registers (also referred to as "hooks"), which specify the addresses in the ROM to be overwritten. This mechanism allows for fundamental alterations of CPU behavior.
• Turing-Complete Micro-Engine: The microcode instruction set was found to support a rich set of operations, including arithmetic, memory access (load/store), and conditional control flow. Kollenda demonstrated the ability to implement a Turing-complete virtual machine (specifically, a Subleq computer) entirely within the microcode, proving its computational universality.
• Architectural Variations Across Zen Generations: Significant differences were observed in microcode behavior and update structures across different Zen generations (Zen 1, Zen 2, Zen 3, Zen 4, Zen 5). These variations include the size of microcode updates (e.g., Zen 1 uses 64 quads, Zen 3 uses 128, Zen 5 is "huge"), specific bit requirements for execution units (e.g., Zen 3), and differing instruction support.
• Physical Address Access: Microcode instructions can directly access memory using physical addresses, bypassing the CPU's Memory Management Unit (MMU) and potentially ignoring memory permissions. This capability highlights a powerful, low-level control mechanism.
• Physical Die Analysis and Bit Reconstruction: Through collaborative work with "Tris," the talk highlighted the use of electron microscopes to image CPU dies, reconstruct the physical bits of the microcode ROM, and thereby obtain a complete dump of the internal microcode. This physical validation significantly expanded the understanding of microcode encoding, moving from hundreds of instructions in updates to thousands from the ROM.
• Internal Decryption Algorithms: The physical ROM dump confirmed the presence of internal decryption algorithms, specifically xxtier, whose key was also obtained. This demonstrates the complex security mechanisms integrated into the CPU at the microcode level.

These findings collectively provide a foundational understanding of AMD Zen microcode, offering both theoretical insights into CPU design and practical tools for further research and security analysis.

Technical Deep Dive

▶ Watch: Angry's key APIs: microcode application, IBS, and DSM (4:40)

The core of Benjamin Kollenda's presentation lies in the intricate technical details of the tools developed and the insights gained into AMD Zen microcode.

Angry - The UEFI-based Testing Framework:

Angry is a UEFI application designed for bare-metal CPU testing. It boots from a USB drive, acting as a minimal operating system substitute. Key features include:

• Network Stack: Leverages UEFI's built-in network stack, eliminating the need for complex network drivers and enabling TCP communication.
• Multi-Processor Support: On boot, Angry initializes Application Processors (APs), which are then used to run supplied instructions. The boot core typically hosts a TCP server that listens for commands.
• Asynchronous Execution: Tests can be run in parallel on multiple APs, with results reported back to the client. This dramatically speeds up testing workflows.
• Robust Recovery: APs can recover from common faults like page faults and general protection faults, writing error information and waiting for the next job, thus minimizing full system reboots.
• APIs: Angry provides APIs for:
• Applying microcode updates.
• Running x64 code on specific cores.
• Higher-level services via ASRs (Angle Service Routes), akin to basic syscalls, including Instruction Based Sampling (IBS) for microcode internals and DSM for even deeper insights.
• Python Client Framework: A client-side Python framework connects to Angry via TCP. It manages test setups, tracks known quirks (e.g., power issues), and provides a high-level API. It integrates with Zentool (for signing microcode updates) and zenutils.
• Power Control: An Angry ESP board, communicating via Home Assistant, allows for remote power cycling or resetting of test setups that become unresponsive, crucial for automated testing.

zenutils - The Microcode Assembler/Disassembler:

zenutils is a custom toolset for working with Zen microcode:

• Single-Line Assembler: Converts a single microcode instruction into its binary representation.
• Disassembler: Deconstructs binary microcode into human-readable assembly. It can disassemble entire updates or even the full ROM if available.
• Macro Assembler with Label Support: Allows writing complex microcode programs using labels for loops and conditional branches, abstracting away relocation and slot management.
• YAML Architecture Definitions: Microcode architectures vary significantly across Zen generations (e.g., Zen 1/2 vs. Zen 3/4/5). zenutils uses YAML definitions to capture these differences, such as update size (Zen 1: 64 quads; Zen 3: 128 quads; Zen 5: "huge") and specific bit requirements for execution units (e.g., Zen 3).
• Python API: Enables scripting of microcode assembly and disassembly, integrating seamlessly with Angry.

Microcode Instruction Examples:

Kollenda provided two key examples of microcode programs written using zenutils:

1. cring_length: This macro assembly snippet calculates the length of a C string.

• Directives: Specifies the target architecture (e.g., zen2) and a revision number. The revision is crucial for the CPU to determine if an update should be applied (only accepts same or higher revision). Setting it to FF ensures it always applies.
• Hooking: The microcode is executed by hooking a specific x64 instruction. In this case, SHLD (opcode 0x0420), an instruction rarely used by Linux, was chosen to avoid system crashes. This instruction serves as the entry point for the custom microcode.
• Internal Registers: Microcode operates on 16 internal registers (e.g., R15), distinct from x64 registers. Registers below R7 have special uses.
• Labels and Control Flow: Supports labels for loops and conditional branches (JNE, JEQ).
• Memory Access: LOAD and STORE instructions require specifying data width (e.g., BYTE, QWORD). Segment annotations (e.g., LOGICAL, PHYSICAL) are used, allowing the micro-engine to bypass MMU translation and directly access physical memory.
• Flags: Microcode has its own internal flags (e.g., zero flag, carry flag), distinct from the x64 flags. Instructions need to explicitly specify which flags to set or read.
• Sequence Words: Instructions are packaged into quads (four instructions + one sequence word). SEQUENCE_COMPLETE is a sequence word that tells the microcode sequencer to finish decoding the current x64 instruction and fetch the next one.

1. subleq (Subtract if Less or Equal, then Jump): A more complex example demonstrating a Turing-complete VM.

• Core Context: The Angry framework sets RBX to point to a core context structure, which contains internal information and pointers, including a result_buffer for returning data.
• Complex Control Flow: Shows multiple flags being set and all 16 variants of conditional jumps.
• Custom Exit: A custom extension allows the subleq VM to exit by jumping to minus one, returning control to Angry.

Physical Analysis and ROM Disassembly:

Tris's work involved imaging CPU dies using a scanning electron microscope to physically reconstruct the bits of the microcode ROM.

• Die Imaging: Nanometers of substrate are carefully removed to reach specific layers, allowing imaging of the physical transistors representing bits.
• Bit Reconstruction: Images are analyzed to determine bit values (bright for one, dark for zero). This requires reordering and significant processing.
• Community Correction: A web-based platform allows multiple researchers to zoom into specific bit locations and correct readout errors, improving accuracy.
• Validation: The reconstructed ROM allowed for disassembly, revealing internal algorithms like xxtier (a decryption algorithm). Following these algorithms validated the mapping between virtual addresses (used in microcode updates) and physical bit locations. This process significantly enhanced the zenutils database of known micro-instructions.

This intricate blend of software development, reverse engineering, and physical analysis has created an unparalleled toolkit for exploring the depths of AMD Zen microcode.

Demo / Proof of Concept

▶ Watch: Python client framework and Zen tool integration (5:20)

While a live demonstration was planned for the conference, it was unfortunately thwarted by internet connectivity issues (attributing blame to "digitalizong in Germany"). However, Benjamin Kollenda presented the anglicat Python code that would have driven the demo, illustrating the workflow for running custom microcode on an AMD Zen CPU using the Angry framework.

The demonstration code showcased the following steps:

1. Import necessary modules from anglicat.
2. Define machine code: A standard x64 instruction sequence is prepared. This typically includes setting up a memory buffer and the specific x64 instruction that will be hooked to trigger the custom microcode. In the subleq example, this machine code sets up some data fields and then executes the subleq VM via the hooked instruction.
3. Define microcode: The custom microcode, written in zenutils macro assembly, is provided as a text string (e.g., the subleq implementation).
4. Select a test setup: anglicat is instructed to select an available Zen 1 setup. The speaker noted that if multiple setups are available, anglicat can pick a free one.
5. Optional Reboot: The code could issue a command to reboot the selected system. This is done by anglicat calling out to a Home Assistant instance, which then signals the Angry ESP board to physically pull the reset pin of the target system. This is crucial for recovering from hard crashes caused by experimental microcode.
6. Execute the update: The core API call is run_update(ucode_text, mcode_text, reuse_core=False).

• ucode_text: The microcode to be applied, in zenutils macro assembly format.
• mcode_text: The x64 machine code that will execute the microcode.
• reuse_core=False: An optional flag that tells anglicat not to reuse the core for subsequent tests if it detects a potential issue, such as a broken microcode updater. If reuse_core is True (default), anglicat will continue using the core if it detects a successful response, optimizing for speed. If reuse_core is False, or if a core fails, anglicat will move to the next available core. If all cores are exhausted (e.g., a six-core CPU has five testable cores plus the boot core), the system will automatically reboot.

1. Process the response: anglicat returns a response object.

• Failure: If the core encounters an error (e.g., page fault, general protection fault, timeout due to an endless loop), the response will indicate failure, providing the interrupt number and any available register information.
• Success: If successful, the response includes a result_buffer, which contains data returned by the custom microcode. In the subleq example, the code extracts 8 bytes from this buffer, representing the subleq result, and prints it.

This detailed walkthrough of the anglicat client code effectively served as a proof of concept, demonstrating how researchers can programmatically control AMD Zen CPUs at the microcode level, deploy custom code, and retrieve results, even without a live demo. Kollenda also noted that thousands of microcode updates have been run using this setup without permanently damaging CPUs, though hot resets were sometimes necessary.

Defensive Implications

▶ Watch: Custom power control board and physical lab setup (6:15)

The detailed exploration of AMD Zen microcode presented in "The Angry Path to Zen" carries significant defensive implications for system security, CPU design, and vulnerability research.

1. Understanding CPU Vulnerabilities: The ability to reverse engineer and run custom microcode is paramount for understanding and mitigating CPU-level vulnerabilities. Flaws like Zenbleed (CVE-2023-20593), which exploited microarchitectural details, highlight that a deep understanding of microcode execution is essential for both discovery and patching. Defenders can use tools like Angry and zenutils to analyze official microcode updates, verify their fixes, and potentially uncover new attack surfaces previously opaque to researchers.

1. Microcode as an Attack Vector: While applying forged microcode updates currently requires known keys (which are typically proprietary and secured), the research demonstrates the profound impact such an update can have. If an attacker were to gain control over the microcode update mechanism (e.g., through a firmware vulnerability or a compromised update signing key), they could effectively reprogram the CPU at its most fundamental level. This could lead to:

• Persistent Backdoors: Injecting malicious logic that persists across reboots, below the visibility of operating systems or hypervisvisors.
• Information Leakage: Microcode capable of exfiltrating sensitive data from registers, memory, or even other cores.
• Privilege Escalation: Bypassing security mechanisms, including MMUs and protection rings, through direct physical memory access.
• Hardware Trojans: Altering CPU behavior to facilitate other attacks or degrade system integrity.

1. Importance of Supply Chain Security: The talk implicitly underscores the critical importance of securing the microcode supply chain. From CPU fabrication to firmware distribution, every stage where microcode updates are handled or stored represents a potential point of compromise. Robust signing, verification, and integrity checks are non-negotiable.

1. Advanced Fuzzing and Testing: Angry provides a "low-noise" environment ideal for advanced fuzzing and architectural testing. Defenders can leverage this framework to:

• Test CPU resilience: Subject CPUs to malformed or unexpected microcode sequences to identify robustness issues or undocumented behaviors.
• Validate security patches: Precisely test whether a microcode update effectively mitigates a reported vulnerability without introducing new regressions.
• Performance Analysis: While not the primary focus of this talk, Angry's capabilities, including Instruction Based Sampling, can be used to analyze the performance implications of microcode changes or to detect microarchitectural side channels.

1. Need for Transparency and Documentation: The challenges faced by researchers in reverse engineering microcode highlight the broader issue of transparency in CPU design. While full disclosure of proprietary IP is unlikely, a more documented interface or a standardized, verifiable microcode format could significantly aid defensive efforts by allowing broader security scrutiny.

In conclusion, this research empowers defenders by demystifying a critical component of CPU operation. It provides the tools and methodologies necessary to conduct in-depth security analysis at the microarchitectural level, fostering a more proactive and informed approach to CPU security.

Key Takeaways

• AMD Zen Microcode is a Powerful, Undocumented VM: The talk conclusively demonstrates that AMD Zen microcode functions as a Turing-complete virtual machine within the CPU, capable of complex logic, arithmetic, and control flow, operating largely outside the visibility of standard software.
• Angry and zenutils are Essential Open-Source Tools: The development of the Angry UEFI testing framework and the zenutils assembler/disassembler provides critical, open-source resources for researchers to interact with, analyze, and experiment with AMD Zen microcode on bare metal.
• Microcode Updates Fundamentally Alter CPU Behavior: Microcode updates replace portions of the CPU's internal ROM via "match registers" (hooks), enabling profound changes to how x64 instructions are executed, with direct implications for security and functionality.
• Physical Analysis Validates and Expands Understanding: The use of electron microscopes to image and reconstruct the microcode ROM from CPU dies significantly expanded the database of known micro-instructions and validated software-based reverse engineering efforts.
• Direct Physical Memory Access is Possible: Microcode instructions can directly access physical memory, bypassing the MMU and potentially memory permissions, underscoring the extreme privilege level of microcode execution.
• Deep CPU Security Research is Crucial: This work highlights the necessity of continuous, low-level security research into CPU microarchitectures to identify, understand, and mitigate vulnerabilities that lie beneath the traditional software layers.

About the Speaker(s)

Benjamin Kollenda is a versatile technologist with a broad range of expertise spanning DevSecOps, security, and embedded systems. He describes himself as a "jack of all trades, master of some." Kollenda has a history of engaging with CPU microcode, having previously presented on AMD K8 and K10 architectures at past C3 conferences, approximately seven years prior to this talk. Currently, he works at Grav Nebula, a company that primarily focuses on "tower stuff." He explicitly states that the opinions and research presented in his talk are solely his own and do not represent those of his employer.