Breaking architecture barriers: Running x86 games and apps on ARM

Breaking architecture barriers: Running x86 games and apps on ARM

Speakers: Tony Wasserka

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=3yDXyW1WERg

Overview

In a compelling presentation at 39C3, Tony Wasserka unveiled the intricacies and triumphs of Fex, an open-source x86 emulator designed to run proprietary x86 applications and games on ARM-based Linux systems. Wasserka, a seasoned software engineer with a background in GPU driver development and emulation, delved into the formidable technical challenges involved in achieving seamless cross-architecture compatibility. This talk highlighted how Fex is not merely an academic exercise but a critical enabler for the broader adoption of ARM hardware in the consumer space, particularly for Linux users.

The core problem Fex addresses is the inability to natively execute x86 binaries on ARM processors, a growing concern as ARM gains traction in laptops and other personal computing devices. While open-source software can often be recompiled for new architectures, proprietary applications like video games, Adobe Photoshop, or Microsoft Office suites remain locked to x86. Fex provides a robust solution by dynamically translating x86 machine code to ARM64, alongside a comprehensive compatibility layer that mimics an x86 kernel and system environment.

The significance of Fex extends beyond individual users; it has become a foundational technology for several high-profile projects. From Asahi Linux enabling Steam gaming on Apple Silicon Macs to Parallels Desktop integrating Fex into its x86 emulation backend, and even Valve leveraging it for the upcoming Steam Deck VR headset, Fex is proving instrumental in bridging the architecture gap. Wasserka's presentation served as both a technical deep dive into Fex's sophisticated mechanisms and a testament to the power of collaborative open-source engineering.

Background

▶ Watch: Introduction and speaker's journey into emulation (0:00)

The computing landscape is undergoing a significant shift towards ARM-based processors. While x86 architecture, dominated by Intel and AMD, has historically been the standard for personal computers, ARM's efficiency and performance gains have led to its widespread adoption in mobile devices and, increasingly, in laptops. Apple's transition to its M-series chips for MacBooks, and the emergence of Windows-on-ARM devices from vendors like Microsoft (Surface), Lenovo, and Dell, signal a clear trend.

For Linux users, adapting to new hardware architectures is often facilitated by the open-source nature of most software. The ability to recompile the world for a new instruction set architecture (ISA) like ARM64 means that many applications can run natively with optimal performance. However, this flexibility does not extend to proprietary software – commercial games, professional creative suites, or productivity applications – for which source code is unavailable. These applications are typically distributed as pre-compiled x86 binaries, rendering them incompatible with ARM-native systems.

This challenge spurred Ryan Houdek to initiate the Fex project seven years ago, aiming to create a robust solution for running x86 software on ARM Linux. Tony Wasserka joined the development team four years ago, bringing his extensive experience in low-level graphics and emulation. The project's goal is to present an entire x86 system stack to the application, from the CPU instruction set to the kernel's ABI and the system's libraries, all while running on native ARM hardware. This involves emulating not just the CPU, but also ensuring that kernel interactions, file system accesses, and library calls behave as an x86 application would expect.

Key Findings

▶ Watch: The challenge: proprietary x86 software on ARM Linux (3:40)

Fex has demonstrated remarkable success in bridging the architectural divide, positioning itself as a leading solution for x86-on-ARM emulation on Linux. A key finding is its ability to rival, and in some aspects even surpass, proprietary emulation layers like Apple's Rosetta for macOS or Microsoft's Prism for Windows on ARM. Notably, Fex was the first to implement AVX and AVX2 instruction sets in a software-powered emulator, showcasing its advanced capabilities and rapid development cycle.

The project's core contributions lie in its sophisticated technical approach to binary recompilation, memory model handling, and system-level compatibility. Fex's use of an Intermediate Representation (IR) for instruction translation enables deep optimization, ensuring that the translated ARM code is as efficient as possible. Furthermore, Fex has tackled the notoriously complex x86 memory model with innovative, multi-pronged strategies, including specific heuristics for common game engines like Unity, to mitigate performance penalties.

Beyond raw emulation, Fex has introduced crucial optimizations such as Vulkan library forwarding and code caching. These features significantly improve performance and user experience by bypassing emulation overhead for critical components and reducing loading times and micro-stuttering in games. The practical success of Fex is evidenced by its adoption by major projects: it enables Steam gaming on Asahi Linux for Apple Silicon, forms part of the x86 emulation backend for Parallels Desktop, is integrated into CodeWeavers' Wine development, and is slated to play a vital role in Valve's Steam Deck VR headset for x86 VR gaming. The project's evolution also includes ambitious plans for a more efficient Wine on ARM approach utilizing ARM64EC for Windows game emulation, further simplifying the execution stack and boosting performance.

Technical Deep Dive

▶ Watch: System architecture layers and emulation considerations (4:40)

The technical foundation of Fex is built upon several intricate layers, each designed to make an ARM system appear as a native x86 environment to the running application. The core components include a binary recompiler, a kernel compatibility layer, and sophisticated optimizations for memory and library handling.

Binary Recompiler

At the heart of Fex is its binary recompiler, responsible for translating x86 machine code into ARM64 instructions on the fly. This process is not a direct one-to-one mapping but involves an intermediate step to ensure efficiency.

1. Decoding: The x86 instructions are first decoded into Fex's internal Intermediate Representation (IR). This abstraction layer separates the instruction decoding from the architectural specifics of the target platform.
2. Optimization: Once in IR, the code can be optimized. This is crucial for performance, as direct translation often results in bloated or inefficient ARM code. Techniques like constant elimination are applied to minimize the number of operations.
3. ARM Code Generation: Finally, the optimized IR is translated into native ARM64 instructions. The goal is to produce ARM code that is as close as possible in size and efficiency to the original x86 input, minimizing the performance penalty.

This recompilation process faces several fundamental challenges:

• Static Overhead: Every time Fex enters or exits an emulated code block, there's an unavoidable overhead. While Fex strives to keep this minimal, it's a constant factor in performance.
• Flag Handling: x86 architecture extensively uses CPU flags (e.g., carry, zero, overflow) that are implicitly set by most arithmetic operations. Many applications, especially games, may not explicitly read these flags after every operation, but Fex, as a recompiler, often cannot statically determine this. Thus, it must unconditionally generate ARM code to simulate these x86 flag behaviors. For example, a simple decrement operation on x86 might require additional ARM instructions like C set and R M I F to correctly update flags that an ARM native subtraction wouldn't. Without IR-based optimization, this could lead to a massive increase in instruction count, from a few to potentially dozens of instructions per x86 operation.
• Instruction Set Complexity: The x86 instruction set is vast, comprising thousands of instructions, including numerous extensions like SSE, SSE2, AVX, and AVX2. Each instruction requires meticulous mapping to ARM, accounting for all corner cases, especially regarding overflows and other arithmetic intricacies, to ensure functional correctness and high performance.

Memory Model Emulation

One of the most "ugly" and challenging aspects of x86 emulation is its memory model. x86 provides strong memory ordering guarantees, meaning that writes to memory by one thread are immediately visible to others in the order they occurred. ARM, by contrast, has a weaker memory model, allowing processors to reorder memory operations for performance, which can lead to unexpected behavior in multithreaded x86 applications.

Wasserka illustrated this with a simple multithreaded example where one thread sets a result variable and then a done flag. On x86, the result would always be visible before the done flag. On ARM, however, it's possible for the done flag to be observed by another thread before the result write has propagated, leading to a race condition where the outdated result (e.g., 0 instead of 42) is read.

Fex employs a combination of strategies to reconcile these differences:

• Atomic Operations: A brute-force approach would be to make all memory operations atomic, but this incurs a significant performance penalty, potentially cutting performance in half for many games.
• Half Barriers: Inspired by the .NET framework, half barriers provide a more granular way to enforce memory ordering.
• ARM Instruction Extensions: Modern ARM instruction sets include extensions like LRCPC (Load-Acquire/Release-Store-Conditional) that offer better control over memory ordering.
• Kernel Patches: Fex also utilizes specific kernel patches to reduce overhead associated with memory synchronization.
• Smart Heuristics: Recognizing that no single solution is a silver bullet, Fex incorporates smart heuristics. For instance, a specific heuristic targets Unity games, which often use a circular buffer with a consistent 61-byte offset for relative stores. By detecting this pattern, Fex can selectively disable strong memory model semantics for these specific operations, providing a "massive performance boost" without affecting the correctness of other parts of the application.

Kernel Compatibility Layer

Beyond CPU instruction translation, Fex provides a kernel compatibility layer that intercepts and translates system calls and other kernel-level interactions. This layer makes the underlying ARM Linux kernel appear as an x86 Linux kernel to the emulated application.

• Thread Management: x86 applications expect specific thread behaviors, including how crashes (seg faults) are handled. Some game engines even use crashes for garbage collection, requiring Fex to intercept and relay these signals correctly to the emulated application's crash handlers.
• File System Redirection: When an x86 application attempts to load an x86 library (e.g., libfoo.so), Fex must redirect this request to an ARM-compatible version of that library, or handle it through its own emulation.
• System Call Wrapping: The Application Binary Interface (ABI) for system calls differs between x86 and ARM. Fex wraps hundreds of x86 system calls, decoding their parameters and invoking the corresponding ARM system calls with the correct ABI. This is a substantial undertaking due to the sheer number and complexity of syscalls.
• GPU Driver Access: Similar to system calls, interactions with the GPU driver often have ABI differences that Fex must mediate.

Library Optimizations and Code Caching

To further enhance performance and user experience, Fex implements two key optimizations:

• Library Forwarding (Vulkan Bridge): Emulating graphics APIs like Vulkan is computationally expensive. Fex uses a Vulkan bridge to hook into x86 Vulkan calls and directly forward them to the native ARM Vulkan driver. This bypasses the costly emulation of the original x86 Vulkan library and also sidesteps the kernel compatibility layer, as the calls are now native ARM.
• Code Caching: To mitigate initial loading times and in-game micro-stuttering, Fex implements code caching. During the first run of an application, Fex tracks all executed code blocks. In subsequent runs, Fex can proactively compile these blocks ahead of time and save the resulting ARM binary to disk. This means that for many scenarios, the recompiler is not needed for subsequent runs, leading to "much lower, faster loading times" and a "vastly reduced risk of micro-stutterings," especially when encountering new game assets.

Windows Games and Wine Integration

Fex extends its capabilities to Windows games through integration with Wine, a compatibility layer for running Windows applications on Linux. Two main approaches are discussed:

1. x86 Wine on Fex: In this setup, Fex emulates an x86 Linux environment, and an x86 build of Wine runs on top of it, providing the Windows API reimplementations. This is the approach used by Asahi Linux for Steam gaming.
2. Wine on ARM (ARM64EC): This more advanced approach aims for greater efficiency. It involves recompiling the Wine runtime itself to native ARM. Critically, Fex's recompiler output for the game code, and the recompiled Windows libraries (from Wine's source), target ARM64EC (Emulation Compatible). ARM64EC is a variant designed to facilitate interoperability between ARM native code and emulated x86 code, similar to Microsoft's approach for Windows on ARM. This significantly simplifies the emulation stack, moving many components to native ARM and reducing the need for the full kernel compatibility layer, resulting in "much simpler emulation stack" and "better performance."

The collective engineering effort behind Fex highlights the complexity of full-system architecture emulation and the innovative solutions required to achieve practical performance and compatibility.

Demo / Proof of Concept

▶ Watch: Optimizing with Intermediate Representation (IR) in Fex (7:20)

Tony Wasserka provided a compelling demonstration of Fex's capabilities directly during his presentation. The entire slide deck was displayed within an x86 build of Firefox downloaded from mozilla.org, running on an ARM-based ThinkPad X13s laptop via Fex. This subtle yet powerful proof-of-concept showed Firefox functioning as a "native application to all intents and purposes," including GPU acceleration and integration with the system's window manager.

The main highlight of the demo was running the Steam client and a 3D game on the ARM laptop. The Steam library launched successfully, and a 3D game was shown running, demonstrating "fluid frame rates" despite minor hiccups attributed to multi-monitor configuration issues rather than Fex itself. Wasserka also mentioned that AAA games like Cyberpunk, Doom Eternal, and Ultimate Epic Battle Simulator 2 have been successfully run using Fex by other users, underscoring its robust gaming compatibility. The hardware used for the demo, a ThinkPad X13s, features a Snapdragon chip with four performance and four efficiency ARM cores, showcasing Fex's performance on modern ARM consumer hardware.

Defensive Implications

▶ Watch: Addressing fundamental issues and overhead in emulation (8:50)

Fex, as an emulation layer, primarily focuses on enabling software compatibility rather than directly addressing security vulnerabilities or providing defensive mechanisms. However, its existence and growing adoption have implications for security professionals and system defenders operating in ARM Linux environments.

Firstly, the ability to run proprietary x86 applications on ARM means that the attack surface of a system is expanded to include these binaries. Defenders must recognize that even though the underlying CPU is ARM, the emulated application is still executing x86 code and will behave according to its original x86 semantics. If an x86 application contains vulnerabilities (e.g., buffer overflows, logic flaws), Fex will faithfully translate and execute the malicious behavior. Therefore, standard security practices like patching, sandboxing, and application whitelisting remain crucial for emulated x86 software.

Secondly, the complexity of Fex's architecture, particularly its kernel compatibility layer and system call wrapping, introduces new potential points of failure or exploitation if not meticulously implemented. Bugs in the translation of system calls or the handling of kernel interfaces could theoretically lead to privilege escalation or sandbox escapes from an emulated x86 process to the native ARM system. While the talk did not detail specific security hardening or auditing of Fex itself, the robust nature of its open-source development and integration into projects like Asahi Linux and Valve's Steam Deck suggests a strong focus on stability and correctness, which indirectly contributes to security.

Finally, security monitoring and incident response tools need to be aware of the Fex layer. Traditional tools that inspect process memory, system calls, or network activity might need to understand the translation layer to accurately attribute behavior to the original x86 application rather than Fex itself. For example, forensic analysis of a compromised ARM system running Fex might require tools capable of interpreting the execution flow through the emulator. Ultimately, Fex enables greater flexibility for users, but defenders must remain vigilant about the security posture of the emulated applications and the integrity of the emulation layer itself.

Key Takeaways

• Fex is a powerful open-source x86-to-ARM emulator for Linux, enabling proprietary x86 applications and games to run on ARM-based systems, including Apple Silicon Macs and Snapdragon laptops.
• The project employs a sophisticated binary recompiler with an Intermediate Representation (IR) for instruction translation and optimization, achieving performance comparable to proprietary solutions like Apple Rosetta and Microsoft Prism. Fex was notably the first to implement AVX/AVX2 instruction sets in software.
• Fex tackles the challenging x86 memory model through a multi-faceted approach, combining half barriers, kernel patches, ARM instruction extensions, and specific heuristics (e.g., for Unity games) to ensure correctness with minimal performance impact.
• Critical optimizations like Vulkan library forwarding (Vulkan bridge) and code caching significantly improve performance, reduce loading times, and eliminate micro-stuttering for a smoother user experience, particularly in gaming.
• Fex is a foundational technology for major initiatives, including Asahi Linux (for Steam gaming on Apple devices), Parallels Desktop, CodeWeavers' Wine development, and Valve's Steam Deck VR headset, demonstrating its practical utility and impact.
• The project is actively advancing Windows game emulation via Wine on ARM, leveraging ARM64EC to create a more efficient and simplified emulation stack, building on concepts pioneered by Microsoft for Windows on ARM.

About the Speaker(s)

Tony Wasserka is a software engineer with a passion for tackling tough technical challenges, particularly in the realm of emulation and low-level graphics. His journey began in university, where he explored video game console emulators, fascinated by the ability to run software on unintended platforms. Though he initially pursued theoretical physics in academia, Wasserka returned to software development, joining Imagination Technologies where he worked on the Vulkan driver for PowerVR GPUs. He later transitioned to the open-source Mesa project, contributing to the Vulkan driver for AMD GPUs, specifically focusing on the shader compiler. Tony joined the Fex development team approximately four years ago, bringing his deep expertise in GPU architectures and system-level programming to the project.