Spectre in the real world: Leaking your private data from the cloud with CPU vulnerabilities

Spectre in the real world: Leaking your private data from the cloud with CPU vulnerabilities

Speakers: Thijs Raymakers

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=_fPanvQcwe4

Overview

Eight years after the initial disclosures of Spectre and Meltdown, CPU vulnerabilities continue to challenge the fundamental security boundaries of modern computing. This talk, "Spectre in the real world: Leaking your private data from the cloud with CPU vulnerabilities," presented by Thijs Raymakers, delves into the practical exploitability of these hardware flaws within multi-tenant cloud environments. Raymakers demonstrates a novel attack chain that combines two known CPU vulnerabilities, L1TF and Half-Spectre, to stealthily exfiltrate sensitive data from co-located virtual machines (VMs) hosted by major cloud providers.

The core premise of the research addresses two critical questions: are CPU attacks still viable outside of controlled lab settings, given the extensive mitigations implemented since 2018? And from an attacker's perspective, what are the realistic and practical use cases for such complex CPU exploits? Raymakers compellingly argues that the cloud presents a prime target, where an attacker with root access to one VM can leverage these vulnerabilities to compromise the isolation enforced by the hypervisor and steal data from other tenants on the same physical hardware. The implications are significant, challenging the security assurances of shared cloud infrastructure and highlighting the persistent threat posed by hardware-level flaws.

This article provides a detailed technical breakdown of Raymakers' research, the vulnerabilities exploited, the attack methodology, and the defensive measures, both proposed and implemented, to counteract these sophisticated threats. It underscores the ongoing arms race between attackers and defenders in the realm of microarchitectural security.

Background

▶ Watch: Introduction to Spectre and cloud as attack target (0:00)

The landscape of CPU security was irrevocably altered on January 3rd, 2018, with the public disclosure of Spectre and Meltdown. These vulnerabilities revealed fundamental flaws in modern CPU architectures, specifically related to speculative execution and out-of-order execution, which are performance-enhancing features. They demonstrated that data could be leaked across previously assumed security boundaries, such as those between user space and kernel space, or between different processes. While not directly allowing arbitrary code execution, they enabled side-channel attacks to infer data that was never intended to be accessible.

In the years since, the industry, including academia and developers, has worked tirelessly to implement software and microcode mitigations for a steady stream of new CPU vulnerabilities, each often accompanied by a catchy name and logo. These efforts have largely addressed many of the initial concerns, leading to a perception that such attacks are primarily theoretical or confined to lab environments. However, Raymakers questioned the efficacy of these mitigations in messy, real-world production systems, particularly those employing defense-in-depth mechanisms.

The second crucial question addressed by the research was the practical utility of CPU attacks for an attacker. Typically, exploiting a CPU vulnerability requires some form of initial code execution. If an attacker already has code execution, there might be more straightforward routes to achieve their objectives. However, the cloud computing model presents a unique and compelling scenario. In the cloud, multiple customers (tenants) share physical hardware, with their virtual machines isolated by a hypervisor. The cloud provider's fundamental promise is robust isolation between these tenants. If an attacker can achieve code execution within their own virtual machine, exploiting CPU vulnerabilities to bypass hypervisor-enforced isolation and covertly read data from a co-located victim virtual machine becomes a highly valuable and stealthy attack vector, leaving no traces in the victim's logs. This is the precise threat model explored by Raymakers.

Key Findings

▶ Watch: Combining L1TF and Half-Spectre for powerful attack (5:50)

The central finding of this research is the demonstration of a practical, real-world cross-VM data exfiltration attack against multi-tenant cloud environments, leveraging a combination of two known CPU vulnerabilities: L1TF and Half-Spectre. This attack successfully bypassed existing software and microcode mitigations, including L1D flushing and core scheduling, by specifically targeting the hypervisor's memory.

Key findings include:

• Hypervisor Vulnerability: Even with robust mitigations like L1D flushing and core scheduling in place, the hypervisor's own memory remains vulnerable to L1TF attacks when Simultaneous Multi-threading (SMT) (Intel's Hyper-threading) is enabled on the underlying physical CPU. Cloud providers often keep SMT enabled due to the significant performance penalty associated with disabling it.
• Discovery of a Gadget: A specific Half-Spectre gadget was identified within the Linux KVM hypervisor code. This gadget allowed an attacker-controlled index to trigger a speculative out-of-bounds read, loading arbitrary host physical memory content into the L1 data cache.
• Arbitrary Host Physical Memory Read: By chaining the Half-Spectre gadget with L1TF, the attacker could achieve arbitrary host physical memory read capabilities. This enabled the attacker to systematically traverse host kernel data structures, including page tables, to locate and access the memory of any co-located victim VM.
• Successful Data Exfiltration: The attack was successfully demonstrated against Google Cloud, enabling the exfiltration of a private key from an Nginx instance running on a victim VM. This led to Google awarding the highest bug bounty in their history at the time: $121,500.
• Mitigation Differences: While successful against Google Cloud's default configurations, the attack was significantly hampered on AWS Nitro, which employs additional hardware-level mitigations like exclusive page frame ownership, preventing cross-VM data leakage.
• Stealth and Undetectability: The attack is entirely stealthy, leaving no traces in victim logs or hypervisor monitoring, making detection extremely challenging and raising concerns about potential past abuse.

These findings highlight the persistent and complex nature of CPU vulnerabilities, forcing a re-evaluation of security assumptions in shared cloud infrastructure and driving the development of new, more fundamental hardware and software mitigations.

Technical Deep Dive

▶ Watch: Detailed explanation of L1TF (L1 Terminal Fault) (6:20)

The attack presented by Thijs Raymakers hinges on the clever combination and exploitation of two distinct CPU vulnerabilities: L1 Terminal Fault (L1TF) and Half-Spectre, specifically targeting the Linux KVM hypervisor. The ultimate goal is to achieve arbitrary host physical memory read capabilities from a guest virtual machine, bypassing existing software mitigations.

L1 Terminal Fault (L1TF)

L1TF, disclosed in August 2018, is an Intel-specific vulnerability (CVE-2018-3620, CVE-2018-3646) that allows data to be leaked from the L1 data cache across security boundaries. When a process accesses memory, the CPU stores frequently used data in the L1 data cache for faster retrieval. L1TF enables an attacker to read data left in the L1 cache by other processes or virtual CPUs (vCPUs) running on the same physical core, even after a context switch. This is problematic in multi-tenant environments where vCPUs from different guests, or a guest and the hypervisor, might share a physical core.

The initial industry response to L1TF involved two main software mitigations:

1. L1D Flushing: Before a context switch from a privileged entity (like the hypervisor or a sensitive guest) to a less privileged one (an attacker's guest), the L1 data cache is flushed. This aims to remove any sensitive data, preventing an L1TF attack from reading stale cache content.
2. Core Scheduling: To address the complexities introduced by Simultaneous Multi-threading (SMT) (Intel's Hyper-threading), core scheduling ensures that vCPUs from different security domains (e.g., attacker and victim, or guest and hypervisor) are never co-scheduled on the same physical core at the same time. SMT allows a single physical core to present itself as multiple logical cores, sharing resources like the L1 cache. Without core scheduling, an attacker vCPU running on a logical core could exploit the tiny window between a victim vCPU's operations and the L1D flush to still leak data.

While these mitigations largely address cross-guest L1TF leakage, Raymakers identified a crucial gap: the hypervisor's own memory remains vulnerable. Cloud providers, for performance reasons, often keep SMT enabled. In such a scenario, even with L1D flushing and core scheduling, an attacker's vCPU and a hypervisor thread might be co-scheduled on different logical cores of the same physical core. When the attacker interacts with the hypervisor (e.g., via a hypercall), the hypervisor loads data into the L1 cache. Before the L1D flush can complete, the attacker's co-scheduled vCPU has a small window to perform an L1TF attack and leak the hypervisor's L1 cache content. This is a known issue, as acknowledged by Intel advisories and Linux administrator guides, which explicitly recommend disabling SMT if untrusted VMs run on vulnerable CPUs, due to the host memory being attackable. However, the prohibitive performance cost (upwards of 30% in some workloads, translating to billions in infrastructure costs for cloud providers) often prevents this.

Half-Spectre and the KVM Gadget

This is where Half-Spectre comes into play. Half-Spectre exploits branch misprediction in the CPU's speculative execution engine. Modern CPUs speculatively execute instructions beyond conditional branches (like if statements) to improve performance. If the prediction is wrong, the CPU rolls back any architectural state changes that would affect correctness. However, it does not roll back changes to the cache state. This means that during a mispredicted speculative execution, data can be loaded into the L1 cache even if the instruction would not have been executed architecturally.

Raymakers specifically looked for a "gadget" within the KVM hypervisor: a code pattern involving a bounds check followed by a load operation, where the index for the load is controlled by the attacker. Such patterns are common for preventing out-of-bounds memory access. The identified gadget in KVM involved a variable named desk_ID, which is controlled by the guest VM. This desk_ID is used as an index into an array called map_fmap_array after a bounds check.

The attack works as follows:

1. The attacker supplies a desk_ID that is in bounds multiple times, training the branch predictor to expect the bounds check to succeed.
2. The attacker then supplies a desk_ID that is significantly out of bounds.
3. Due to the trained branch predictor, the CPU speculatively proceeds with the load operation using the out-of-bounds desk_ID before the bounds check fully evaluates and determines the prediction was wrong.
4. This speculative out-of-bounds load brings content from an arbitrary physical memory address (determined by the attacker-controlled desk_ID) into the L1 data cache.

The map_fmap_array itself is located in the direct map of the system – a one-to-one kernel virtual memory mapping of physical RAM. By triggering this Half-Spectre gadget, arbitrary physical memory can be speculatively loaded into the hypervisor's L1 data cache.

L1TF Reloaded: Chaining the Attack

Combining these two vulnerabilities forms the L1TF Reloaded attack:

1. On one vCPU, the attacker continuously interacts with the hypervisor, triggering the Half-Spectre gadget with a carefully chosen desk_ID. This causes arbitrary host physical memory to be speculatively loaded into the hypervisor's L1 data cache.
2. Simultaneously, on another co-scheduled vCPU (on the same physical core, leveraging SMT), the attacker performs an L1TF attack to leak the contents of the hypervisor's L1 data cache.

This chain allows an attacker to read arbitrary host physical memory.

Locating Victim Data: The 2D Page Table Translation

Once arbitrary host physical memory can be read, the next challenge is to find sensitive data belonging to a victim VM. The map_fmap_array entries themselves are not direct victim data, but rather kernel pointers, specifically to KVM_lapic_state structs, which in turn point to KVM_vcpu structs, and so on. By dereferencing these pointers, the attacker can eventually reach the host's task_struct linked list. This linked list contains information about all processes running on the host, including the hypervisor's own components (e.g., QEMU, DomVCPU nanny).

By iterating through this linked list and examining process names, the attacker can identify processes related to the hypervisor and, crucially, gain access to the root of their page tables (CR3 register values). From these page tables, the attacker can then perform a complex 2D Page Table Translation:

• Translate an attacker's guest virtual address to a host physical address.
• Translate the host physical address to a victim guest physical address.
• Translate the victim guest physical address to a victim process virtual address.

This intricate series of page walks, performed in software, allows the attacker to map the memory layout of the victim VM and locate specific regions, such as the heap where sensitive data like private keys might reside. The process is error-prone, requiring a "chase and check" methodology with sanity checks to ensure pointers are valid and data matches expected patterns (e.g., kernel pointers starting with FFF). While slow, this systematic approach enables precise targeting and exfiltration of sensitive information from co-located tenants.

Demo / Proof of Concept

▶ Watch: Visualizing L1TF: leaking data from L1 data cache (6:40)

The practical demonstration of the L1TF Reloaded attack was a highlight of the talk, showcasing its real-world viability against a production cloud environment. The setup involved an attacker VM, over which Raymakers had root access, and a victim VM running an Nginx web server, which stored a private key on its heap. The critical aspect of the attack is its entirely covert nature: the attacker does not need to interact with the victim VM, meaning no suspicious activity is logged on the victim's side.

The demonstration, though significantly sped up for the presentation (the actual attack took approximately six hours initially), illustrated the following key phases:

1. Finding map_fmap_array Base Address: The first step involved brute-forcing host physical memory to locate the base address of the map_fmap_array. In the demo, the host had 825 GB of RAM, making this a time-consuming but necessary initial step. Once found, this value can be reused, accelerating subsequent attacks.
2. Breaking Host Kernel ASLR: With the base address, the attacker proceeds to break Kernel Address Space Layout Randomization (ASLR) on the host. This establishes a reliable understanding of the host kernel's memory layout.
3. Host Process Enumeration: By traversing the task_struct linked list in the host kernel memory, the attacker can effectively get a "top-like" view of all processes running on the hypervisor. This reveals hypervisor components like QEMU or DomVCPU nanny (Google's internal VM manager name).
4. 2D Page Table Translation Setup: Once hypervisor processes are identified, the attacker uses their page table information to set up the complex 2D page table translation mechanism. This involves reading and dereferencing numerous pointers to establish the mapping between the attacker's, host's, and victim's memory spaces.
5. Breaking Victim Kernel ASLR and Process Enumeration: The same technique is applied to the victim VM's kernel, breaking its ASLR and enumerating its processes. The demo showed identifying systemd, P1, SSHD, and eventually, the target Nginx process.
6. Locating and Exfiltrating the Private Key: With the Nginx process identified and its memory mapped, the attacker then targeted its heap to locate the private key. The process of reading the key involved extracting raw bytes. To verify the correctness of the leaked data, a primality test was used, as private keys often contain large prime numbers. This allowed the attacker to filter out incorrect bytes resulting from the inherent error-proneness of cache-based side channels. The demo successfully showed the identification of both prime factors, confirming the exfiltration of the Nginx private key.

Raymakers reported this vulnerability to Google Cloud, who promptly acknowledged the report and requested a proof of concept. Following meetings and validation, Google awarded a bug bounty of $121,500 on May 29th, 2023, which was the highest bug bounty they had ever given for this type of research. The public disclosure occurred on August 11th at USENIX Security (USENIX S&P).

Similar research was conducted on AWS, but with different results. The attack was unable to leak data from other guests due to AWS Nitro, AWS's custom KVM-based hypervisor, which incorporates additional mitigations like exclusive page frame ownership. Contact with other cloud providers yielded mixed responses, with one anonymous provider declining to comment on their affected status, interpreted by Raymakers as a likely indication of vulnerability.

The talk emphasized the stealthy nature of the attack, leaving no logs or traces, making it extremely difficult to detect if it had been abused in the past.

Defensive Implications

▶ Watch: Hardware vulnerability fixes: replacing all CPUs is infeasible (8:05)

The L1TF Reloaded attack underscores the persistent challenges posed by CPU vulnerabilities in cloud environments and necessitates a multi-faceted defensive strategy at both the cloud provider and tenant levels.

Cloud Provider Mitigations

Cloud providers are at the forefront of implementing large-scale mitigations:

1. Gadget Removal: The specific Half-Spectre gadget exploited in the Linux KVM hypervisor has been removed from all Linux kernel releases since September 2023. This emphasizes the importance of keeping host kernels up-to-date. However, this only addresses one specific gadget, and similar patterns might exist or emerge.
2. New Architectural Mitigations: Two primary architectural mitigations are being upstreamed into the Linux kernel and are already in production at major cloud providers:

• Exclusive Page Frame Ownership (EPFO): Implemented in AWS Nitro, EPFO aims to prevent cross-VM information leakage by ensuring that guest physical memory is never directly mapped into the hypervisor's address space. Instead, a complex XOR-based mapping is used, effectively removing guest data from the direct map accessible by the hypervisor, thus preventing the Half-Spectre gadget from loading sensitive guest data into the L1 cache. This makes it significantly harder for an attacker to relate hypervisor cache contents to victim data.
• Address Space Isolation: Being rolled out by Google Cloud, this mitigation involves switching the hypervisor's address space when transitioning between different guest contexts. When the hypervisor handles a request from Guest A, it operates within an address space that only includes Guest A's memory. If it then needs to interact with Guest B, it first switches to an address space that includes Guest B's memory. This ensures that the hypervisor's L1 cache only ever contains data relevant to the currently active guest or its own isolated components, preventing leakage of other guests' data.

These mitigations are complex to implement and have performance implications, which is why their upstreaming into the kernel is a careful and ongoing process.

Recommendations for Cloud Tenants

For individual users and organizations renting VMs in the cloud, understanding and mitigating this class of attacks involves asking critical questions and making informed choices:

1. Host Kernel Updates: Inquire with your cloud provider about their policies and frequency for updating host kernels. The specific gadget exploited has been removed, but this requires timely updates across the fleet, which can be challenging due to the need for live migrations.
2. Microcode Updates: Ask about the application of CPU microcode updates, which are firmware-level patches for hardware vulnerabilities.
3. SMT Status: Understand whether SMT (Hyper-threading) is enabled on the physical CPUs hosting your VMs. If your threat model includes co-tenant attacks, disabling SMT is the most robust software mitigation against L1TF-like vulnerabilities that leverage SMT's shared cache. However, this comes with a significant performance penalty (potentially 30% or more), which may not be acceptable for all workloads.
4. Core Scheduling: Ask if your cloud provider utilizes core scheduling to prevent co-scheduling of untrusted guests on the same physical core. While effective for guest-to-guest isolation, it does not fully protect against guest-to-hypervisor leakage if SMT is enabled.
5. Dedicated Machines / Self-Hosting: For applications with extremely high security requirements and a threat model that encompasses sophisticated co-tenant attacks, consider opting for dedicated physical machines or self-hosting. This eliminates the risk of sharing physical hardware with untrusted tenants, putting the entire machine under your control.

Ultimately, the L1TF Reloaded attack highlights that securing cloud environments against microarchitectural vulnerabilities is a shared responsibility, requiring continuous vigilance and collaboration between hardware vendors, cloud providers, and tenants.

Key Takeaways

• CPU Vulnerabilities are a Real Cloud Threat: Despite years of mitigations, CPU vulnerabilities like L1TF and Half-Spectre remain exploitable in real-world multi-tenant cloud environments, enabling cross-VM data leakage.
• Hypervisor is the Target: The attack bypasses existing guest-to-guest mitigations by specifically targeting the hypervisor's memory, leveraging its interaction with the attacker's VM.
• SMT is a Critical Factor: Simultaneous Multi-threading (SMT) or Hyper-threading, while a performance enhancer, creates critical side-channel leakage opportunities for CPU vulnerabilities when enabled on cloud host machines.
• Performance vs. Security Trade-off: Cloud providers face immense pressure to balance performance and security, often leading to SMT being enabled despite known risks, as disabling it incurs prohibitive costs and performance penalties.
• Novel Mitigation Strategies Emerging: New, more fundamental architectural mitigations like Exclusive Page Frame Ownership (AWS Nitro) and Address Space Isolation (Google Cloud) are crucial for addressing these deep-seated hardware issues.
• Attacks are Stealthy and Undetectable: The demonstrated attack leaves no traces in victim logs or hypervisor monitoring, making detection extremely difficult and raising concerns about potential past exploitation.

About the Speaker(s)

Thijs Raymakers is a researcher who conducted this work as part of his master's thesis at the Vrije Universiteit Amsterdam (VU Amsterdam). His project, which "spiraled out of control," involved significant collaboration and expertise from others. He specifically thanked Mate for immense help in iterating on the initial proof of concept and optimizing it for speed and reliability on cloud platforms. He also acknowledged Dave for supervision, and Marius Herbert and Eric for their guidance, feedback, and expertise throughout the project. His research has been published on GitHub and as an S&P paper.