How to render cloud FPGAs useless

How to render cloud FPGAs useless

Speakers: Dirk

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=pOaxFUXns7o

Overview

In this compelling talk from 39C3, Dirk, a researcher from the novel computing technologies group, delves into the vulnerabilities of Field-Programmable Gate Arrays (FPGAs) when deployed in cloud environments. The presentation, titled "How to render cloud FPGAs useless," explores sophisticated remote attack vectors that can lead to denial-of-service (DoS) and accelerated aging of these critical hardware resources. Dirk's work highlights a significant shift in hardware security research, moving from traditional physical side-channel attacks requiring direct board access to purely remote methodologies capable of impacting multi-tenant cloud infrastructure.

The talk meticulously details various techniques for generating extreme power consumption and localized hotspots within FPGAs, demonstrating how these can circumvent cloud provider defenses and degrade hardware performance. By leveraging low-level FPGA features, attackers can induce conditions that range from temporary service disruption to permanent, albeit subtle, performance reductions over time. This research is particularly relevant as FPGAs become more prevalent in cloud computing for accelerating diverse workloads, raising critical questions about their resilience against malicious tenants and the efficacy of current cloud security paradigms.

Dirk's findings underscore the challenges faced by cloud providers in securing shared hardware resources and offer practical insights for both defenders and researchers. The presentation not only showcases the ingenuity of the attack methods but also provides a rare glimpse into the speaker's direct engagement with a major cloud provider, Amazon Web Services (AWS), regarding these discovered vulnerabilities. The implications extend beyond FPGAs, touching upon the broader security posture of hardware-as-a-service models and the need for robust, hardware-aware monitoring and mitigation strategies.

Background

▶ Watch: Introduction to cloud FPGAs and hardware security attacks (0:00)

The increasing availability of Field-Programmable Gate Arrays (FPGAs) in cloud infrastructures like Amazon F1, Microsoft Azure, and Alibaba Cloud has opened new frontiers for both high-performance computing and hardware security research. Traditionally, hardware security investigations often involved physical access to devices, utilizing tools like oscilloscopes or electromagnetic radiation sensors to conduct side-channel analysis and extract sensitive information, such as cryptographic keys. However, the advent of cloud FPGAs shifts this paradigm dramatically, enabling purely remote attacks where malicious tenants can interact with and potentially compromise the underlying hardware without ever touching it physically.

FPGAs are essentially grids of reconfigurable logic, composed of configurable logic blocks (CLBs) and programmable interconnects. Each CLB typically contains lookup tables (LUTs), which are small memories configured to implement arbitrary boolean functions, along with flip-flops and multiplexers. The flexibility of FPGAs allows users to deploy custom digital circuits, which, in a multi-tenant cloud environment, raises concerns about potential misuse.

A fundamental building block often exploited in these remote attacks is the ring oscillator. A ring oscillator is created by chaining an odd number of inverters (or LUTs configured as inverters) into a loop. The signal continuously toggles around the loop, generating a self-sustaining oscillation. The frequency of this oscillation is highly sensitive to environmental factors such as operational temperature and supply voltage. This characteristic makes ring oscillators excellent remote sensors, effectively acting as tiny "oscilloscopes" on the chip. Researchers can measure their speed with high accuracy, often down to less than 10 picoseconds using time-to-digital converters (TDCs), to infer changes in the FPGA's operating conditions or even extract side-channel information from co-located circuits. The challenge for attackers, as Dirk demonstrates, is to deploy these sensitive circuits while bypassing cloud providers' design rule checks (DRCs), which often scan for known malicious patterns like direct ring oscillator implementations. This can be circumvented by simple obfuscation, such as including a transparent latch that can be enabled to effectively act as a wire, thus passing the initial design checks.

Key Findings

▶ Watch: How to deploy ring oscillators on cloud FPGAs (3:40)

Dirk's research reveals that cloud FPGAs, despite layers of virtualization and monitoring, are susceptible to remote attacks capable of inducing severe power consumption and localized thermal stress, ultimately leading to significant performance degradation or denial of service. The core findings are multi-faceted:

First, it is demonstrably possible to construct FPGA designs that draw kilowatts of power from a single chip. By strategically deploying a high density of specialized ring oscillators or high-frequency shift registers, attackers can push the FPGA far beyond its typical operational power envelope. For instance, just 3% of a smaller FPGA's lookup tables could consume 11 watts, saturating the board's maximum power delivery. On datacenter-grade FPGAs, designs using only 10% of LUT resources were shown to draw 350 watts, with theoretical projections suggesting potentials exceeding 15 kilowatts if all resources were fully utilized. This level of power consumption equates to an energy density in the same order of magnitude as the sun's surface, highlighting the extreme local stress on the silicon.

Second, cloud providers' existing defenses, such as design rule checks (DRCs) and online power monitoring, can be circumvented or overwhelmed. Simple obfuscation techniques, like inserting a transparent latch into a ring oscillator, allow malicious designs to pass initial DRCs. While Amazon's power monitoring system did detect anomalous power draws, issuing warnings at 85 watts and clock-gating at 105 watts, the talk demonstrated that custom-generated clocks within the FPGA could bypass this clock-gating mechanism. Ultimately, while Amazon would cut off access to the FPGA at higher power levels (e.g., 100+ watts), the attacks still managed to crash instances, requiring manual intervention and significantly increasing the downtime of affected hardware.

Third, the research confirmed that prolonged, high-power operation, particularly when concentrated into hotspots, can induce hardware aging mechanisms like electromigration and hot carrier injection. While outright "frying" a chip proved more difficult than initially anticipated (partially due to the large amount of "dark silicon" and silicon's excellent heat spreading properties), the observed aging effects were substantial enough to "render them useless" for reliable operation. Experiments on datacenter FPGAs, running a 360-watt hotspot in just 1.2% of the chip area over three weeks, resulted in an average performance degradation of 1.39%. Critically, this degradation was not uniform; some individual wires experienced up to a 70% increase in delay, leading to an estimated 15-20% reduction in the maximum operating frequency (Fmax) of the chip. This means a 500 MHz design could only safely operate at 400-425 MHz, making the FPGA unsuitable for its intended high-performance acceleration tasks.

Finally, the talk highlighted the practical challenge for cloud providers in detecting and mitigating these sophisticated attacks. Dirk's direct communication with Amazon, including the humorous exchange of "you may run whatever test you'd like" followed by an account suspension after crashing 100 instances, underscores the novelty and impact of these findings. The ability to "fingerprint" specific FPGA devices using physical unclonable functions (PUFs) derived from ring oscillators further complicates the issue by allowing attackers to track specific vulnerable hardware.

Technical Deep Dive

▶ Watch: Using ring oscillators as remote oscilloscopes for attacks (4:20)

The technical foundation of these attacks lies in manipulating fundamental FPGA structures to achieve extreme operational conditions.

Ring Oscillators as Sensors and Power Wasters:

As discussed, ring oscillators (ROs) are central to these attacks. They are formed by an odd chain of inverters, typically implemented using Lookup Tables (LUTs) configured as inverters, with the output feeding back to the input. The frequency of these ROs is a sensitive indicator of local temperature and supply voltage. To measure this frequency with high precision, Time-to-Digital Converters (TDCs) are employed, capable of resolving delays down to less than 10 picoseconds. This allows remote monitoring of chip conditions. While direct ROs are often detected by Design Rule Checks (DRCs), Dirk demonstrated bypassing this by integrating a transparent latch that, when enabled, acts as a wire, allowing the RO to function while appearing benign during design validation.

Denial of Service (DoS) through Extreme Power Draw:

Dirk detailed several mechanisms to induce high power consumption:

1. Short Circuits: By manipulating the configuration bits of internal multiplexers, specifically pass transistors or transmission gates, it's possible to activate multiple inputs simultaneously while driving them to conflicting logic levels (one high, one low). This creates a direct short circuit. Dirk accidentally discovered this, damaging a Spartan 3 FPGA, and measured up to 7 milliamps of extra current per lookup table. Scaled across hundreds of thousands of LUTs, this can sum to significant power.

1. High-Speed Shift Registers: Modern FPGAs contain millions of flip-flops. By configuring a large number of these (e.g., 20-30 million flip-flops) into a ring shift register and clocking them at high frequencies (e.g., half a gigahertz), massive amounts of power can be consumed. This switching activity can generate 2 kilowatts or more. Such rapid load changes can also induce voltage drops, potentially leading to fault injection attacks in co-located sensitive circuits (e.g., causing a CPU compare instruction to set a flag incorrectly).

1. Glitch Amplification: This technique involves taking a signal from a toggle flip-flop and routing it to an XOR gate via two paths with slightly different latencies. Each time the signal arrives at the XOR gate with a different timing, it causes the output to toggle, effectively generating a faster switching activity than the input signal. This allows for localized high-frequency switching even when the primary clock speed is lower.

Achieving Extremely Fast Oscillators and Hotspots:

To maximize power draw and create targeted hotspots, achieving the highest possible oscillation frequencies is crucial:

• Optimized LUT Paths: Dirk identified specific internal routing paths within LUTs that offer extremely low propagation delays. For instance, one path was measured at just 41 picoseconds from input to output, enabling ring oscillators to reach frequencies of around 6 GHz. This path specifically leverages the multiplexer tree structure within the LUT.
• Leveraging On-Chip Resources: Besides LUTs, other resources like user logic multiplexers, carry chain logic, DSP blocks, and even transparent latches or asynchronous resets can be used to construct fast oscillators.
• Overclocking Clock Generators and Phase Shifters: FPGAs include sophisticated clock management units with phase shifters. By running a base clock at a fast frequency (e.g., 2 GHz) and generating multiple phase-shifted versions of it, these signals can be routed to an XOR gate to create glitch amplification. With four phases, this can multiply the effective switching frequency by four, reaching over 8 GHz from a 2 GHz base clock.

Quantifying Power Consumption and Hotspot Potential:

The research quantified the immense power potential:

• A simple experiment filling a chip with ring oscillators showed that 3% of LUTs could draw 11 watts, exceeding the board's maximum power delivery.
• Individual CLBs could draw 60 milliwatts, or 2 milliwatts per LUT.
• Rough estimates for an entire FPGA, fully utilized with these techniques, suggest 2 kilowatts for oscillators, 5 kilowatts for driving wires, and 2 kilowatts for flip-flops, with additional power from block RAMs, DSPs, and gigabit transceivers. The total potential could easily exceed 15 kilowatts.
• On a datacenter card (Alveo U200 equivalent to Amazon F1), a design using 10% of LUTs was shown to draw 350 watts while passing all design rule checks. This highlights the ability to generate "kilowatts of power hammering potential."

Aging Mechanisms and Performance Degradation:

Sustained high power and temperature induce various aging mechanisms:

• Hot Carrier Injection (HCI): Occurs when transistors operate at high temperatures, causing charge carriers to gain enough energy to tunnel into the gate oxide, altering transistor characteristics.
• Electromigration (EM): The movement of metal atoms in a conductor due to momentum transfer from electrons, leading to voids and shorts, especially at high current densities and elevated temperatures over time.

Experiments involving 2-3 weeks of continuous operation with hotspots showed:

• On smaller FPGAs, a measurable 0.3 picosecond jump in delay was observed within two days.
• On datacenter FPGAs, a 135-watt hotspot concentrated in just 1.2% of the chip area over three weeks led to an average performance degradation of 1.39%. Crucially, the distribution was wide, with a few wires experiencing up to 70% delay aging.
• The practical implication is an estimated 15-20% reduction in Fmax, meaning a 500 MHz design might only operate reliably at 400-425 MHz. While not immediately "fried," the chip becomes "useless" for its intended high-performance applications.

Demo / Proof of Concept

▶ Watch: High power shift registers for voltage drop attacks (6:40)

The practical validation of these attacks involved deploying malicious designs on Amazon F1 instances, which utilize Xilinx Alveo U200 FPGAs.

Bypassing Design Rule Checks (DRCs):

Dirk's team successfully created designs capable of burning kilowatts of power that passed Amazon's DRCs. The key technique involved obfuscating known power-wasting structures. For example, a simple ring oscillator that would normally be flagged was made undetectable by including a transparent latch. When this latch is disabled, it behaves as a standard storage element; when enabled, it acts as a simple wire, completing the ring oscillator and allowing it to spin at high frequencies. This clever trick ensures the design appears benign during static analysis but becomes malicious during runtime.

FPGA Fingerprinting with PUFs:

To track which specific FPGA instance they were interacting with, the researchers developed a method to "fingerprint" the chips. They deployed multiple ring oscillators designed as Physical Unclonable Functions (PUFs). Due to inherent manufacturing process variations, each FPGA chip exhibits unique, subtle differences in transistor characteristics, causing these PUFs to oscillate at slightly different frequencies. By measuring the relative speeds of these PUFs, even at different temperature levels (by heating the chip with other power-wasting circuits), they could generate a unique signature for each physical FPGA device. This allowed them to know if they were interacting with a previously attacked or degraded chip, or a fresh one.

Interacting with Amazon's Power Monitoring:

The team deployed gradual power-ramping designs on Amazon F1 to understand the provider's defensive mechanisms. They observed the following thresholds:

• At 85 watts, Amazon's system issued a warning about high power consumption.
• At 105 watts, the system would gate the clock to the FPGA. However, because the malicious designs generated their own internal clocks (e.g., using ring oscillators or phase shifters), this external clock gating was effectively bypassed, and the designs continued to operate at high power.
• At power levels exceeding 100+ watts (the exact threshold varied), Amazon would completely cut off access to the FPGA device, sometimes even losing contact with the entire instance. This typically required the user to request a new instance.

Crashing Instances and Account Suspension:

The ultimate proof of concept involved intentionally crashing Amazon F1 instances. The team deployed designs capable of creating a 360-watt hotspot within just 1.2% of the FPGA's area. They continuously requested, utilized, and released instances, fingerprinting each one.

• If an instance did not crash, it would typically become available again within minutes.
• If an instance did crash due to the power attack, it would take significantly longer – often an hour or even several hours – before a new instance became available. This suggested that physical intervention (e.g., "somebody with a trolley has to pop by and restart the whole thing") might be necessary.
• After conducting these experiments on approximately 100 instances, Amazon suspended their account. This clearly indicated the severity of the attacks and their impact on cloud infrastructure.

Collaboration and Hotspot Design:

Following the account suspension, Amazon took the findings seriously, leading to extended discussions with Dirk's team. Xilinx (now AMD), the manufacturer of the FPGAs, also donated boards for further destructive testing, acknowledging the importance of understanding these edge cases. Dirk emphasized that the real challenge isn't just drawing high total power, but creating concentrated hotspots. The team developed designs capable of generating 100 milliwatts per CLB (Configurable Logic Block), which, if scaled, could lead to over 15 kilowatts of power, but critically, focused in a very small area to maximize thermal stress and accelerate aging.

Defensive Implications

▶ Watch: Glitch amplification to generate faster switching activity (7:50)

Dirk's research highlights several critical areas where cloud FPGA security can be enhanced. While cloud providers like Amazon AWS already implement various protective measures, these have proven insufficient against the sophisticated remote attacks demonstrated.

Current Cloud Provider Defenses (as observed):

1. Design Rule Checks (DRCs): Cloud providers scan user-submitted designs for known problematic patterns. However, as shown, simple obfuscation (e.g., transparent latches) can bypass these checks.
2. Encrypted Design Checkpoints: Users submit encrypted design checkpoints, meaning the raw bitstream is not directly accessible. This prevents users from directly inserting malicious bitstream patterns. However, Dirk noted that these checkpoints still contain original design hierarchy and names, which can be "quite revealing."
3. Bitstream Generation by Provider: The actual bitstream for configuring the FPGA is always generated by the cloud provider, and users never directly interact with it.
4. Hypervisor API for Configuration: FPGA configuration is managed through a hypervisor API, abstracting the hardware layer from the user.
5. Online Power Monitoring: Cloud providers actively monitor the power consumption of FPGAs, issuing warnings, clock-gating, and ultimately cutting off devices that exceed thresholds. As demonstrated, this can be bypassed by internal clocks or overwhelmed by rapid power surges.

Recommendations for Defenders:

1. Enhanced Bitstream Scanning and Analysis:

• Advanced "Virus Scanner" for Bitstreams: Dirk proposed a more sophisticated scanning mechanism that can reverse-engineer a bitstream back to a netlist level. This allows for detailed analysis to detect hidden ring oscillators, high-fanout nets, or other "nasty circuits" that could lead to excessive power consumption or aging.
• Glitch Score and Heat Maps: The scanner should be able to compute a "glitch score" for a given circuit, estimating its switching activity. By correlating this with area and routing information, it can generate "heat maps" that predict potential hotspots and power draw. This would be far more effective than just looking for known patterns. For example, Bitcoin miner bitstreams, which are inherently power-hungry, could be analyzed and scored.

1. Trusted Entity for Scanning: To address user concerns about intellectual property leakage, the scanning process could be performed by a trusted third party rather than directly by the cloud provider. This would allow for thorough analysis without the user having to "yield their design to Amazon."
2. Restrict Access to Critical Infrastructure:

• Clock Trees: Users should not be given direct, unrestricted access to manipulate global clock trees. Malicious manipulation of clock managers and phase shifters was key to achieving extremely high frequencies and glitch amplification. Tighter controls or sandboxing around these critical resources could mitigate some of the most potent attacks.
• Low-Level Configuration: While users need flexibility, certain low-level configuration options that allow for direct short-circuiting or highly localized power concentration might need to be restricted or more heavily scrutinized.

1. Improved Power Management and Thermal Monitoring: While existing power monitoring is useful, it needs to be more robust against rapid changes and internal clock generation. More granular and real-time thermal monitoring, perhaps with predictive capabilities based on design characteristics, could help detect and mitigate hotspot formation before significant damage occurs.
2. Hardware-Aware Scheduling and Isolation: Cloud providers could implement more intelligent scheduling to avoid placing multiple high-risk or potentially malicious tenants on physically adjacent or vulnerable FPGA regions. Better hardware-level isolation mechanisms, beyond logical virtualization, might also be necessary.

Dirk concluded that while damaging FPGAs is harder than initially thought due to factors like "dark silicon" (large idle areas) and silicon's excellent heat spreading properties, the demonstrated aging effects are severe enough to render FPGAs "useless" for their intended high-performance applications. The good news is that these situations can be spotted with better scanning and architectural safeguards.

Key Takeaways

• Cloud FPGAs are Vulnerable to Remote DoS and Aging: Malicious tenants can deploy designs that induce extreme power consumption and localized hotspots, leading to denial-of-service or premature hardware aging.
• Provider Defenses Can Be Circumvented: Cloud providers' design rule checks and power monitoring systems can be bypassed (e.g., with transparent latches) or overwhelmed (e.g., with custom internal clocks), allowing high-power designs to operate.
• Extreme Power Consumption is Achievable: Designs using a small percentage of FPGA resources can draw hundreds of watts, with theoretical potential for kilowatts across the entire chip, creating energy densities comparable to the sun's surface.
• Aging Causes Significant Performance Degradation: Prolonged high-power operation, particularly in hotspots, accelerates hardware aging, leading to an average 1.39% performance degradation and up to 70% delay increase on individual wires, reducing the chip's maximum operating frequency (Fmax) by 15-20%.
• Outright Destruction is Difficult, but "Uselessness" is Not: While physically "frying" an FPGA is challenging due to inherent design characteristics like "dark silicon" and silicon's heat spreading, rendering it unreliable or unsuitable for high-performance tasks through aging is a practical outcome.
• Advanced Detection and Mitigation are Needed: Current defenses are insufficient. New strategies, including sophisticated bitstream analysis (like a "virus scanner" for netlists), glitch scoring, heat mapping, and stricter controls over critical FPGA resources like clock trees, are essential for securing cloud FPGA infrastructure.

About the Speaker(s)

Dirk is a researcher associated with the "novel computing technologies" group. His work focuses extensively on FPGA-related issues, encompassing a broad range of topics including CAD tools, reliability aspects, and various applications of FPGAs. Notably, his group is also involved in making FPGA chips, a topic he presented on in a previous year. A significant area of his research, and the focus of this talk, is hardware security. He also works extensively on partial reconfiguration techniques for FPGAs. As a testament to his group's capabilities, he proudly mentioned that his postdoc successfully ran a soft RISC-V CPU on an FPGA at 101 megahertz.