Rowhammer in the Wild: Large-Scale Insights from FlippyR.AM

Rowhammer in the Wild: Large-Scale Insights from FlippyR.AM

Speakers: Martin Heckel, Florian Adamsky, Daniel Gruss

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=OZcfRCTUcEI

Overview

This talk, presented by Martin Heckel, Florian Adamsky, and Daniel Gruss at 39C3, delves into the pervasive and evolving threat of Rowhammer attacks, leveraging insights from their large-scale study, FlippyR.AM. Daniel Gruss, a prominent figure in Rowhammer research, marked his 10-year anniversary of first presenting on the topic at the same conference, underscoring the enduring relevance and continuous evolution of this hardware vulnerability. The presentation serves as both a comprehensive recap of Rowhammer fundamentals and an update on the latest research, culminating in a data-driven analysis of real-world systems.

The core of the talk highlights that Rowhammer, a disturbance error in Dynamic Random-Access Memory (DRAM), remains a critical security concern, capable of inducing bit flips in memory cells that can lead to privilege escalation or data corruption. The FlippyR.AM study collected data from over a thousand systems, revealing that a significant percentage of modern computers are still vulnerable to fully automated Rowhammer attacks. This article will explore the technical underpinnings of Rowhammer, the challenges in its detection and mitigation, the novel attack vectors discovered, and the crucial implications for both system defenders and hardware manufacturers.

The speakers emphasize that despite industry efforts to implement hardware-based countermeasures like Target Row Refresh (TRR), these proprietary solutions are consistently bypassed by researchers. The study exposes significant hurdles in developing universally effective Rowhammer tools, particularly the difficulty in reverse engineering DRAM addressing functions across diverse hardware configurations. Ultimately, the talk advocates for a paradigm shift from security by obscurity to transparent, robust defense mechanisms, stressing the urgent need for a collaborative approach to address this fundamental hardware flaw.

Background

▶ Watch: Introduction and Rowhammer's history at 39C3 (0:00)

To understand Rowhammer, it's essential to grasp the basics of DRAM (Dynamic Random-Access Memory), which serves as the short-term memory in modern computers. DRAM modules, often referred to as DIMMs, are comprised of multiple chips, each containing several banks. Within these banks, data is stored in a grid-like structure of rows and columns. Each memory cell stores a single bit of information as an electrical charge in a tiny capacitor. Due to the inherent physical properties of capacitors, they gradually lose charge over time, necessitating periodic refreshes to maintain data integrity—hence the term "dynamic" RAM. This refresh typically occurs every 64 milliseconds, though it can be faster in newer DDR generations like DDR5.

A critical aspect of DRAM operation is the destructive read process. When data is read from a specific row, the act of sensing the charge destroys the information in that row. To preserve the data, the contents of the activated row are first copied into a temporary storage area called the row buffer. After the read operation, the data is written back from the row buffer to the original row. This read-then-write-back cycle is fundamental to DRAM functionality. Operating systems typically work with 4-kilobyte pages, while DRAM rows are often 8 kilobytes or larger, meaning a single DRAM row can contain multiple OS pages, potentially belonging to different processes.

The Rowhammer vulnerability exploits a physical side effect of this destructive read/write-back process. If a memory row (an "aggressor row") is accessed repeatedly and rapidly, the high electrical activity can induce charge leakage in adjacent or "nearby" memory cells, causing their stored bits to spontaneously "flip" from 0 to 1 or vice versa. These affected cells reside in "victim rows" that were not explicitly accessed by the hammering process. To mount an effective Rowhammer attack, an attacker must ensure that memory accesses genuinely reach the DRAM, bypassing CPU caches. This is typically achieved by using cache-flushing instructions (e.g., clflush on x86) after each memory access. Early Rowhammer techniques involved single-sided hammering (repeatedly accessing one aggressor row) or double-sided hammering (alternating accesses between two aggressor rows sandwiching a victim row), but many more sophisticated patterns have emerged over the years, optimized for different DDR generations and hardware characteristics.

In response to Rowhammer, the memory industry introduced Target Row Refresh (TRR) as a hardware-based countermeasure. TRR operates with two main components: a sampler and an inhibitator. The sampler monitors memory access patterns, counting activations on specific rows. If the access count for a row exceeds a predefined threshold, indicating potential hammering, the sampler alerts the inhibitator. The inhibitator then proactively refreshes the suspected victim rows in the vicinity of the aggressor, thereby preventing bit flips. However, TRR implementations are proprietary, vendor-specific, and often considered "security by obscurity." Research like the Blacksmith paper (mentioned at [07:00]) has demonstrated that TRR can be bypassed by fuzzing for specific access patterns that evade the sampler's detection logic, finding bit flips on all 41 DIMMs tested.

A significant challenge for Rowhammer attackers is the translation between virtual addresses (used by software) and physical addresses (used by hardware), specifically identifying which physical addresses map to physically adjacent DRAM rows on the same bank. This requires reverse engineering the DRAM addressing functions. These functions determine how bits from a physical address are mapped to specific DRAM components like the row index, column index, and bank ID. The talk explains that bank addressing functions often involve XOR operations on various bits of the physical address, combined with bit masks, to derive the bank bits. Researchers use timing side channels (measuring access latency) to infer if two addresses share the same row buffer (indicating they are on the same bank) or different ones, thereby reverse engineering these functions. A verification method involves removing one hypothesized addressing function and observing a predictable drop in the correctness score when grouping addresses, confirming the function's validity.

Key Findings

▶ Watch: Why DRAM is 'Dynamic': Refresh and Destructive Reads (3:00)

The talk presents several crucial findings, encompassing both recent Rowhammer research developments and the empirical results from the FlippyR.AM study.

Recent Rowhammer Research Developments (since 38C3):

1. Column Disturbance Errors: Researchers have discovered that the high electrical activity during Rowhammer is not confined to adjacent rows but can also affect bit lines that span across adjacent DRAM subarrays. This means that repeatedly accessing a row in one subarray can induce disturbance errors in bit lines that extend into neighboring subarrays, potentially causing bit flips in cells that are physically more distant than traditional Rowhammer models predict (as noted at [12:00]). This broadens the scope of potential victim locations.

1. Rowhammer on GDDR6 (GPUs): The vulnerability has successfully been demonstrated on GDDR6 memory, commonly found in modern GPUs. While the eviction strategy differs from CPU caches (requiring specific GPU-centric techniques), researchers were able to degrade machine learning models' accuracy from 80% down to 0.02% through Rowhammer attacks (see [13:00]-[14:00]). Currently, this is more prevalent on server-grade GPU hardware, but the principle confirms that GPUs are not immune.

1. Improved Fingerprinting: Novel research has significantly advanced the ability to fingerprint DRAM modules based on their unique Rowhammer characteristics. A new publication achieved 99.91% fingerprinting accuracy on 98 DIMMs, capable of extracting a fingerprint in less than 5 seconds (as highlighted at [14:00]). While the long-term, large-scale validation was somewhat limited, this capability could have implications for tracking specific vulnerable hardware or even for device identification.

1. ECC Bypass on DDR4: Researchers demonstrated how to bypass Error-Correcting Code (ECC) protection on DDR4 RAM, specifically focusing on Hynix T modules. They achieved this by confusing the proprietary TRR sampler and reverse engineering the ECC matrix. This allowed them to craft specific Rowhammer patterns that resulted in undetectable bit flips, meaning the ECC mechanism would not register them as errors, effectively bypassing a key integrity defense (detailed at [14:00]-[16:00]). This is a significant advancement over previous ECC bypasses, which were often limited to older DDR generations.

1. Post-Hammer Attacks: Exploiting a clause in the DDR standard that allows for postponed refreshes, the "Post-Hammer" paper demonstrated that attackers can precisely time their hammering activities to occur just before a batch of postponed refreshes is executed. This window of opportunity allows for more effective hammering and subsequent bit flips, even when refreshes are eventually performed (mentioned at [16:00]).

1. DDR5 Challenges and Bypass: While DDR5 incorporates several new mitigations—including a higher refresh rate and a new refresh command designed to protect neighboring rows—these are often proprietary and optional. DDR5 also features on-die ECC. Crucially, the talk highlights that the TRR sampler on DDR5 can skip certain memory access intervals, creating windows of inactivity. The Phoenix paper exploited this behavior, launching attacks precisely when the sampler was not active, thereby bypassing DDR5's TRR (as discussed at [17:00]-[18:00]).

FlippyR.AM Study Results (Large-Scale Data Collection):

The FlippyR.AM study, conducted with user participation, involved distributing bootable USB thumb drives to collect data on real-world systems. The study faced numerous practical challenges, including issues with cheap USB sticks, flashing hundreds of drives, and even a bug that led to two different hashes for the bootable image (described at [19:00]-[21:00]).

1. Addressing Function Reverse Engineering Failure: A critical finding was that reverse engineering DRAM addressing functions failed on 50% of the 1,060 collected datasets (at [22:00]). Reasons included lack of 1GB huge page support (required by some tools), tool crashes, or tools running but failing to report valid functions. This represents a significant open challenge for the research community, hindering the development of universal Rowhammer tools.

1. Rowhammer Tool Success Rates: While tools like Flip Floit and Rowhammer Test generally had a return code of zero (meaning they didn't crash), many tools failed due to the missing addressing functions or inability to acquire 1GB huge pages. The Blacksmith tool, for instance, had additional compilation issues when addressing functions were broken (detailed at [23:00]-[24:00]).

1. Widespread Vulnerability: Despite the tooling challenges, the study found that 126 systems, or 12.5% of all tested systems, were vulnerable to fully automated Rowhammer attacks (as stated at [24:00]). This implies that Rowhammer is "weaponizable right now" on a significant portion of the installed computer base.

1. DRAM Generation Specifics: Bit flips were observed exclusively on DDR3 and DDR4 systems. For DDR3, simple, fast hammering patterns proved most effective. For DDR4, with TRR enabled, more complex, non-uniform, fuzzing-based patterns were necessary to trigger bit flips (explained at [27:00]-[28:00]).

1. CPU Vendor Bias: The study observed a higher number of bit flips on Intel processors compared to AMD. However, the speakers attribute this bias to the historical focus of Rowhammer tooling on Intel architectures. They explicitly state that AMD is expected to be equally affected, as the vulnerability lies in the DRAM, not the CPU. Some bit flips were still found on AMD systems using one-location hammering, which doesn't require addressing functions (discussed at [26:00]).

1. DRAM Vendor Differences: Samsung and other resellers' DIMMs appeared similarly affected, while Micron DIMMs showed fewer bit flips. Again, this difference might be related to research tools being more focused on specific vendors due to the complexities of reverse engineering (at [28:00]).

Technical Deep Dive

▶ Watch: Introducing Rowhammer: The Disturbance Error (4:30)

The Rowhammer vulnerability fundamentally exploits the physical characteristics of modern DRAM. Each bit in DRAM is stored as a charge in a capacitor, which is inherently leaky. To maintain data, these capacitors must be periodically refreshed. A DRAM module is organized hierarchically: channels allow parallel memory access, containing DIMMs (Dual In-line Memory Modules), which have ranks (sides with chips). Each chip is further divided into multiple banks to enable internal parallelism. Within a bank, data is arranged in rows (typically 8 kilobytes) and columns.

When a CPU requests data from a specific memory address, the entire 8-kilobyte row containing that address is first activated. This activation process involves reading the row's contents into a temporary storage area called the row buffer. As mentioned, this read is destructive; the original row loses its charge. After the data is copied to the row buffer, it can be accessed by the CPU. To preserve the original data, the contents of the row buffer are then written back to the original row. This activate-read-precharge-write-back cycle is critical.

The Rowhammer effect occurs during this rapid activation and precharge process. When an aggressor row is repeatedly activated and precharged in quick succession, the high voltage fluctuations and current surges can induce voltage noise in adjacent or physically close victim rows. This noise can cause the small charge in the victim row's capacitors to leak faster than usual, or even cause a charge to build up where there was none, leading to a bit flip. The effectiveness of Rowhammer depends on the speed and frequency of these activations, requiring the attacker to bypass the CPU's caches (e.g., L1, L2, L3) using instructions like clflush to ensure that memory accesses directly hit the DRAM.

A significant technical hurdle is translating virtual addresses (used by applications) to physical addresses (used by the memory controller) and then mapping these physical addresses to specific DRAM components like rows, columns, and banks. The DRAM addressing functions are proprietary and vary between manufacturers and even models. These functions determine how bits from a physical address are permuted and combined to form the row index, column index, and bank ID. For Rowhammer, identifying addresses that map to the same DRAM bank and physically adjacent rows is paramount.

The speakers detailed the reverse engineering process for bank addressing functions (at [09:00]). A physical address is broken down into its constituent bits. Some bits directly correspond to the row index, others to column index, and an offset for the cache line. The challenge lies in determining the bank bits. These are often derived by applying XOR operations to several non-contiguous bits of the physical address. For example, bank_bit_0 = (physical_address_bit_X XOR physical_address_bit_Y). Multiple such XOR combinations are used to generate all necessary bank bits. These combinations can be represented using bit masks, where a 1 indicates a bit used in the XOR combination and a 0 indicates one not used. If, for instance, four bank bits are identified, this allows addressing $2^4 = 16$ distinct DRAM banks.

To verify the correctness of these reverse-engineered functions, a method published by the speakers involves a correctness score (at [11:00]). If a hypothesized bank addressing function is correct, removing one of its constituent XOR combinations (effectively "arbitrarily setting" one bank bit) should halve the number of addressable banks. For a system with 16 banks (4 bank bits), removing one correct function should reduce the effective addressability to 8 banks (a 50% correctness score), removing two to 4 banks (25%), and so on. Deviations from this expected exponential decay indicate incorrect functions.

Target Row Refresh (TRR), the primary hardware countermeasure, attempts to mitigate Rowhammer by dynamically refreshing victim rows. The TRR sampler monitors the frequency of activations for specific rows. If a row's activation count exceeds a vendor-defined threshold within a certain time window, the inhibitator is triggered. The inhibitator then issues refresh commands to the suspected victim rows in the vicinity of the aggressively accessed row. However, as demonstrated by the Blacksmith paper, TRR can be bypassed by crafting non-uniform access patterns that evade the sampler's counting logic or by exploiting specific timing windows.

The talk highlighted several advanced attack techniques:

• Column Disturbance Errors (at [12:00]) represent a broader form of disturbance. Instead of just adjacent rows, the electrical noise propagates along bit lines that might traverse multiple DRAM subarrays. This means a single row activation can affect columns in distant, non-adjacent memory regions.
• The ECC bypass on DDR4 (at [14:00]) is particularly sophisticated. It involves not only bypassing TRR but also understanding the specific ECC matrix used by the DRAM vendor. By knowing which bits are correlated by the ECC, attackers can strategically flip multiple bits (e.g., exactly four bits, as mentioned at [35:00]-[36:00]) in such a way that the resulting error pattern is either correctable without being reported or falls into a state that the ECC mechanism cannot detect as an error, thus achieving undetectable bit flips.
• DDR5 introduces higher refresh rates, a new refresh command that refreshes neighboring rows, and on-die ECC. However, the Phoenix paper exploited a flaw in DDR5's TRR sampler: it periodically becomes inactive, creating "skip" intervals where it doesn't count memory accesses (at [17:00]-[18:00]). By precisely timing Rowhammer attacks to these inactive windows, attackers can bypass the DDR5 TRR.

Demo / Proof of Concept

▶ Watch: Breaking TRR: The Blacksmith Fuzzing Technique (7:00)

The core "demo" or proof of concept discussed in the talk is the FlippyR.AM study itself, a large-scale, real-world data collection effort. The study aimed to assess the prevalence and exploitability of Rowhammer on a diverse set of user systems.

The methodology involved distributing bootable USB thumb drives to volunteers. The process for participants was:

1. Agree to participate in the study.
2. The system would collect basic hardware information.
3. It would then attempt to reverse engineer the DRAM addressing functions for the system's memory.
4. These functions would be verified using the correctness score approach described earlier.
5. Finally, a suite of various Rowhammer tools (e.g., roheamjs, blacksmith, Flip Floit, Rowhammer Test) would be executed.
6. If the user consented, the results, including system information and bit flip data, would be uploaded to the researchers' servers.

The speakers provided a humorous and candid account of the logistical challenges encountered during the setup and execution of the study (at [19:00]-[21:00]):

• Mass Production of USB Sticks: The team, notably Daniel's family, flashed thousands of cheap USB sticks, often working through holidays.
• Software Bugs: An initial bug in the framework meant the first 700 flashed drives had an 8-hour runtime for the test, leading to two different hashes for the bootable image.
• Hardware Quality Issues: The cheap USB sticks had physical problems like "real scratches" and unreliable connections. The team resorted to using tape, only to find the labels peeled off with the tape. Their "engineering flowchart" involved using WD40 to remove glue residues and isopropanol to clean the drives, highlighting the unexpected practical difficulties of a large-scale hardware-focused study.

Despite these hurdles, the FlippyR.AM framework was designed for extensibility and reproducibility (at [31:00]-[32:00]). Researchers can easily add new Rowhammer tools to the framework, allowing for comparison against existing tools and testing on additional systems (e.g., university computer labs). The use of an ISO image also helps in reproducing results by ensuring consistent software versions and configurations across different test environments. The study itself, by collecting and analyzing real-world data, served as a large-scale proof of concept for the widespread applicability and challenges of Rowhammer.

Defensive Implications

▶ Watch: Reverse Engineering: Virtual to Physical Address Mapping (7:45)

The FlippyR.AM study and the ongoing research presented in the talk underscore the urgent need for robust, transparent, and universally applicable defensive strategies against Rowhammer. The current industry approach, heavily reliant on proprietary Target Row Refresh (TRR) mechanisms, is repeatedly shown to be insufficient, as researchers consistently find ways to bypass these "security by obscurity" measures.

The speakers strongly advocate for a fundamental change in mindset from hardware vendors: publish their approaches (at [28:00]-[29:00]). This transparency would allow the broader security community to scrutinize and improve mitigations, fostering a collaborative approach rather than a perpetual cat-and-mouse game.

Several potential defensive avenues were discussed:

1. Manufacturing Process Improvements:

• Increased DRAM Cell Size: Returning to larger cell sizes, common in older DDR generations, would inherently make cells more resistant to charge leakage and interference. However, this comes at a significant cost (at [29:00]), as it reduces memory density and increases manufacturing expenses, making it economically unfeasible for consumer-grade DRAM.

1. Architectural and Isolation-Based Defenses:

• Spatial Isolation: Distributing data more widely across memory to avoid physically adjacent sensitive data. However, this is increasingly challenged by new attack vectors like column disturb errors, which affect physically more distant cells (at [30:00]).
• Changing Refresh/Access Ratio: This is essentially what TRR attempts to do. While it hasn't been fully effective, refining the balance between memory accesses and refresh commands is still a research area.
• Memory Band-Aid: This proposed solution aims to limit the memory bandwidth per process (at [30:00]-[31:00]). By reducing the number of activations an attacker can perform per refresh cycle, it limits the effectiveness of Rowhammer. This would ideally require a per-bank limit in hardware, which current processors typically lack, though a per-process limit could still offer low overhead for untrusted processes. This would likely require hardware changes in memory controllers.

1. Cryptographic Defenses:

• CSI-Rowhammer (Cryptographic Scrambling and Integrity for Rowhammer): This is presented as a promising, strong security guarantee (at [30:00]-[31:00]). The idea is to store cryptographic hashes of memory contents on a separate, dedicated bank. These hashes can then be used to verify the integrity of data and detect bit flips by brute-forcing and checking the checksums. This approach could serve as a replacement or enhancement for ECC, offering stronger assurances. During the Q&A, CSI-Rowhammer was recommended if an additional ECC chip is available, otherwise TRR or Memory Band-Aid (at [33:00]).

1. Software and Tooling Improvements:

• The study highlighted that the inability to reverse engineer addressing functions on 50% of systems is a major bottleneck (at [32:00]). Improving the accuracy and universality of addressing function reverse engineering tools is crucial for both detection and mitigation efforts, as it would enable more effective targeting of vulnerable rows.
• Developing more Rowhammer tools for diverse platforms (e.g., AMD, LPDDR, embedded devices) is also necessary to get a full picture of the vulnerability landscape and create comprehensive defenses. While LPDDR and embedded devices might behave differently due to voltage and clock rate variations, the underlying principle of DRAM vulnerability remains (at [34:00]-[36:00]).

Ultimately, the conclusion is that current proprietary mechanisms are insufficient. A shift towards open, verifiable, and hardware-supported defense mechanisms is necessary to move beyond the cycle of bypasses and truly secure DRAM against Rowhammer.

Key Takeaways

• Widespread Vulnerability: Despite years of research and industry mitigations, Rowhammer remains a critical threat, with the FlippyR.AM study demonstrating that 12.5% of real-world systems are vulnerable to fully automated Rowhammer attacks today, with potential for this figure to double with improved tooling.
• Failure of Security by Obscurity: Proprietary, hardware-based countermeasures like Target Row Refresh (TRR) are consistently bypassed by researchers through sophisticated patterns and exploitation of implementation flaws (e.g., Blacksmith, Phoenix), highlighting the inadequacy of non-transparent security mechanisms.
• Addressing Function Reverse Engineering is a Bottleneck: A major challenge for practical Rowhammer exploitation and defense is the difficulty in reverse engineering DRAM addressing functions, which failed on 50% of systems in the study. Universal and reliable methods for this are crucial for developing effective tools.
• Evolving Attack Surface: New Rowhammer variants continue to emerge, including column disturb errors (affecting distant cells), attacks on GDDR6 (GPUs), and advanced ECC bypasses on DDR4 (crafting undetectable bit flips), indicating a continuously expanding threat landscape.
• DDR5 Not Immune: While DDR5 introduces new mitigations like higher refresh rates and on-die ECC, it also presents new bypass opportunities, such as exploiting TRR sampler skips, demonstrating that newer memory generations are not inherently secure against Rowhammer.
• Need for Fundamental Defensive Shift: Effective long-term defenses require a move away from proprietary, reactive patches towards open standards, transparent mitigation designs, or fundamental hardware changes like CSI-Rowhammer (cryptographic integrity) or Memory Band-Aid (bandwidth limiting) to truly address the root cause of the vulnerability.

About the Speaker(s)

The talk "Rowhammer in the Wild: Large-Scale Insights from FlippyR.AM" was delivered by Martin Heckel, Florian Adamsky, and Daniel Gruss.

Daniel Gruss is a well-known figure in the field of hardware security, particularly for his extensive research on Rowhammer and speculative execution vulnerabilities (e.g., Meltdown, Spectre). He opened the talk by noting his special connection to the Congress, as his first talk on Rowhammer was given on the same stage exactly 10 years prior, in 2015. This underscores his long-standing commitment and expertise in this area, having been at the forefront of Rowhammer research since its early days.

Martin Heckel and Florian Adamsky are also active researchers in the field, working alongside Daniel Gruss. Their contributions to the FlippyR.AM study, including the development of the framework and the arduous process of preparing the test environment (such as flashing thousands of USB sticks), were instrumental in gathering the large-scale empirical data presented in the talk. Their work reflects a deep technical understanding of DRAM internals and the practical challenges of hardware security research. Together, they represent a leading team in uncovering and analyzing hardware-level vulnerabilities.