LoRDMA: A New Low-Rate DoS Attack in RDMA Networks

LoRDMA: A New Low-Rate DoS Attack in RDMA Networks

Speakers: Shicheng Wang

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=rV5Ru2M4UQs

Overview

The proliferation of Remote Direct Memory Access (RDMA) from private data centers into multi-tenant cloud environments has ushered in unprecedented performance benefits, primarily by enabling direct memory-to-memory communication that bypasses the CPU and traditional network stack. While this paradigm, particularly with RoCEv2 encapsulating InfiniBand over UDP, delivers ultra-low latency and high throughput, it also introduces a new frontier for security vulnerabilities, especially concerning the network's traffic control mechanisms. This talk, presented by Shicheng Wang at the NDSS Symposium, unveils a novel low-rate Denial-of-Service (DoS) attack named LoRDMA, which specifically targets the interplay between Priority-based Flow Control (PFC) and Datacenter QCN (DCQCN) in RDMA networks.

LoRDMA exploits a critical, previously overlooked security vulnerability: the "PFC-misleading-DCQCN" interaction. The research demonstrates that carefully crafted, short-duration bursts of traffic can cause drastic and widespread performance degradation on legitimate flows, even those several hops away from the direct point of attack. The attack is adaptive, leveraging Round-Trip Time (RTT) as a feedback signal to intelligently coordinate compromised bots and precisely schedule burst parameters for maximum impact with minimal detectability. Evaluations in both ns-3 simulations and a real-world cloud RDMA cluster (Kuaishou and Alibaba Cloud) show LoRDMA's superior effectiveness, achieving significantly higher victim flow coverage and performance loss (e.g., 18.23% to 56.12% degradation for NCCL Tests) compared to existing attacks, all while maintaining a substantially lower attack traffic volume.

This groundbreaking work serves as a crucial wake-up call for the RDMA community, including researchers, engineers, and operators. It highlights the urgent need to re-evaluate the security posture of traffic control mechanisms in lossless RDMA networks, which are foundational to modern high-performance computing and distributed machine learning infrastructures. The findings underscore that the very mechanisms designed to ensure lossless transmission can be maliciously manipulated to induce widespread and stealthy performance disruption.

Background

▶ Watch: RDMA security, PFC-DCQCN vulnerability, LoRDMA overview (0:00)

RDMA is a technology designed to enable high-throughput, low-latency, and low-CPU-occupation remote memory access. It achieves this by allowing network adapters (RNICs) to directly access memory from a remote computer without involving the remote computer's CPU or operating system, eliminating multiple data copies and bypassing the kernel. RDMA applications utilize verbs APIs to establish Queue Pairs (QPs), which can be configured for reliable connection, unreliable connection, or unreliable datagram communication. Data transmission can be two-sided (requiring both SEND and RECV operations) or one-sided (WRITE, READ, ATOMIC operations that bypass CPU involvement entirely). RNICs offload the entire transport protocol stack, contributing to ultra-high performance.

A critical requirement for RDMA is a lossless network to prevent retransmissions that would negate its latency benefits. To achieve this, several traffic control mechanisms are employed. Priority-based Flow Control (PFC), standardized by IEEE 802.1Qbb, is a hop-by-hop mechanism commonly used in RoCEv2 (RDMA over Converged Ethernet version 2). When an egress queue on a switch exceeds a predefined threshold (X-OFF), the switch sends a PAUSE frame to the upstream device, temporarily halting transmission for a specific priority class. Transmission resumes when the queue drains below another threshold (X-ON). While effective at preventing packet loss, PFC is known to introduce issues such as Head-of-Line Blocking (HLB), unfairness, PFC storms, and even network deadlocks, especially in complex topologies.

To mitigate the negative side effects of PFC, end-to-end congestion control (CC) mechanisms are introduced. Datacenter QCN (DCQCN), the default congestion control algorithm in NVIDIA/Mellanox RNICs, is widely adopted. DCQCN operates by having switches act as Congestion Points, monitoring egress queue lengths and marking packets with ECN (Explicit Congestion Notification) using a RED (Random Early Detection)-like algorithm when congestion is detected. Crucially, ECN marking is typically configured to trigger before PFC PAUSE frames are generated. Receivers, or Notification Points, respond to ECN-marked packets by sending CNP (Congestion Notification Packet) messages back to the senders. Senders, or Reaction Points, then adjust their transmission rates using an Additive-Increase/Multiplicative-Decrease (AIMD) mechanism, cutting rates rapidly in response to CNPs and recovering moderately. Unlike TCP, DCQCN flows typically begin at full line rate, without a slow start phase. While other CC algorithms like QCN, TIMELY, and HPCC exist, PFC and DCQCN remain the most prevalent in production RDMA networks.

Prior research into RDMA security has largely concentrated on system-level vulnerabilities, including issues related to confidentiality, integrity, authenticity, resource exhaustion, and side-channel attacks. Studies addressing transport layer security in RDMA are scarce. Some work has explored the interaction between PFC and DCQCN to improve congestion control efficiency, but not to intentionally exploit it for malicious purposes. For instance, Snyder et al. identified ways to exploit DCQCN for unfair bandwidth allocation but did not consider the broader impact of PFC. In the broader context of network security, low-rate DoS attacks are a classic type of DoS that target feedback-based control systems. They achieve their effect by crafting short, high-volume requests to temporarily overwhelm systems and trigger misleading feedback signals. Examples include TCP-based attacks like Shrew, which exploits AIMD and RTO mechanisms, RoQ attacks analyzing Lyapunov stability, and TCPwn using fuzzing techniques. The LoRDMA research distinguishes itself by being the first to analyze low-rate DoS attacks in RDMA networks, specifically focusing on the complex, coupled interaction between PFC and DCQCN, and qualitatively revealing attack impact convergence with RTT, rather than relying solely on complex mathematical modeling.

Key Findings

▶ Watch: LoRDMA evaluation: high impact, low detectability (1:15)

The core discovery of this research is the identification of PFC-misleading-DCQCN security vulnerabilities within RDMA traffic control mechanisms. This previously unaddressed interaction allows short-duration, high-rate traffic bursts to inflict drastic and widespread performance loss on legitimate flows, impacting connections across multiple network hops. The researchers meticulously established that this performance degradation is directly and predictably linked to the burst's peak rate (δ) and its duration (τ), providing a crucial foundation for crafting highly efficient low-rate DoS attacks.

The study revealed two distinct types of victim flows and their degradation mechanisms:

1. Direct Victim Flows: These flows experience immediate and severe performance loss when their packets directly contend with attack bursts in an egress queue. The congestion leads to ECN marking, and DCQCN multiplicatively cuts their rates, often to zero, with slow AIMD recovery.
2. Indirect Victim Flows: Surprisingly, these flows suffer significant performance degradation even without direct link sharing or queue contention with the attack bursts. This occurs because the initial congestion caused by the bursts triggers PFC PAUSE frames, which propagate upstream, transiently dominating the control system and spreading congestion to other switches. This upstream congestion misleads DCQCN on indirect paths, causing unintended rate cuts.

Based on these fundamental insights, the researchers proposed LoRDMA, a novel low-rate Denial-of-Service attack. LoRDMA is designed to be adaptive, utilizing Round-Trip Time (RTT) as a critical feedback signal. This allows attackers to intelligently coordinate compromised bots to different target switch ports, maximizing victim flow coverage (both direct and indirect), and to precisely schedule burst parameters (duration and period) to achieve optimal performance loss.

Extensive evaluations, conducted in both ns-3 simulations and real-world cloud RDMA clusters (Kuaishou and Alibaba Cloud), conclusively demonstrated LoRDMA's superior effectiveness and efficiency. Compared to existing baseline attacks, LoRDMA achieved significantly higher victim flow coverage and greater performance loss while maintaining a substantially lower attack traffic volume and, consequently, lower detectability. For instance, distributed machine learning training applications using NCCL Tests experienced performance degradation ranging from 18.23% to a staggering 56.12% under a LoRDMA attack. This work marks the first study to identify these specific PFC-misleading-DCQCN vulnerabilities that can impact traffic multiple hops away and to propose a well-crafted attack exploiting them, serving as a critical reminder for the RDMA community to re-evaluate the security of traffic control in lossless RDMA networks.

Technical Deep Dive

▶ Watch: RDMA basics, PFC, and DCQCN traffic control (2:00)

To understand and quantify the impact of traffic bursts on RDMA networks, the researchers initiated a systematic measurement study. The experimental setup utilized an ns-3 simulated testbed based on the open-source DCQCN project. The chosen topology, depicted in Figure 1(a) in the original presentation, represented a common CLOS network unit: four senders (S1-S4) communicating with three receivers (R1-R3) via top-of-rack (ToR) switches (SW1-SW4) and intermediate leaf switches (SW3-SW4), converging at a target ToR switch (SW5). All links were configured for 100Gbps with a 5µs propagation delay. Legitimate flows (F1-F4) were initiated from S1-S4 to R1-R3. Attackers, or bots (A1-An), were positioned to manipulate their RNICs to generate line-rate bursts, specifically targeting and congesting the egress port SW5.P4, with the goal of degrading legitimate flows. All PFC and DCQCN parameters were set to their default recommended values.

The bursts were characterized by three key parameters: peak rate (δ), duration (τ), and period (T). δ was defined as an integer multiple of the line rate, representing the number of bots simultaneously sending line-rate bursts. To quantify the attack's impact, three performance metrics were introduced:

• Performance Loss (PL): The integral of the difference between the original flow rate (R0) and the instantaneous rate (R(t)) over the burst duration τ, geometrically represented as the shadowed area in Figure 1(c).
• Performance Loss Factor (ΔR): The difference between the original flow rate (R0) and the lowest flow rate observed during the bursts (R_low).
• Normalized Performance Loss Factor (Norm ΔR): ΔR/R0, providing a normalized measure of severity.

The experiment results revealed distinct degradation mechanisms for two types of victim flows. Direct Victim Flows (e.g., F2, F3) experienced straightforward degradation: the attack bursts rapidly congested the egress queue SW5.P4, leading to ECN marking of F2 and F3 packets. DCQCN then multiplicatively cut their rates, often to zero. Recovery was slow due to DCQCN's AIMD property. For these flows, the ΔRd (Performance Loss Factor for direct flows) gradually increased with τ, converging to R0 when τ was sufficiently long. It became relatively constant with increasing bot numbers, with as few as two bots capable of cutting the rate to nearly zero. The PLd (Performance Loss for direct flows) increased super-linearly with τ for short durations (approximated as (ΔRd)²/2k) and then linearly when τ was long enough and the flow rate approached zero (R0\*τ).

More surprisingly, Indirect Victim Flows (e.g., F1, F4) also suffered significant performance degradation despite having no direct link sharing or queue contention with the attack bursts. Their congestion points were upstream at SW3.P2 and SW4.P2. For these flows, ΔRi (Performance Loss Factor for indirect flows) increased with τ, becoming constant when τ was long. It also increased with δ, converging at higher δ, but with diminishing returns after approximately 500Gbps of attack traffic. The PLi (Performance Loss for indirect flows) increased with τ for short durations but converged to an upper bound (constant) when τ was long enough. This upper bound increased with δ, also with diminishing returns.

The underlying mechanism for this indirect performance loss is crucial: when attack bursts overwhelmed SW5.P4, packets from F2 and F3 accumulated rapidly in their ingress ports (SW5.P1 and SW5.P2). These queue lengths quickly exceeded the PFC X-OFF threshold, triggering the generation of PFC PAUSE frames upstream to switch ports like SW3.P2 and SW4.P2, which then halted the transmission of F1 and F4 packets. Since PFC operates at microsecond levels, significantly faster than DCQCN's 10s of microsecond update periods, PFC transiently dominates the control system, effectively spreading congestion upstream. This congestion spread by PFC then misleads DCQCN's congestion detection at SW3.P2 and SW4.P2, causing an additional, unnecessary rate cut on F1 and F4. Later in the burst, DCQCN eventually dominates, cutting F2/F3 to zero and thereby mitigating the upstream queue buildup, allowing F1/F4 to recover.

These findings formed the basis for the LoRDMA attack design, which exploits two critical vulnerabilities:

1. PFC-driven Congestion Spreading: Attackers can strategically congest a few switch port queues with short, high-load bursts. PFC then broadly and indirectly shuts down many more links, even those not directly contended by the bursts. This achieves high victim flow coverage with fewer directly congested links and lower link load, significantly reducing detectability by traditional DoS defenses.
2. Long-term Degradation from Short Bursts: Due to DCQCN's AIMD property, a burst lasting less than 1ms can reduce victim flow rates to nearly zero, but recovery takes tens of milliseconds. This allows attackers to suppress average victim flow rates significantly (e.g., a 50Gbps flow reduced to a 1.6Gbps average rate with a 1ms burst and 60ms period) using very low average burst traffic (1Gbps). Such short, low-rate bursts are difficult for prevalent second-level network monitoring tools to capture and are often dropped by RNICs without end-host notification, further reducing detectability.

LoRDMA's design adheres to two key principles: maximizing impact while minimizing detectability by exploiting the indirect victim flow effect, and optimizing burst parameters (τ and T) for an efficient trade-off between performance loss and average burst rate. The threat model assumes an adversary within a shared RDMA network (cloud or private cluster) with malicious hosts (bots) capable of crafting line-rate burst traffic. This is feasible because RDMA networks allow line-rate flow starts, attackers can generate line-rate "mice flows" shorter than 1 RTT (uncontrollable by end-to-end CC), and mainstream RNICs (e.g., Mellanox CX-4 and newer) support IBV_QPT_RAW_PACKET for kernel-bypass raw Ethernet programming. Attackers are assumed to have some knowledge of the network topology and potentially target flows. The primary challenges are bot coordination (an NP-hard problem) and burst scheduling, both of which are difficult due to the complex, coupled PFC-DCQCN system and limited attacker information.

The key observation addressing these challenges is that RTT (Round-Trip Time) serves as a natural side-channel signal that qualitatively reflects whether the attack impact (ΔR and PL) converges with increasing δ and τ. RTT is linearly mapped to queue length, which directly correlates with congestion severity. Traditional RTT probing tools are inaccurate for RDMA due to QoS differences, but LoRDMA leverages a side channel: RDMA CM ConnectReject packets, sent in response to ConnectRequest from unknown hosts, are in the RoCEv2 protocol and accurately reflect RDMA RTT. This enables bots to craft these requests and monitor RTT.

LoRDMA employs a two-step adaptive attack:

1. Coordination Procedure (Algorithm 1): This procedure determines which egress port each bot should target to maximize high and wide performance loss, especially on indirect victim flows. It uses a heuristic greedy algorithm to select target ports based on a weighted sum of covered indirect and direct victim flows. Bots are incrementally deployed to selected ports, and RTT sequences for both direct (RTTd) and indirect (RTTi) victim flows are monitored. Bots are added until the range of RTTi (max RTT - min RTT) approximately equals the range of RTTd, indicating that ΔRi has converged for that port. Bots with the lowest "footprint" (lower ingress link flow density) are prioritized for reduced detectability.
2. Schedule Procedure (Algorithm 2): This procedure aims to find an efficient burst duration (τ) that maximizes performance loss. The RTTi sequence during a burst typically shows a two-phase pattern: a peak followed by a gradual fall, as DCQCN eventually cuts direct victim flows, alleviating upstream congestion. The key insight is that the lower RTTi subsequence can be trimmed without reducing attack impact, while the higher RTTi subsequence cannot. The algorithm uses a bisection search method to adaptively reduce τ, filtering the RTTi sequence to identify the point where trimming τ starts to cut into the peak congestion pattern, thereby finding the optimal τ that maintains impact efficiently.

Demo / Proof of Concept

▶ Watch: Experimental setup for burst impact measurements (4:00)

The efficacy and practicality of LoRDMA were rigorously validated through a prototype implementation, with the attack tools made publicly available on GitHub. The implementation focused on two critical components: the Burst Generator and the RTT Prober.

The Burst Generator was engineered to meet two primary requirements:

1. Craft Line-Rate Bursts: To achieve the necessary high-rate, short-duration traffic, the generator leveraged advanced RNIC capabilities. Specifically, modern Mellanox CX-4 and newer RNICs, coupled with RDMA Core driver v5.0 and newer, support IBV_QPT_RAW_PACKET. This new Queue Pair type enables high-performance kernel-bypass raw Ethernet programming directly on the RNIC. Attackers can utilize this to craft packets at line rate to any host in the network. To maximize bandwidth utilization, packets were set to the Maximum Transmission Unit (MTU) length. The destination IP address was carefully chosen to ensure the packets traversed the intended target egress port. Crucially, the DSCP field (Type of Service bits in the IP header) was precisely set to classify the packets as legitimate RoCEv2 traffic, preventing misclassification as normal TCP/IP traffic and ensuring they interact with the RDMA traffic control mechanisms as intended.
2. Control Burst Duration (τ) and Period (T): Precise temporal control over bursts is fundamental to LoRDMA. The generator recorded time during burst crafting, and to avoid consuming excessive CPU cycles for this task, a dedicated parallel thread was created to function as an accurate timer.

The RTT Prober was developed to provide the crucial feedback signal for LoRDMA's adaptive mechanisms:

1. Side-Channel Probing: The prober capitalized on the raw Ethernet programming feature of the RNIC to craft CM ConnectRequest packets. These are part of the RDMA Connection Manager (CM) protocol.
2. Reply Sniffing: The RNIC was configured to sniff for the corresponding ConnectReject reply packets. These replies are typically sent by a remote host when it receives a connection request from an unknown or unauthorized source.
3. RTT Estimation: By accurately subtracting the timestamp of the outgoing ConnectRequest from the timestamp of the incoming ConnectReject packet, the RTT to any host in the network could be reliably estimated. This side-channel approach provided an accurate measure of RDMA RTT, bypassing the inaccuracies of traditional probing tools in QoS-differentiated RDMA environments.

Extensive evaluation and results confirmed the practical feasibility and significant impact of LoRDMA across various environments.

Burst Generator Validation (Figure 10a) demonstrated that the custom generator could indeed achieve line-rate bursts: 40Gbps in a lab testbed, 100Gbps in a Kuaishou cloud RDMA cluster, and 25Gbps in an Alibaba Cloud instance. This validated its ability to precisely control burst duration and period.

The RTT Prober Validation (Figure 10b) showed a stable correlation between the estimated RTT and actual network delay, accurately reflecting both low delays under normal conditions and the long-tailed delays characteristic of network congestion during an attack.

The most compelling proof of concept involved deploying LoRDMA against NCCL Tests in a real-world Kuaishou cloud RDMA cluster. NCCL (NVIDIA Collective Communications Library) is widely used for collective communication primitives in distributed machine learning training. The attack targeted typical collective operations: AllReduce, AllGather, ReduceScatter, and AlltoAll. An indirect attack scenario was set up, where bursts from hosts H4, H7, and H8 targeted host H2, impacting victim flows on H1, H5, and H6. The burst duration was set to 10ms with a period of 110ms.

The results (Figure 11a) revealed significant performance degradation for all tested primitives:

• AllReduce: 35.09% degradation
• AllGather: 56.12% degradation
• ReduceScatter: 22.71% degradation
• AlltoAll: 18.23% degradation

Further network diagnostics (Figure 11b) during the AllReduce attack empirically validated the core mechanism. It was observed that ToR1, an upstream switch, generated a large number of PFC PAUSE frames, while ToR3, another switch in the path, received numerous CNP packets. This confirmed the researchers' key insight: the PFC back-spread mechanism severely misleads DCQCN, causing the observed widespread performance degradation. These real-world results unequivocally demonstrate LoRDMA's practical feasibility and its devastating impact on critical RDMA applications in cloud environments.

Defensive Implications

▶ Watch: Mechanism of direct victim flow performance loss (4:45)

The LoRDMA attack exposes fundamental weaknesses in how RDMA networks handle congestion and highlights the inadequacy of current defense mechanisms. Addressing this novel threat requires a multi-faceted approach, focusing on enhancing detection, monitoring, and architectural resilience.

1. PFC-driven Network Anomaly Detection and Analysis: Traditional DoS defenses primarily focus on metrics like link load or queue contention. However, LoRDMA's effectiveness stems from PFC's ability to spread congestion indirectly across multiple hops with low direct link utilization, making these conventional metrics less effective. Future defense systems must incorporate an understanding of PFC PAUSE frame spreading into their anomaly detection logic. This involves building causality dependency constructions that can accurately trace the propagation of congestion triggered by PFC, thereby pinpointing the true root cause of performance degradation and identifying the source of attack traffic. Without this context, an attack might appear as disparate, unrelated slowdowns across the network, obscuring the coordinated low-rate DoS.

1. Fine-grained Burst Monitoring: LoRDMA's stealth is largely attributed to its use of short-duration, low-rate bursts, which are notoriously difficult for prevalent second-level network monitoring tools to capture. The attack's characteristics, such as the line-rate start property of DCQCN and PFC's rapid response, mean that bursts are not "outstanding" enough to trigger alarms in coarse-grained monitoring systems. There is an urgent need for millisecond- or even microsecond-level monitoring granularity to detect these transient, yet impactful, bursts. Developing such responsive, precise, and low-overhead mechanisms is inherently challenging in the context of RDMA's hardware-kernel-bypass paradigm, where visibility into the network stack is limited. This may require new in-network telemetry techniques or specialized hardware-based monitoring solutions.

1. Network Fabric Level Isolation & Virtualization: The feasibility of LoRDMA fundamentally relies on the shared nature of network infrastructure in multi-tenant environments. Implementing fabric isolation, which provides strict, bounded bandwidth guarantees per tenant or virtual network, could theoretically eliminate the resource sharing that LoRDMA exploits. Similarly, advanced virtual RDMA overlay networking might prevent attackers from accurately probing or understanding the physical network topology, complicating bot coordination and target selection. However, a significant challenge remains: even with logical isolation, the underlying physical infrastructure could still be impacted by attacks, potentially degrading the performance of other tenants sharing the same physical resources. Further research is required to fully understand how fabric-level isolation and virtualization can be robustly applied to mitigate LoRDMA without introducing new performance bottlenecks or vulnerabilities.

Beyond these direct defense schemes, the research also opens avenues for future work that could inform better defenses. Understanding the effect of background traffic on attack impact and RTT measurement accuracy, or the attack feasibility with different queue disciplines (where ideal fair queuing is currently impractical at line-rate), could lead to more robust network designs. Investigating the extensibility of LoRDMA to other AIMD-based congestion control variants (like QCN, TIMELY) and the complexities introduced by ECMP (Equal-Cost Multi-Path) routing, will further refine the threat landscape and guide the development of comprehensive countermeasures.

Key Takeaways

• The interaction between Priority-based Flow Control (PFC) and Datacenter QCN (DCQCN) in lossless RDMA networks creates a critical, previously overlooked security vulnerability.
• LoRDMA is a novel, low-rate Denial-of-Service attack that exploits this "PFC-misleading-DCQCN" vulnerability, enabling widespread, multi-hop performance degradation on legitimate RDMA flows.
• The attack is highly adaptive, using Round-Trip Time (RTT) as a real-time feedback signal to intelligently coordinate compromised bots and precisely schedule burst parameters (peak rate, duration, period) for maximum impact and minimal detectability.
• Real-world evaluations on cloud RDMA clusters demonstrated significant performance loss for critical applications like NCCL Tests, with degradation ranging from 18.23% to 56.12% for collective communication primitives.
• Current network monitoring tools and DoS defenses, which typically rely on link load or queue contention, are largely ineffective against LoRDMA due to its low average traffic volume and the indirect nature of its congestion spreading.
• There is an urgent need for the RDMA community to re-evaluate the security implications of traffic control mechanisms in lossless RDMA networks, focusing on more granular monitoring, PFC-aware anomaly detection, and robust fabric-level isolation.

About the Speaker(s)

Shicheng Wang is the speaker who presented this research at the NDSS Symposium. Based on the provided transcript and metadata, the talk details his work on identifying and demonstrating the LoRDMA attack in RDMA networks. Further biographical details about the speaker are not available in the provided materials.