ReqsMiner: Automated Discovery of CDN Forwarding Request Inconsistencies and DoS Attacks with Grammar-based Fuzzing

ReqsMiner: Automated Discovery of CDN Forwarding Request Inconsistencies and DoS Attacks with Grammar-based Fuzzing

Speakers: Linkai Zheng

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=MjekJfXVW1o

Overview

Content Delivery Networks (CDNs) are fundamental components of modern internet infrastructure, crucial for accelerating content delivery and providing robust security against various online threats. However, their role as man-in-the-middle proxies often leads to unintended modifications of client requests as they are forwarded to origin servers. These alterations, while sometimes aimed at performance optimization or security, can introduce subtle yet critical operational inconsistencies that can be exploited for severe security vulnerabilities, including HTTP Request Smuggling, Cache Poisoned Denial of Service (CPDoS), and various forms of Denial of Service (DoS) attacks. Prior research has identified several such issues, but a systematic, automated approach to uncover these inconsistencies broadly has been lacking.

This talk introduces REQSMINER, an innovative fuzzing framework designed to systematically and automatically discover previously unexamined inconsistencies in CDN forwarding requests. Developed by Linkai Zheng and his team, REQSMINER leverages advanced techniques inspired by reinforcement learning, specifically a novel UCT-based black-box fuzzing algorithm (UCT-Rand), and incorporates real-world field values into its grammar-based fuzzer. This allows REQSMINER to generate valid and effective test cases even with minimal feedback from black-box CDN services, significantly improving the efficiency and comprehensiveness of vulnerability discovery.

Through a comprehensive evaluation across 22 major CDN providers, REQSMINER successfully uncovered a wealth of unstudied CDN forwarding request inconsistencies. By integrating specialized analyzers, the framework further evolved to detect specific attack types. Most notably, the research identified three novel categories of HTTP amplification DoS attacks: HEAD Request-based HTTP Amplification (HeadAmp), Conditional Request-based HTTP Amplification (CondAmp), and Accept-Encoding-based HTTP Amplification (AEAmp). These attacks collectively expose 74 new potential DoS vulnerabilities across 19 CDN providers, demonstrating astonishing amplification factors that can reach up to 1,920,000 under specific conditions. The findings underscore a critical need for enhanced security measures within CDN ecosystems, and the team responsibly disclosed these vulnerabilities to affected vendors, contributing significantly to fortifying global CDN security.

Background

▶ Watch: Introduction to REQSMINER and problem motivation (0:00)

To fully grasp the significance of REQSMINER, it's essential to understand the foundational concepts underpinning CDN operations and modern security testing methodologies.

CDN Overview: A Content Delivery Network (CDN) is a geographically distributed network of proxy servers that work together to provide fast delivery of Internet content. Services like Cloudflare, Akamai, and Amazon CloudFront are integral to the internet's fabric, with over 64% of the top 10,000 websites utilizing CDN services. CDNs serve two primary functions: enhancing website performance by caching content closer to end-users and providing crucial security features like Web Application Firewalls (WAFs) and Distributed Denial of Service (DDoS) defense. Architecturally, a CDN typically comprises a central node for load balancing and content management, and numerous edge nodes (ingress and egress) responsible for caching and content distribution. The CDN acts as a man-in-the-middle, splitting the client-origin connection into two distinct stages. When a client requests content, the CDN first attempts to serve it from its cache. If the content is not found or is stale, the CDN forwards the request to the origin server, caches the response, and then delivers it to the client. This back-to-origin request mechanism is critical for both performance and DDoS defense.

HTTP Standards: The foundation of web communication is the Hypertext Transfer Protocol (HTTP). Specifically, HTTP/1.1 protocols, defined in RFC 9110, specify message syntax using Augmented Backus-Naur Form (ABNF) rules. ABNF rules are a formal grammar used to describe the syntax of programming languages and communication protocols. They consist of a rule name and a definition, employing operators for concatenation (space), selection (forward slash), and repetition (asterisk, leading digit, brackets). HTTP header fields, such as Host:example.com, are key-value pairs that are generally case-insensitive and registered in the HTTP Field Name Registry. Deviations from these standards by CDNs can introduce inconsistencies.

Grammar-based Fuzzing: Fuzzing is an automated software testing technique that involves injecting malformed or unexpected inputs into a program to expose software bugs and vulnerabilities. Grammar-based fuzzing is a more sophisticated variant that uses a predefined set of rules, or a grammar, to generate syntactically valid yet potentially semantically unusual inputs. This approach ensures that the generated test cases conform to the protocol's structure, allowing for more comprehensive testing and the detection of subtle vulnerabilities that might be missed by purely random fuzzing. It has proven successful in uncovering protocol security vulnerabilities in critical infrastructure systems.

Monte Carlo Tree Search (MCTS) and UCT: Monte Carlo Tree Search (MCTS) is a heuristic search algorithm that explores potential actions through simulations to identify the most promising ones. It operates in four stages: Selection, where the algorithm traverses the search tree to find a node to expand; Expansion, where new child nodes are added to the tree; Simulation, where a random playout (or simulation) is performed from the new node to estimate its value; and Backpropagation, where the results of the simulation are used to update the statistics of the nodes along the traversed path. Upper Confidence Bounds Applied to Trees (UCT) is a popular variant of MCTS that strategically balances exploration (trying new, unvisited paths) and exploitation (focusing on paths that have historically yielded good results). UCT uses a specific formula that combines a node's estimated mean value with an exploration term based on visit counts, ensuring the algorithm prioritizes less frequently visited nodes with higher discovery potential, thus allowing for effective state space search. This balancing act is crucial for efficient fuzzing in black-box environments.

Related Work: Prior research has identified various CDN forwarding inconsistencies and related attacks, including forwarding loop attacks, RangeAmp attacks, HTTP/2 header compression amplification, CPDoS, web cache deception, and HTTP desync attacks. Notable examples include "Host of Trouble" and "HTTP Request Smuggling." However, a significant limitation of these previous studies is their reliance on manual analysis, which is inherently time-consuming, prone to human error, and lacks the scalability required to address the increasing complexity and dynamic nature of CDN configurations. REQSMINER distinguishes itself by offering a systematic, automated, and extensible methodology to uncover these inconsistencies, thereby bridging a critical gap in existing vulnerability discovery techniques.

Key Findings

▶ Watch: REQSMINER's discovery of novel DoS attacks (1:00)

REQSMINER's comprehensive analysis yielded several significant findings and contributions that have a profound impact on CDN security:

• Automated Discovery of Inconsistencies: The framework successfully achieved its primary goal: the automated discovery of a vast array of previously unstudied CDN forwarding request inconsistencies across 22 major CDN providers. These inconsistencies encompass alterations in request lines, header fields, and even message bodies.
• Three Novel HTTP Amplification DoS Attacks: Building upon the discovered inconsistencies, REQSMINER identified and characterized three new, high-impact HTTP amplification Denial of Service attacks:
• HEAD Request-based HTTP Amplification (HeadAmp): Exploits CDNs that convert HEAD requests to GET requests, leading to the origin sending full resources while the CDN returns only headers to the attacker.
• Conditional Request-based HTTP Amplification (CondAmp): Targets CDNs that remove conditional headers (e.g., If-Match, If-None-Match), causing the origin to send full resources when it should respond with a small "not modified" status.
• Accept-Encoding-based HTTP Amplification (AEAmp): Leverages CDNs that discard Accept-Encoding headers, forcing the origin to send uncompressed content which the CDN then compresses for the attacker, leading to amplification based on compression ratios.
• Widespread Vulnerabilities with Massive Amplification Factors: The research identified a total of 74 new potential DoS vulnerabilities impacting 19 different CDN providers. These attacks demonstrated alarming amplification factors:
• HeadAmp and CondAmp attacks achieved factors up to 1,920,000 for 1GB files, while AEAmp reached factors close to 1,000 for 1MB zero-byte files.
• The average amplification factor was around 2,000.
• Efficient and Low-Cost DDoS Attacks: These newly discovered attacks are highly efficient, requiring minimal attacker resources. An attacker can use a standard laptop and leverage CDN egress nodes to launch devastating DoS attacks, bypassing common DDoS protections by manipulating the CDN's internal logic. Cache-busting strategies ensure these attacks impact the origin server directly, leading to significant resource consumption.
• Huge Financial and Reputational Impacts: Such attacks can trigger severe website outages, result in substantial financial losses for affected organizations due to increased bandwidth costs, damage business reputation, and necessitate investments in additional DDoS security services.
• Novel Fuzzing Techniques: REQSMINER introduces two key technical innovations in fuzzing:
• A Field Values-assisted Rule Fusioner that incorporates real-world HTTP field values from RFCs and traffic logs into the ABNF grammar, creating more realistic and effective test cases.
• The UCT-based Request Generator (UCT-Rand), which adapts Monte Carlo Tree Search for black-box fuzzing, balancing grammar exploration with successful request forwarding even with limited feedback.
• Responsible Disclosure and Industry Impact: All identified security issues were responsibly reported to the affected CDN vendors. Notably, Azure and Cloudflare acknowledged and rectified the vulnerabilities, with Cloudflare awarding a bounty. Fastly also acknowledged the report and is open to re-evaluation. The open-sourcing of REQSMINER on GitHub makes this powerful tool available to the security community for further research and defense.

Technical Deep Dive

▶ Watch: Foundational concepts: CDNs, HTTP, fuzzing, MCTS (2:00)

REQSMINER's effectiveness stems from its innovative design, which tackles the unique challenges of fuzzing black-box CDN services.

Threat Model and Challenges:

The core threat model assumes an attacker, posing as a legitimate client, can send carefully crafted requests to a vulnerable CDN. The CDN, due to its forwarding inconsistencies, modifies and relays these requests to the attacker's target: the origin server. The attacker's goal is to exploit these discrepancies to manipulate requests within the CDN-origin connection, executing various attacks such as DoS, CPDoS, Forwarding Loops (FL), and Web Cache Poisoning (WCP) against the victimized origin.

Fuzzing CDN forwarding behavior presents significant challenges:

1. Lax Grammar: HTTP's ABNF rules are often broad and unbounded. For instance, a request method can be "any token," but CDNs only support a subset. Generating random values from such a broad definition is highly inefficient, as most invalid test cases are simply rejected by the CDN, wasting resources.
2. High Costs: CDNs are commercial, black-box services. Traditional fuzzing requires a vast number of test cases, leading to substantial monetary costs for the fuzzer operator.
3. Black Box Nature: CDNs provide minimal feedback on test requests. This limited visibility makes it difficult for a fuzzer to accurately discern and effectively detect all request differences and their security implications.

REQSMINER's Solutions:

To overcome these obstacles, REQSMINER introduces two novel techniques:

1. Field Values-assisted Rule Fusioner: This component directly addresses the "lax grammar" problem. It systematically extracts valid field values from RFC documents and real-world HTTP traffic logs (e.g., from web server access logs or network security scanners like Xray). These extracted field values are then merged with the ABNF rules to construct a more constrained and realistic ABNF grammar tree. By incorporating expert domain knowledge and real-world usage patterns, this process effectively sets practical search bounds for the fuzzer, significantly enhancing the effectiveness and validity of the generated test cases.
2. UCT-based Request Generator (UCT-Rand): To tackle the high costs and limited feedback inherent in black-box CDN testing, REQSMINER draws inspiration from reinforcement learning, specifically the Upper Confidence Bounds Applied to Trees with Weighted Randomization (UCT-Rand) algorithm. This algorithm iteratively refines the quality of generated test cases based on the minimal feedback received. UCT-Rand meticulously balances exploration (discovering new paths within the grammar tree) with exploitation (focusing on promising paths that have led to successful request forwarding), making the fuzzing process both cost-effective and efficient, even in environments with scarce information.

System Design and Architecture:

REQSMINER's architecture is divided into two primary modules: the Rule Generator and Grammar-based Fuzzing.

1. Rule Generator Module: This module constructs an effective ABNF grammar tree for test case generation.

• ABNF Parser: This component takes ABNF rules, meticulously extracted from HTTP RFCs (e.g., RFC 9110-9112, 3986, 4647, 5234, 5646), and transforms them into an internal ABNF grammar tree. This tree visually represents the hierarchical structure of HTTP messages. The parser defines three types of nodes:
• AND nodes: Represent concatenation, meaning all child nodes must be visited recursively to form a valid part of the request.
• OR nodes: Indicate selection, where exactly one child node is chosen from a set of alternatives or limited repetitions.
• RAND nodes: Mark random repetitions, where the number of times a child node is visited is determined randomly. An example for the Accept-Charset header would illustrate how these nodes structure the grammar.
• Rule Fusioner (Field Values-assisted Rule Fusioner): This crucial component enhances the ABNF grammar tree with real-world intelligence. It parses and extracts concrete field values from RFCs and collected web server logs. For instance, from an Accept-Language header value like en-US,en;q=0.9,en-GB;q=0.8,zh;q=0.7,ja, the fusioner extracts specific Language-Ranges such as en-US, en, zh. These extracted non-terminal symbol values are then intelligently inserted as new leaf nodes into the ABNF grammar tree, specifically as children of existing OR nodes. This expansion significantly increases the number of realistic options available to the UCT-Rand algorithm, ensuring that generated test cases are not only syntactically correct but also semantically plausible and more likely to be accepted and processed by real-world CDNs.

2. Grammar-based Fuzzing Module: This module is responsible for generating test cases, orchestrating their delivery, and detecting inconsistencies.

• UCT-based Request Generator: This is the core of REQSMINER's fuzzing engine. It generates valid HTTP test cases by intelligently traversing the ABNF grammar tree. Inspired by MCTS, the proposed UCT-Rand algorithm (Algorithm 1 in the paper) employs weighted random selection instead of a deterministic argmax function to balance exploration and exploitation more effectively. It operates in four phases:
• Expansion: The generator starts from the root node of the ABNF grammar tree and builds a derivation tree, which directly corresponds to a specific HTTP request. AND nodes are traversed directly, while OR and RAND nodes initiate the selection phase.
• Selection: For RAND nodes, the number of traversals is randomly decided. For OR nodes, if any child nodes remain unvisited, a random unvisited one is selected. If all child nodes have been visited, the UCB formula (Equation 2: π(v) := weighted rand (Q(v, v') + 2 ln N(v) / N(v, v'))) is used. Here, Q(v,v') represents the estimated probability of successful forwarding for a child node v', N(v) is the parent node's visit count, and N(v,v') is the child node's visit count. This formula guides the selection towards nodes that are both promising (high Q) and underexplored (low N(v,v') relative to N(v)).
• Simulation: The generated HTTP requests are dispatched to the target CDN via the client component. The generator then retrieves the CDN's forwarding status from a controlled origin server, logging whether the request was successfully forwarded.
• Backpropagation: Based on the results of the simulation (successful forwarding or failure), the Q(v,v') (success probability) and N(v,v') (visit counts) parameters for each node in the ABNF grammar tree are updated, iteratively refining future selections and improving the fuzzer's efficiency.
• Courier Platform: This platform acts as the orchestration layer for the entire testing process. It comprises the client, which sends test requests; the server (a controlled origin server), which receives forwarded requests; and the targeted CDNs. The client dispatches test requests to the CDN, which then forwards them to the controlled origin server. Both the client and server log the original client request and the forwarded server request, respectively. These logs are then sent to the Difference Analyzer. Crucially, the origin server also provides real-time feedback to the Request Generator regarding the successful forwarding of requests, informing the UCT-Rand algorithm.
• Difference Analyzer: This module is responsible for detecting and categorizing alterations made by the CDN. It extracts simplified request structures from both the original client request (from client logs) and the forwarded server request (from server logs) using a simplified set of ABNF rules. By meticulously comparing these structures, it identifies various types of modifications: Alteration (a header value was modified), Insertion (a new header was added by the CDN), Deletion (an original header was removed), Duplicate Header Insertion (an existing header was re-added), and Duplicate Header Deletion (an existing header was removed despite other duplicates). These detected modifications are then archived for further in-depth analysis and vulnerability assessment.

Implementation Details:

REQSMINER's experimental setup was robust and ethical.

• Target CDNs: The study analyzed 22 prominent CDN providers, including industry leaders like Cloudflare, Akamai, CloudFront, Aliyun, Azure, and Google Cloud, selected based on market share.
• Platform: Experiments were conducted on two Linux servers (Ubuntu 20.04.2 LTS, GNU/Linux 5.4.0-125-generic x86_64), each equipped with a 2.40GHz 32-core Intel Xeon E5-2640 v4 CPU, 16GB RAM, and 1000Mbps bandwidth. One server acted as the client, and the other as the controlled origin server.
• Test Workflow: The process began with a thorough analysis of HTTP/1.1 specifications (RFC 9110-9112) and supplementary RFCs, yielding 442 ABNF rules. Real-world field values (63 sets) were collected from a lab homepage's access logs and HTTP logs from the Xray network security scanner. The UCT-based Request Generator then generated 100,000 HTTP test requests (100 rounds, 1,000 cases/round) for each CDN. Each request embedded a unique Universal Unique Identifier (UUID) in its request-target URL to prevent CDN cache hits and facilitate precise logging by the origin server. Requests were sent at a low frequency (6 requests per second) using low-level network programming (raw sockets) to ensure precise packet parsing and minimize impact.
• Experiment Cost: The total cost was less than $10, primarily utilizing free tiers or trial services. Paid services for specific CDNs were subscribed to.
• Ethical Considerations: Strict ethical guidelines were followed. All CDN services were purchased, requests were sent at a low frequency to avoid operational impact, and all origin servers were under the researchers' control. No attacks were launched against real users or external servers. All tested CDN vendors were actively informed, and coordinated disclosure best practices were adhered to for all identified vulnerabilities.

Evaluation of Fuzzer Performance:

REQSMINER's efficiency was evaluated based on effectiveness (proportion of successfully forwarded requests) and exploration (number of traversed ABNF grammar tree nodes). A comparison was made between Random, UCT, and UCT-Rand algorithms:

• Random: Showed less than 20% effectiveness, indicating its inability to generate valid requests for CDNs.
• UCT: Achieved over 90% effectiveness from the second round but its exploration rate significantly slowed after 50 rounds. This suggested UCT's tendency towards local optimization and overfitting, failing to adequately explore the entire grammar tree.
• UCT-Rand: While starting with lower effectiveness (<10%), it gradually rose to over 60% by the final round. Crucially, UCT-Rand consistently explored more nodes than both Random and UCT after 50 rounds (Figure 6 in the paper). This demonstrated UCT-Rand's superior ability to balance successful forwarding with broad ABNF exploration, effectively escaping local optima and achieving more comprehensive coverage. Consequently, UCT-Rand was selected as the generative algorithm for all experiments.

Demo / Proof of Concept

▶ Watch: REQSMINER's threat model and attack surface (4:00)

The core "proof of concept" for REQSMINER's capabilities lies in its ability to not only detect forwarding inconsistencies but also to leverage specialized analyzers to identify specific attack patterns, particularly novel HTTP amplification DoS attacks. The talk detailed three such attacks, alongside a broader categorization of inconsistencies found.

HTTP Amplification Attacks Overview:

The threat model for amplification attacks (Figure 7 in the paper) involves an attacker sending a legal but crafted request to a vulnerable CDN (step 1). The CDN is then manipulated to fetch a large amount of content from the origin server (step 2), which returns the large content (step 3). Crucially, the CDN returns only a small response to the attacker (step 4). The primary goal is to maximize the traffic between the CDN and the origin server while minimizing the traffic between the attacker and the CDN. Attackers must employ cache-busting strategies (e.g., using unique UUIDs in URLs) to ensure the CDN fetches content from the origin rather than serving from its cache. While CDNs have implemented mitigations against simple connection termination attacks, REQSMINER's research discovered novel attacks that bypass these. The framework's analysis module records client and server traffic sizes, flagging potential amplification vectors if the server's traffic is more than ten times that of the client, followed by controlled experiments to calculate the precise amplification factor (CDN-origin traffic / client-CDN traffic).

Three Novel HTTP Amplification Attacks:

1. HEAD Request-based HTTP Amplification Attack (HeadAmp):

• Attack Details (Figure 8): This attack exploits CDNs that incorrectly convert HEAD requests into GET requests when forwarding to the origin.

1. The attacker sends a small HEAD request to the CDN.
2. The CDN, due to its inconsistency, converts this to a GET request and forwards it to the origin.
3. The origin server, receiving a GET, responds with the full resource body (potentially very large).
4. The CDN, still complying with the original HEAD request's semantics, returns only the response line and headers (a small response) to the attacker.

• This creates a massive traffic disparity: small client-CDN traffic vs. large CDN-origin traffic. The target resource must be cacheable to ensure the CDN processes it.
• Real-World Analysis: 12 CDNs were found vulnerable to HeadAmp.
• Results (Table I, Figure 9): Amplification factors varied significantly with resource size. For a 1MB file, factors ranged from 23.79 (CDN77) to 1725.39 (Gcore). For a 1GB file, the maximum amplification factor reached an astonishing 1,920,000 (observed in CDN77, CDNetworks, ChinaNetCenter, Gcore, StackPath). Cloudflare and BunnyCDN showed slightly lower factors due to adding more headers, increasing the client-CDN traffic. Some CDNs had partial mitigations: Aliyun, Azure, CDN77, and KeyCDN stopped requests after receiving headers; Cloudflare terminated requests over 512MB; Fastly returned a 503 Service Unavailable for resources over 20MB; Udomain added a Range header and didn't request the full file.

1. Conditional Request-based HTTP Amplification Attack (CondAmp):

• Attack Details (Figure 10): This attack targets CDNs that inappropriately remove or ignore conditional headers (e.g., If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since, If-Range).

1. The attacker crafts a conditional request, typically requesting a resource it doesn't have or that has been modified, along with a cache-busting UUID.
2. The CDN removes the conditional headers and forwards the request as a standard GET to the origin.
3. The origin server, receiving a non-conditional GET, returns the complete resource.
4. The attacker then manipulates the conditional header (e.g., by sending a false ETag) to prevent the CDN from returning the full resource, resulting in a small response to the attacker. The resource must be uncached but cacheable for the attack to work.

• Real-World Analysis: 16 CDNs were found vulnerable to CondAmp.
• Results (Table II): For a 1MB file, amplification factors ranged from 24.59 (CDNetworks) to 1955.17 (CDNSun). For a 1GB file, maximum amplification factors reached up to 1,927,288 (CDNSun).

1. Accept-Encoding-based HTTP Amplification Attack (AEAmp):

• Attack Details (Figure 11): This attack exploits CDNs that discard or ignore the Accept-Encoding header.

1. The attacker sends a request with Accept-Encoding: gzip (indicating it prefers compressed content).
2. The CDN removes this header and forwards the request to the origin.
3. The origin, unaware of the client's preference for compression, returns the uncompressed resource.
4. The CDN, based on its own configuration or the (now absent) Accept-Encoding header, compresses the resource before returning it to the attacker.

• The amplification factor is higher for resources with greater compression rates. Attackers can maximize this by uploading files with highly compressible content (e.g., all zero bytes).
• Real-World Analysis: 4 CDNs were found vulnerable to AEAmp.
• Results (Table III, Figure 12): For a 1MB zero-byte file, amplification factors ranged from 202.03 (Udomain) to 650.43 (CDNSun). For larger files, CDN77 and CDNSun showed factors exceeding 960. Udomain exhibited lower factors (230) due to relatively low compression. Baidu Cloud's attack failed for files larger than 10MB because it returned the uncompressed version to the client, preventing amplification.

Broader Inconsistency Findings:

Beyond these amplification attacks, REQSMINER uncovered a multitude of other inconsistencies categorized as follows:

• Differences in Request Line:
• Request Method: 11 CDNs changed HEAD requests to GET. Aliyun and Qiniu Cloud also changed special methods like LOCK or MERGE to GET.
• Request URL Target: 6 CDNs discarded the part after the hash (#). Akamai combined adjacent slashes (//) into one. Azure and BunnyCDN parsed and removed /./ and /dir/../ segments.
• HTTP Version: Most CDNs modified the HTTP version to 1.1 when forwarding. Verizon changed 0.9/1.0 to 1.0, and StackPath changed 0.0-0.9 to 0.9. This can lead to CPDoS if the origin doesn't support the modified version and the CDN caches the error.
• Differences in Header Fields:
• Duplicate Headers: Most CDNs forwarded all duplicates, but some merged values (e.g., BunnyCDN merged with semicolons instead of commas, violating RFC). Google Cloud combined multiple Host headers with commas. Most discarded all but the first Host header (6 rejected with 400/500). For Content-Length, most rejected with 400, but Aliyun and Fastly accepted (keeping the first), and Tencent Cloud accepted (keeping the last).
• Adding Headers: CDNs often append client-related headers (e.g., Cloudflare adds CF-Connecting-IP, CF-IPCountry, CF-RAY, CF-Visitor). To prevent forwarding loops, CDNs should add CDN-Loop or Via headers; however, only 5 CDNs added CDN-Loop, and 5 others added neither, leaving most CDNs vulnerable to forwarding loop attacks.
• Removing Headers: While TE and Upgrade were removed as per RFC, 12 CDNs removed Range headers (leading to RangeAmp attacks), and 17 removed conditional headers (leading to DoS attacks).
• Altering Headers: CDNs altered Accept-Language (Azure, BunnyCDN, Verizon) and Accept-Encoding (15 CDNs).
• Differences in Message Body:
• Removing Message Body: Akamai and Azure removed message bodies from GET/HEAD requests (fat requests). If the origin rejects fat requests and the CDN caches the error, CPDoS attacks are possible.
• Transfer Encoding: CDNs like Aliyun, CDN77, Cloudflare, and Gcore sent chunked transfer data as a single piece, altering the Transfer-Encoding. This can defend against Slowloris-like attacks but also enables pulse-based DoS attacks.

These detailed findings provide concrete evidence of the pervasive nature of CDN forwarding inconsistencies and their direct applicability to novel and impactful DoS attacks.

Defensive Implications

▶ Watch: Specific challenges in fuzzing CDN black-box services (4:22)

The findings from REQSMINER highlight critical areas where both CDN providers and website owners (origin servers) must strengthen their defenses.

For CDN Providers:

1. Strict RFC Adherence and Request Validation: CDNs must rigorously adhere to HTTP/1.1 RFC specifications (e.g., RFC 9110-9112) for request parsing and forwarding. This includes consistent handling of request methods, URL targets, HTTP versions, and especially header fields. Implement robust validation and sanitization of all incoming requests before forwarding them to the origin.
2. Consistent HEAD vs. GET Handling: CDNs must ensure that HEAD requests are not implicitly converted to GET requests for the origin, or if they are, that the origin's response body is explicitly discarded before it consumes significant CDN-origin bandwidth. Alternatively, implement explicit blocks or rate limits for HEAD requests targeting large resources.
3. Conditional Header Integrity: Avoid removing or altering critical conditional headers such as If-Match, If-None-Match, If-Modified-Since, and If-Range unless there's a specific, secure, and documented reason. If removal is necessary, ensure that the CDN itself handles the conditional logic appropriately to prevent full resource fetches from the origin.
4. Accept-Encoding Header Processing: CDNs should honor the Accept-Encoding header from the client. If the header is removed, the CDN should not then compress the origin's uncompressed response for the client, as this creates an amplification vector. Ensure consistent compression policies throughout the request-response chain.
5. Forwarding Loop Prevention: Implement and correctly configure CDN-Loop or Via headers to prevent forwarding loop attacks. The observation that many CDNs lack these headers is a significant security oversight.
6. Duplicate Header Management: Standardize the handling of duplicate headers according to RFCs (e.g., combining comma-separated values for certain headers, rejecting ambiguous ones like Host or Content-Length). Avoid non-standard merging behaviors (like BunnyCDN's semicolons).
7. Traffic Amplification Monitoring and Mitigation: Implement advanced monitoring for unusual traffic patterns and amplification ratios between CDN edge nodes and origin servers. Deploy real-time anomaly detection for spikes in origin-bound traffic that do not correlate with client-facing traffic.
8. Request Size and Rate Limiting: Enforce strict rate limiting and request size limits at the edge for individual client connections to mitigate the impact of even small, amplified requests. Implement specific limits for the size of resources fetched from the origin in response to specific request types (e.g., HEAD requests, conditional requests).
9. Vulnerability Rectification: Proactively address the specific vulnerabilities identified in this research. Cloudflare and Azure's swift action serves as a positive example.
10. Transparency and Documentation: Clearly document how CDNs handle various HTTP requests, headers, and edge cases, especially regarding modifications and forwarding logic.

For Website Owners / Origin Servers:

1. Understand CDN Behavior: Do not assume that CDNs forward client requests verbatim. Recognize that CDNs act as proxies and may alter requests. Understand the specific modifications your chosen CDN applies.
2. Robust Origin-Side DDoS Protection: Implement strong DDoS mitigation strategies directly on the origin server. While CDNs offer protection, vulnerabilities like those found by REQSMINER can bypass them, shifting the attack burden to the origin.
3. Monitor Origin Traffic: Continuously monitor origin server traffic for unusual spikes, especially from known CDN IP ranges or egress nodes. Look for discrepancies between expected client traffic volume and actual origin server load.
4. Validate Incoming Requests: Even if requests come from a trusted CDN, implement robust validation on the origin server. Do not blindly trust that all headers or request parameters are pristine. For example, if your application expects a compressed response, ensure the Accept-Encoding header is present and valid.
5. Handle HTTP Versions and Duplicates Gracefully: Ensure origin servers are configured to gracefully handle various HTTP versions and correctly process or reject duplicate headers according to standard specifications.
6. Cache Poisoning Awareness: Be aware of how CDN forwarding inconsistencies could lead to Web Cache Poisoning (WCP) or Cache Poisoned Denial of Service (CPDoS). If a CDN caches an error response due to an altered request, it could serve that error to legitimate users. Implement cache-control headers carefully.
7. Collaborate with CDN Providers: Maintain open communication with your CDN provider. Inquire about their security posture, how they handle forwarding inconsistencies, and their plans for addressing newly disclosed vulnerabilities.

By adopting these defensive postures, both CDN providers and their customers can collectively enhance the security posture of the internet against sophisticated attacks leveraging CDN forwarding inconsistencies.

Key Takeaways

• CDNs, despite their security benefits, frequently alter client requests as they forward them to origin servers, leading to widespread and often unstudied operational inconsistencies that can be exploited for serious security vulnerabilities.
• REQSMINER is a novel, automated fuzzing framework that systematically uncovers these CDN forwarding request inconsistencies using innovative techniques, including a Field Values-assisted Rule Fusioner for realistic test cases and a UCT-based Request Generator (UCT-Rand) for efficient black-box exploration.
• The research identified 74 new potential DoS vulnerabilities across 19 CDN providers, including three novel HTTP amplification attacks: HeadAmp, CondAmp, and AEAmp, which can achieve unprecedented amplification factors up to 1,920,000.
• These amplification attacks are highly efficient and low-cost, requiring minimal attacker resources to generate massive traffic loads on origin servers, bypassing conventional DDoS protections.
• Proactive detection, strict adherence to HTTP standards, and rigorous validation of requests by CDN providers are crucial to mitigate these risks. Origin server owners must also implement robust monitoring and validation, recognizing that CDNs are not transparent proxies.
• Responsible disclosure efforts led to acknowledgments and rectifications from major CDN vendors like Azure and Cloudflare, demonstrating the immediate positive impact of this research on real-world security.

About the Speaker(s)

The talk was presented by Linkai Zheng, a researcher involved in the development and presentation of REQSMINER. His work focuses on systematically exploring and addressing critical security vulnerabilities within Content Delivery Network (CDN) architectures, particularly concerning request forwarding inconsistencies and their exploitation for Denial of Service attacks.