Why So QUIC! Racing and Fuzzing HTTP/3 with QuicDraw UI

Why So QUIC! Racing and Fuzzing HTTP/3 with QuicDraw UI

Speakers: Maor Abutbul, Vulnerability Researcher, CyberArk Labs

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=irc91zTMCg8

Overview

Maor Abutbul's Nullcon talk, "Why So QUIC! Racing and Fuzzing HTTP/3 with QuicDraw UI," delves into the often-overlooked security implications of HTTP/3, the latest iteration of the Hypertext Transfer Protocol. Despite its widespread adoption—powering over 36% of internet-facing websites, including major services like Google Search and YouTube—HTTP/3 remains an under-researched area within the security community, particularly concerning practical attack vectors and tooling. This presentation addresses that gap by exploring race conditions in the context of HTTP/3 and introducing QuickDraw, an open-source tool designed for fuzzing and racing HTTP/3 web applications.

The talk highlights a critical challenge: the current lack of robust security tools and interception proxies that fully support HTTP/3. This absence creates a blind spot for security researchers and practitioners, making it difficult to analyze, test, and secure applications leveraging this modern protocol. Abutbul's research aims to shed light on potential vulnerabilities arising from HTTP/3's underlying transport protocol, QUIC, and provide a practical methodology and toolset for identifying them.

This article dissects Abutbul's journey, from understanding HTTP/3's core features and its differences from HTTP/2, to identifying a novel race condition attack vector—dubbed Quick Fin Sync—and demonstrating its efficacy against a real-world application. The insights and tools presented are crucial for developers building HTTP/3-enabled services and security professionals tasked with defending them, emphasizing the need for a proactive approach to securing next-generation web protocols.

Background

▶ Watch: Introduction and HTTP/3 adoption statistics (0:00)

HTTP/3 represents a significant evolution in web communication, primarily distinguished by its reliance on QUIC (Quick UDP Internet Connections) as its transport layer, rather than TCP. To understand the impetus behind HTTP/3, it's essential to briefly recap HTTP/2 and its limitations.

HTTP/2 brought several improvements over HTTP/1.1, including being always encrypted, binary, and supporting header compression and server push. Crucially, it introduced multiplexing, allowing multiple requests and responses to be sent over a single TCP connection concurrently, thereby eliminating HTTP head-of-line blocking. However, HTTP/2 still suffered from TCP head-of-line blocking. From a network perspective, if a single packet is lost within a TCP connection, all subsequent packets on that same connection are buffered until the lost packet is retransmitted and acknowledged. This delay affects all active HTTP streams on that TCP connection, even if they are logically independent.

HTTP/3 addresses this fundamental limitation by building on QUIC, which is often referred to as "TCP 2.0" or the "next version of TCP." QUIC operates over UDP, offering several key advantages:

• Always Encrypted and Binary: Similar to HTTP/2, but encryption is inherent to QUIC itself, not just HTTP/3.
• Header Compression: Uses QPACK for efficient header compression.
• Connection Migration: A significant feature allowing a client to seamlessly switch network interfaces (e.g., from Wi-Fi to mobile data) without dropping the active QUIC connection, as the connection is identified by a Connection ID rather than the source IP/port tuple.
• Fast Connection Setup and Zero-RTT (0-Round Trip Time): For subsequent connections, QUIC can often establish a secure connection with zero round trips, significantly reducing latency.
• Multiplexing without Head-of-Line Blocking: This is QUIC's crowning achievement. Because QUIC streams are independent at the transport layer, a lost packet within one stream does not block the delivery of packets for other streams. If a packet for one stream is delayed, other streams can continue processing, effectively eliminating the TCP head-of-line blocking issue.

For practical HTTP/3 research, Abutbul highlighted several tools and tips. Identifying HTTP/3 support on a server can be done by looking for the Alt-Svc header in the HTTP response, which indicates alternative service protocols, including h3. Modern browsers will automatically prefer HTTP/3 when available. For inspection, standard browsers are useful, but local proxies typically do not support HTTP/3. cURL can be compiled or downloaded with HTTP/3 support and used with the --http3-only flag. To decrypt traffic for inspection, an SSL key log file can be used with tools like Wireshark, which offers specific filters for quic and http3 traffic. These tools form the foundational toolkit for anyone looking to analyze HTTP/3 traffic.

Key Findings

▶ Watch: HTTP/2's TCP head-of-line blocking problem (4:00)

The initial phase of the research involved identifying potential security leads within HTTP/3's feature set. Abutbul and his team explored several attack scenarios, including amplification attacks, slow attacks, and user tracking. However, after thorough investigation, they found that the first two were already addressed by the HTTP/3 and QUIC RFCs, rendering them largely irrelevant as novel research leads. User tracking, while a valid concern, had already been extensively researched by others. This left the team at a crossroads, with all initial leads seemingly exhausted.

A pivotal moment came from a prior research project involving Keycloak, an open-source identity provider. In that research, the team discovered a race condition vulnerability. This experience sparked a new line of inquiry: Is it possible to make race conditions work in HTTP/3? The unique characteristics of HTTP/3 and QUIC, particularly its multiplexing capabilities and UDP transport, presented both challenges and opportunities for exploiting timing-sensitive vulnerabilities.

The core challenge was to find a mechanism to send or release multiple HTTP/3 requests in bulk, ensuring they hit the server's application logic simultaneously, thereby increasing the race window. This led to the development of the Quick Fin Sync concept, which became the central technical finding of the research. Unlike HTTP/2 where TCP's head-of-line blocking could be manipulated, HTTP/3's stream independence meant a new approach was needed. The Quick Fin Sync leverages QUIC's stream multiplexing by carefully controlling the transmission of the FIN flag (indicating the end of a stream) for multiple requests. By holding back the FIN flag for several requests and then releasing them all in a single UDP packet, the researchers hypothesized they could synchronize their arrival and processing on the server, triggering application-layer race conditions.

This novel approach, coupled with the observation that no popular HTTP/3 security tools or interception proxies existed, underscored the necessity of building a custom tool. This led to the creation of QuickDraw, an open-source utility designed specifically for fuzzing and racing HTTP/3 applications, filling a critical gap in the security testing landscape.

Technical Deep Dive

▶ Watch: HTTP/3 over QUIC: features and no HOL blocking (5:30)

The fundamental challenge in exploiting race conditions in HTTP/3 lies in its design to prevent head-of-line blocking at the transport layer. Traditional race condition attacks often leverage network-level delays or packet manipulation to synchronize requests. Abutbul's team initially considered IP fragmentation as a potential avenue, given that QUIC runs over UDP, which in turn runs over IP. The idea was to fragment UDP datagrams containing HTTP/3 requests, delaying their reassembly and processing on the server. However, this approach was quickly invalidated by the QUIC RFC, which explicitly states that UDP datagrams must not be fragmented at the IP layer. Furthermore, even if fragmentation were possible, each HTTP/3 request operates on its own independent stream within QUIC, meaning fragmenting one stream wouldn't necessarily interfere with others.

The breakthrough came from a deeper understanding of QUIC's multiplexing. While each request has its own stream, multiple streams (and thus multiple requests) can be sent over the same QUIC connection ID within a single UDP datagram. This crucial insight led to the Quick Fin Sync concept. The strategy is as follows:

1. Prepare Multiple Requests: Construct multiple HTTP/3 requests targeting the same vulnerable endpoint.
2. Delay FIN Flag: For each request, send most of its data but intentionally withhold the FIN flag. The FIN flag in QUIC signifies the end of a stream. Without it, the server receives the request data but typically won't process it at the application layer until the stream is complete.
3. Synchronized Release: Once all desired requests have their data sent (minus the FIN flag), a final UDP packet is constructed containing the FIN flags for all these pending requests. This packet is then sent.

Upon receiving this "sync" packet, the server's QUIC layer processes the FIN flags for all the requests simultaneously. This effectively "releases" all the requests to the application layer at nearly the same instant, creating a tight race window where the application might process them concurrently. This technique is inspired by "single packet" or "less byte sync" attacks seen in other protocols, adapted for QUIC's unique stream management.

To implement and test this concept, the team developed QuickDraw. QuickDraw is an open-source tool written in Python, leveraging the AIOQUIC library for QUIC protocol implementation. Its core functionality for racing involves:

• --total-requests (-tr) parameter: Specifies how many parallel requests to send.
• Single Packet Transmission: QuickDraw is designed to pack multiple HTTP/3 requests into a single UDP datagram when possible, ensuring they arrive at the server almost simultaneously.
• FIN Flag Control: Crucially, it manages the timing of sending the FIN flag for these requests, either immediately or with a configurable delay using the -sd (sync delay) parameter. This parameter controls the delay between sending the bulk of the requests and the final packet containing the FIN flags, allowing researchers to fine-tune the race window for different network conditions and server processing times.

For fuzzing, QuickDraw incorporates a mechanism similar to other web fuzzers:

• fuzz keyword: Users can place a fuzz keyword within the request body, headers, or URL.
• Wordlist integration: QuickDraw then iterates through a provided wordlist, replacing the fuzz keyword with each entry for subsequent requests. When combined with the racing capabilities, this allows for race-condition fuzzing, attempting to trigger race conditions with various input values.

The absence of industry-standard HTTP/3 interception proxies (like Burp Suite or OWASP ZAP) or dedicated security tools made QuickDraw a necessity. Existing tools like cURL require custom builds and lack the sophisticated synchronization and fuzzing capabilities needed for advanced security research. QuickDraw, therefore, fills a critical void, enabling researchers to interact with and test HTTP/3 applications in ways previously impossible.

Demo / Proof of Concept

▶ Watch: Introducing QuickDraw for HTTP/3 fuzzing and racing (10:50)

The practical demonstration of QuickDraw's capabilities focused on exploiting a previously identified race condition in Keycloak, an open-source identity and access management system. The test setup involved:

• Vulnerable Application: A Keycloak instance running on an AWS EC2 machine.
• HTTP/3 Frontend: AWS CloudFront was used as the frontend server, providing a real-world HTTP/3 endpoint. CloudFront was chosen for its ease of setup and widespread adoption, ensuring the test was against a production-grade HTTP/3 server.

The vulnerability in Keycloak involved an API for creating client applications. An administrator could create an API token and set a limit on the number of clients that could be created using that token (e.g., a limit of two). When a developer used this API token to create a client, the token's remaining count would be decremented. The race condition manifested when multiple parallel requests to create clients were made using the same token. If processed concurrently, the application's logic failed to correctly decrement the count, leading to an unexpected state where more clients than the limit could be created, and the remaining count would display a negative number in the UI, clearly indicating a race condition exploit.

The demo proceeded in several stages using QuickDraw UI, a graphical interface for QuickDraw:

1. Baseline Single Request:

• An initial request using an expired API token resulted in a 401 Unauthorized error.
• A new API token with a limit was generated. A single request with this new token successfully created a client (201 Created), and the token's remaining count became zero.
• A subsequent request with the same token again resulted in 401 Unauthorized, confirming the token was fully used.

1. Fuzzing with Default QuickDraw Settings:

• A new API token was created.
• QuickDraw UI was configured to send 120 requests using the fuzzing mechanism, replacing a placeholder with sequential numbers.
• The initial run, using QuickDraw's default timing, resulted in only two concurrent executions being registered by Keycloak. This indicated that while multiple requests were sent, the default timing wasn't optimal for triggering a high number of race conditions on this specific target and network (the speaker was using a mobile hotspot, introducing latency).

1. Optimized Racing with sync-delay (-sd):

• Recognizing the need to fine-tune the race window, the speaker utilized QuickDraw's advanced -sd (sync-delay) flag. This flag controls the delay (in milliseconds) between sending the initial request data and the final "sync" packet containing the FIN flags.
• Attempt 1 (-sd 13 seconds): With a new API token and the sync-delay set to 13 seconds, QuickDraw sent 120 requests. This attempt significantly improved the outcome, resulting in 30 concurrent executions being registered by Keycloak, demonstrating the effectiveness of the Quick Fin Sync concept when timed correctly.
• Attempt 2 (-sd 14 seconds): Further adjusting the sync-delay to 14 seconds with a fresh API token yielded an even more impressive result: 99 concurrent executions out of 120 requests. This high success rate unequivocally proved that race conditions can be effectively triggered and exploited in HTTP/3 applications using the Quick Fin Sync method and QuickDraw. The Keycloak UI would show a negative remaining count, confirming the successful exploitation.

The demo clearly showcased QuickDraw's ability to precisely control HTTP/3 request timing, overcome the protocol's inherent head-of-line blocking elimination, and exploit application-layer race conditions against real-world HTTP/3 deployments.

Defensive Implications

▶ Watch: Identifying HTTP/3 security research leads and attack scenarios (12:00)

The research presented by Maor Abutbul highlights several critical defensive implications for organizations deploying and developing with HTTP/3:

1. Assume Race Conditions Still Exist: Despite HTTP/3's advancements in eliminating transport-layer head-of-line blocking, this research demonstrates that application-layer race conditions are still very much a threat. Developers should not assume that the shift to HTTP/3 inherently mitigates these types of vulnerabilities. All critical operations, especially those involving resource limits, financial transactions, or state changes, must implement robust server-side synchronization mechanisms (e.g., locks, atomic operations, transaction management) to prevent race conditions.

1. Validate Input and State Carefully: As seen in the Keycloak example, the application failed to correctly manage the remaining count when multiple requests arrived simultaneously. This underscores the importance of rigorous input validation and state management at the application layer, ensuring that even if multiple requests are processed concurrently, the final state remains consistent and secure.

1. Invest in HTTP/3 Security Tooling: The glaring absence of popular HTTP/3 security tools and interception proxies creates a significant blind spot. Security teams need to advocate for and contribute to the development of such tools. In the interim, tools like QuickDraw become indispensable for security testing. Organizations should integrate QuickDraw into their security assessment workflows for HTTP/3-enabled applications.

1. Perform Comprehensive Load and Concurrency Testing: Traditional load testing might not fully uncover race conditions. Security teams should perform specialized concurrency testing and race condition analysis, specifically designed to simulate simultaneous requests, leveraging tools like QuickDraw. This is particularly important for APIs and endpoints that handle sensitive operations.

1. Stay Updated on QUIC and HTTP/3 Security Research: As relatively new protocols, QUIC and HTTP/3 are still evolving, and their independent implementations may harbor protocol-level vulnerabilities. Organizations should monitor the latest security research, RFC updates, and vendor patches related to these protocols.

1. Review Third-Party HTTP/3 Implementations: Many organizations use CDNs (like AWS CloudFront) or third-party libraries that handle HTTP/3. While these abstract away much of the complexity, it's crucial to understand their security posture and how they might interact with application-layer logic under high concurrency. Regular security audits and penetration tests should include these components.

1. Educate Developers: Developers should be educated on the nuances of HTTP/3, QUIC, and the persistent threat of race conditions. Training should emphasize secure coding practices for concurrent environments, regardless of the underlying transport protocol.

By proactively addressing these defensive implications, organizations can better secure their HTTP/3 deployments against sophisticated timing-based attacks.

Key Takeaways

• Widespread HTTP/3 Adoption: HTTP/3, leveraging QUIC, is widely adopted across the internet (over 36% of websites), offering benefits like connection migration and elimination of TCP head-of-line blocking.
• Tooling Gap in Security: There is a significant lack of security tools, including interception proxies, that fully support HTTP/3, creating a blind spot for security research and testing.
• QuickDraw Fills the Gap: QuickDraw is an open-source Python tool designed for fuzzing and racing HTTP/3 web applications, providing much-needed capabilities for security professionals.
• Quick Fin Sync Attack: HTTP/3 applications can be vulnerable to race conditions through the "Quick Fin Sync" attack, which leverages QUIC's multiplexing to synchronize the release of multiple requests by controlling the FIN flag.
• Demonstrated Efficacy: The Quick Fin Sync attack, implemented with QuickDraw, successfully exploited a race condition in Keycloak, achieving up to 99 concurrent executions out of 120 requests against a real-world HTTP/3 server (AWS CloudFront).
• Broader Applicability: The concept of "less byte sync-like attacks" may apply to other multiplexed protocols, warranting further research in this area.

About the Speaker(s)

Maor Abutbul is a Vulnerability Researcher at CyberArk Labs, which is part of Palo Alto Networks. With approximately nine years of experience transitioning from engineering to research, he focuses on discovering security vulnerabilities in areas such as open source software, artificial intelligence (AI) systems, low-level components, and identity-related systems. Abutbul is also a father, an engineer, and a gamer. His team at CyberArk Labs contributes internally to enhance product security and externally to give back to the broader security community.