Large Language Model guided Protocol Fuzzing

Large Language Model guided Protocol Fuzzing

Speakers: Ruijie Meng

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=31JPvlrXfQ0

Overview

This article delves into "Large Language Model guided Protocol Fuzzing," a groundbreaking approach presented at the NDSS Symposium. The talk, delivered by Ruijie Meng, addresses a critical and persistent challenge in cybersecurity: the vulnerability of network protocol implementations. These protocols, fundamental to internet communication, are often informally specified in natural language documents like RFCs, making it exceptionally difficult to automatically generate valid, diverse test inputs for fuzzing. Traditional mutation-based fuzzers, while prevalent, struggle with limited seed diversity, unknown message structures, and premature coverage plateaus, leaving deep-seated vulnerabilities undiscovered.

The presented solution, LLM-guided protocol fuzzing, leverages the immense knowledge contained within pre-trained Large Language Models (LLMs). By systematically interrogating LLMs, the approach extracts machine-readable information to guide the fuzzing process in three key ways: extracting grammars for structure-aware mutation, increasing the diversity of initial seed message sequences, and generating messages to break out of coverage plateaus. The developed tool, CHATAFL, implements this methodology, demonstrating significant advancements over state-of-the-art fuzzers by substantially increasing state and code coverage, and, most notably, discovering nine distinct and previously unknown zero-day vulnerabilities in widely-used protocol implementations.

The significance of this work cannot be overstated. Protocol implementations represent a vast and critical attack surface; a single arbitrary code execution flaw can compromise even the most robust systems. By automating the generation of sophisticated, context-aware test cases, LLM-guided fuzzing promises to revolutionize how security flaws are found in these complex, informally specified systems, ultimately making internet infrastructure more secure.

Background

▶ Watch: Introduction: The challenge of protocol fuzzing (0:00)

Network protocol implementations are inherently stateful systems, meaning their behavior and output depend not only on the current input but also on the sequence of previous inputs. Discovering vulnerabilities in such systems often necessitates sending a precise sequence of messages in the correct order to reach a specific, vulnerable state. For instance, a handshake or initialization message might be a prerequisite before other message types can be processed. Without prior knowledge of valid message structures or the required sequence, fuzzing these systems effectively becomes a formidable task.

Historically, protocol fuzzers fall into two main categories:

1. Generator-based fuzzers: These tools generate messages from scratch, requiring a machine-readable specification of the protocol. However, manually creating these specifications is a notoriously tedious, error-prone, and time-consuming process, often resulting in incomplete coverage of the full protocol specification.
2. Mutation-based fuzzers: Approaches like AFLNET and NSFUZZ start with pre-recorded message sequences as initial "seeds." They then apply simple mutations to these seeds, aiming to preserve protocol validity while corrupting messages enough to trigger errors. This method has become prevalent due to its relative simplicity compared to manual specification.

Despite their popularity, state-of-the-art mutation-based fuzzers face several critical challenges, which the talk identifies as C1, C2, and C3:

• C1: Dependence on initial seeds. The effectiveness of these fuzzers is heavily limited by the quality and diversity of the provided initial seed inputs. Pre-recorded sequences, often derived from benign network traffic, frequently fail to cover the vast array of protocol states and input structures, leading to an incomplete exploration of the attack surface.
• C2: Unknown message structure. Without machine-readable information about the internal structure of protocol messages, fuzzers struggle to make "structurally interesting" changes. They are largely blind to the protocol's grammar, making it difficult to construct messages of unseen types, or to coherently add, remove, or substitute entire data structures within a message without breaking its fundamental validity.
• C3: Unknown state space. Lacking explicit knowledge of the protocol's state machine, fuzzers cannot effectively identify the current state or intelligently direct their exploration towards previously unseen states. This often leads to premature coverage plateaus, where the fuzzer gets stuck, repeatedly exploring already-covered code paths without discovering new ones.

The motivation behind LLM-guided protocol fuzzing is to overcome these limitations by harnessing the power of Large Language Models (LLMs). Pre-trained on billions of internet samples, including extensive collections of RFCs and other technical documentation, LLMs possess a deep understanding of natural language and have demonstrated remarkable capabilities in text generation and translating natural language specifications into structured formats or even executable code. Given that many network protocol messages are text-based, LLMs are uniquely positioned to provide an automatic, easy-to-use, and intelligent guidance mechanism for fuzzing, directly addressing the challenges of seed diversity, message structure inference, and state space exploration.

Key Findings

▶ Watch: Key challenges in current protocol fuzzing methods (2:00)

The research presented by Ruijie Meng yielded compelling evidence for the efficacy of LLM-guided protocol fuzzing, demonstrating significant advancements over existing methods across multiple metrics:

• Superior State Space Coverage: CHATAFL exhibited a remarkable ability to explore the complex state spaces of network protocols. On average, it exercised 47.60% more state transitions than AFLNET and 42.69% more state transitions than NSFUZZ-v. Furthermore, CHATAFL achieved the same level of state transition coverage an average of 48.12 times faster than AFLNET and 15.94 times faster than NSFUZZ-v. In terms of distinct states, CHATAFL covered 29.55% more states than AFLNET and 25.75% more states than NSFUZZ-v, consistently exploring a substantially larger proportion of the reachable state space. For instance, in Live555, CHATAFL covered 14.2 out of 15 reachable states, compared to 10 for AFLNET and 11.7 for NSFUZZ.
• Enhanced Code Coverage: Beyond state exploration, CHATAFL also achieved significantly higher code coverage. It covered 5.81% more branches than AFLNET and 6.74% more branches than NSFUZZ-v, on average. The speed advantage was also evident, with CHATAFL reaching the same branch coverage 6.14 times faster than AFLNET and 10.18 times faster than NSFUZZ-v. These metrics confirm that LLM guidance leads to a more thorough and efficient exploration of the target program's codebase.
• Discovery of Nine Zero-Day Vulnerabilities: Perhaps the most impactful finding is CHATAFL's success in discovering nine distinct and previously unknown zero-day vulnerabilities in widely-used and extensively-tested open-source protocol implementations (Live555, ProFTPD, and Kamailio). These critical security flaws included use-after-free, buffer overflows, and memory leaks, issues with significant security implications such potentially enabling remote code execution or information disclosure. In stark contrast, AFLNET only found three of these bugs, and NSFUZZ-v found four, highlighting CHATAFL's superior bug-finding capability. Seven of these bugs have been confirmed by developers, and three were fixed at the time of paper submission, with CVE IDs requested.
• Positive Contribution of All LLM-Guided Strategies: Ablation studies confirmed that all three LLM-guided strategies—grammar-guided mutation (SA), enriching initial seeds (SB), and surpassing coverage plateaus (SC)—contributed positively and synergistically to CHATAFL's overall performance. Enabling all three strategies (CHATAFL) yielded the highest average increase in branch coverage (5.81%) and the fastest execution (6.14x faster than the baseline).
• Validation of LLM Capabilities: A detailed case study using RTSP and its implementation, LIVE555, with ChatGPT 3.5, empirically validated the core hypotheses about LLMs' utility:
• Grammar Lifting: The LLM successfully generated accurate machine-readable grammars for 9 out of 10 RTSP client request types, identical to manually extracted ground truth.
• Seed Enrichment: The LLM generated diverse and valid messages, with 99% placed accurately in sequences and 100% accepted by the server when provided with appropriate context (e.g., server responses for session IDs).
• State Transition Induction: The LLM effectively inferred protocol states and generated client requests that induced state transitions, with high success rates (e.g., 81% for INIT state, 89% for PLAY state), covering all possible transitions for individual states.

These findings collectively establish LLM-guided protocol fuzzing as a highly effective and promising new paradigm for automated vulnerability discovery in complex network protocols.

Technical Deep Dive

▶ Watch: Harnessing LLMs to overcome fuzzing limitations (3:35)

The LLM-Guided Protocol Fuzzing (LLMPF) approach, implemented in the tool CHATAFL, augments classical mutation-based fuzzing by integrating three key strategies powered by Large Language Models. These strategies directly tackle the challenges of seed dependence (C1), unknown message structure (C2), and unknown state space (C3).

1. Grammar-guided Mutation (Addressing C2: Unknown Message Structure)

The first core component enables structure-aware mutations, moving beyond simple byte-level modifications to generate more semantically valid and interesting test cases.

• Grammar Extraction: The primary challenge here is converting the LLM's natural language understanding into a machine-readable grammar format usable by a fuzzer. CHATAFL employs in-context few-shot learning for prompt engineering. The LLM is provided with a few examples of desired input prompts and corresponding output grammar formats. For instance, the prompt includes examples for RTSP PLAY and HTTP GET requests, demonstrating how message keywords should remain immutable, while mutable regions are replaced with a (Value) placeholder. Using two examples helps prevent over-adherence to a single pattern. To mitigate LLM stochasticity (e.g., generating random message types), CHATAFL engages in multiple conversations and uses a majority-vote or self-consistency check to determine the final grammar. In cases of partial grammar generation, the LLM is simply re-prompted. Once extracted, these grammars are stored in a grammar corpus, G, for the duration of the fuzzing campaign, minimizing subsequent LLM interaction overhead.
• Mutation based on Grammar: With the grammar corpus G available, CHATAFL performs structure-aware mutations. When the fuzzer selects a message portion for mutation, LLMPF first attempts to match it with its corresponding grammar in G (which is structured as a map from message type to grammar). Regular expressions are then used to identify and mark the (Value) regions within the message as mutable. During the mutation process, LLMPF only modifies these marked regions, ensuring that the message largely retains a valid structural format. If no grammar match is found, the fuzzer falls back to treating all regions as mutable, preserving the ability to explore corner cases or potentially malformed messages that might also expose bugs. However, structure-aware mutations are prioritized due to their higher potential for exploring a larger, more relevant state space with valid inputs.

2. Enriching Initial Seeds (Addressing C1: Dependence on Initial Seeds)

This strategy aims to overcome the limitations of sparse or undiverse initial seed corpora by leveraging the LLM to generate new, diverse, and context-aware messages.

• Challenge (i): Correct Context Information: Protocols often require dynamic context, such as dynamically assigned session IDs. LLMs, by default, lack this real-time information. CHATAFL addresses this by including existing initial seeds (e.g., from PROFUZZBENCH) in the prompt. These seeds, captured from live network traffic, contain correct context information, allowing the LLM to learn and incorporate it. For instance, by including server responses in the prompt, the LLM can generate client requests with the correct session IDs.
• Challenge (ii): Maximizing Diversity: To introduce new message types, the fuzzer first identifies which client request types are missing from the initial seeds by comparing AllTypes (from grammar extraction) with ExistingTypes. The LLM is then instructed to generate these MissingTypes and insert them into the seed corpus. To manage sequence length, CHATAFL typically adds two missing types at a time.
• Challenge (iii): Generating an Entire Modified Sequence: To ensure the validity and coherence of the generated sequence, a "continuation format" is used in the prompt. The LLM is provided with an existing sequence and asked to provide the "modified sequence of client requests is:". This enables the LLM to intelligently insert new messages into appropriate locations within the existing sequence, maintaining structural and semantic integrity. The LLM's output can then be directly used as new seeds after minor post-processing.

3. Surpassing Coverage Plateau (Addressing C3: Unknown State Space)

This mechanism is designed to reactivate fuzzing progress when the fuzzer becomes stuck in a coverage plateau, unable to discover new code paths or states.

• Plateau Detection: A global variable, PlateauLen, tracks the number of consecutive "uninteresting" seed executions (i.e., those that don't crash the program or increase coverage). PlateauLen is reset upon a crash or coverage increase. When PlateauLen exceeds a predefined maximum length (MaxPlateau, set to 512 in experiments), the fuzzer is considered to be in a coverage plateau.
• LLM Intervention: Upon detecting a plateau, LLMPF queries the LLM to generate the "next suitable client requests that may induce state transitions to other states." The prompt provides the LLM with the complete communication history between the server and client (both requests and responses). To ensure authentic message generation, a real message example from the initial seed corpus is included as a format guide. The LLM then infers the current protocol state and generates a new client request (M2') designed to transition the server to a different, unexplored state. For example, if the server is in the READY state and the fuzzer is stuck, the LLM, given the SETUP, 200-OK history, might suggest a PLAY or RECORD message, effectively breaking the plateau.

Implementation and Configuration

CHATAFL is implemented by integrating these LLM-guided strategies into AFLNET, a popular greybox fuzzer for network protocols. AFLNET's existing framework, which infers state machines based on response codes and uses state/code feedback, provides a robust foundation for CHATAFL's enhancements. CHATAFL is specifically designed for text-based protocols with publicly available RFCs, as these are typically included in the LLM's training data.

Experimental configurations utilized the gpt-3.5-turbo model. For grammar extraction and seed enrichment, where factual and precise responses are crucial, the temperature was set to 0.5. For breaking coverage plateaus, which requires more generative and diverse responses, a temperature of 1.5 was used. Five repetitions were employed for self-consistency checks during grammar extraction to filter out incorrect outputs. The MaxPlateau value was set to 512, a heuristic chosen to balance LLM query costs and fuzzer stagnation, typically achievable within 10 minutes.

The evaluation used six text-based protocol implementations from the PROFUZZBENCH benchmark: Live555 (RTSP), ProFTPD (FTP), PureFTPD (FTP), Kamailio (SIP), Exim (SMTP), and forked-daapd (DAAP). These are mature, widely-used open-source projects. Baselines included AFLNET and NSFUZZ-v. Code coverage was measured as branch coverage using PROFUZZBENCH's automated tooling, and state space coverage by distinct response codes and state transitions. Address Sanitizer (ASAN) was used to detect bugs, with crashing sequences recorded and reproduced for confirmation.

Demo / Proof of Concept

▶ Watch: LLM's ability to extract accurate message grammars (4:10)

While the talk did not feature a live coding demonstration, it effectively presented several proofs of concept through a detailed case study and the in-depth analysis of a discovered zero-day vulnerability. These examples clearly illustrate how the LLM-guided strategies function and their combined impact.

RTSP/LIVE555 Case Study: Validating LLM Capabilities

The initial validation of LLMs' utility for protocol fuzzing was conducted using the Real Time Streaming Protocol (RTSP) and its implementation, LIVE555, from the PROFUZZBENCH benchmark, specifically leveraging the ChatGPT 3.5 model. This served as a foundational proof of concept for each of CHATAFL's core strategies:

1. Lifting Message Grammars: The LLM was prompted to generate machine-readable grammars for 10 RTSP client request types. The results were highly accurate, with 9 out of 10 generated grammars being identical to manually extracted ground truth. Even with some stochasticity (e.g., occasionally overlooking an optional field), the LLM consistently demonstrated a strong ability to infer and structure protocol syntax.
2. Enriching the Seed Corpus: The initial LIVE555 seed corpus in PROFUZZBENCH was notably sparse, covering only 4 of 10 client request types. The LLM was tasked with generating 100 new messages across all 10 types and inserting them into existing sequences. A remarkable 99% of these generated requests were placed in accurate positions, and approximately 55.1% were directly accepted by the LIVE555 server. The remaining unsuccessful cases were not due to LLM errors but external factors like unsupported functionalities or incorrect session IDs. Critically, by including server responses (containing dynamic session IDs) in the prompt, the LLM learned to generate requests with correct context, leading to 100% acceptance. This showcased the LLM's ability to create diverse, context-aware, and valid seed inputs.
3. Inducing Interesting State Transitions: To address the challenge of coverage plateaus, the LLM was prompted with communication history and asked to generate messages that would lead to new states. For the four key RTSP states (INIT, READY, PLAY, RECORD), the LLM-generated requests successfully induced state transitions at high rates: 81% for INIT, 74% for READY, 89% for PLAY, and 69% for RECORD. This demonstrated the LLM's capability to infer current protocol states and intelligently suggest actions to explore new state pathways.

These preliminary results provided strong evidence that LLMs could effectively address the core challenges of protocol fuzzing, paving the way for the full CHATAFL implementation.

Bug #1: Heap-Use-After-Free in Live555

A compelling proof of concept for the synergistic power of CHATAFL's strategies is the discovery of Bug #1, a heap-use-after-free vulnerability found in Live555. This bug, a critical memory corruption issue, perfectly illustrates how LLM guidance enabled the discovery of a complex flaw that eluded state-of-the-art baselines.

• Triggering Condition: The bug required a very specific sequence of client requests: SETUP - PLAY - PAUSE - PLAY.
• Baseline Limitations: The crucial PAUSE client request was entirely absent from the initial seeds used by AFLNET and NSFUZZ. While theoretically possible for mutation-based fuzzers to randomly generate such a specific, valid message, it is highly improbable in practice. Indeed, neither AFLNET nor NSFUZZ managed to generate PAUSE requests in any of their experimental runs, thus failing to reach the vulnerable state.
• CHATAFL's Synergistic Advantage:
• Seed Enrichment: CHATAFL, guided by the LLM during the seed enrichment phase, was able to identify PAUSE as a MissingType (from the full RTSP grammar) and intelligently insert it into the initial seed corpus. This provided the fuzzer with the necessary building block for the exploit sequence.
• Surpassing Coverage Plateau: Once the PAUSE message was available, triggering the bug still required specific state transitions. When the fuzzer got stuck in the PLAY state, the LLM's guidance during the surpassing coverage plateau phase became instrumental. Given the communication history, the LLM could infer the PLAY state and suggest the PAUSE message, enabling the PLAY - READY transition, which was critical for the subsequent re-PLAY request to trigger the use-after-free.
• Grammar-guided Mutation: Throughout this process, the grammar-guided mutation strategy played a vital role in maintaining the validity and structural integrity of the complex SETUP - PLAY - PAUSE - PLAY sequence. Traditional structure-unaware mutations from baselines would have a high likelihood of corrupting such a precise sequence, rendering the bug untriggerable even if the messages were present.

This detailed example unequivocally demonstrates that the combined application of all three LLM-guided strategies—seed enrichment, plateau breaking, and grammar-guided mutation—was essential for CHATAFL to navigate the complex state machine and message structures required to uncover this sophisticated, previously unknown vulnerability.

Defensive Implications

▶ Watch: LLM's ability to enrich diverse seed corpora (5:20)

The findings from CHATAFL have profound implications for defenders and developers responsible for securing network protocol implementations:

1. Acknowledge Limitations of Traditional Fuzzing: The discovery of nine zero-day vulnerabilities in mature, widely-used, and extensively-tested protocols (like Live555, ProFTPD, and Kamailio) underscores that current state-of-the-art fuzzing techniques (e.g., AFLNET, NSFUZZ-v) are insufficient for comprehensively testing complex, stateful protocols. Defenders should recognize that even systems considered "well-fuzzed" may harbor deep-seated vulnerabilities.
2. Integrate LLM-Guided Fuzzing into Security Pipelines: Organizations developing or deploying software with critical protocol implementations should strongly consider incorporating LLM-guided fuzzing approaches into their continuous integration/continuous deployment (CI/CD) pipelines and security testing methodologies. CHATAFL's ability to automate the extraction of protocol knowledge and intelligently guide fuzzing significantly lowers the barrier to entry for robust protocol testing, especially for text-based protocols with publicly available RFCs.
3. Prioritize Robust Protocol Specification and Implementation: The effectiveness of LLM-guided fuzzing highlights the importance of precise and unambiguous protocol specifications. While LLMs excel at interpreting natural language RFCs, ambiguities can still lead to misinterpretations. Developers should strive for strict adherence to protocol standards and implement robust input validation and state management to mitigate risks.
4. Focus on State Machine Design and Transitions: CHATAFL's success in navigating complex state machines and inducing state transitions (e.g., PLAY - PAUSE - PLAY for Bug #1) indicates that vulnerabilities often lie in the transitions between states or in how context (like session IDs) is handled across these transitions. Developers should pay particular attention to the design and implementation of their protocol's state machine, ensuring all possible transitions are handled securely and edge cases are thoroughly considered.
5. Be Aware of Cost-Benefit for Proprietary Protocols: While highly effective for protocols with public RFCs, the current LLM-guided approach may not be optimal for proprietary protocols without publicly available specifications that are part of the LLM's training data. Organizations dealing with proprietary protocols may need to explore fine-tuning LLMs on private specifications or adapting techniques for scenarios with limited public information.
6. Continuous Vigilance Against Memory Corruption: The types of bugs found (use-after-free, buffer overflows, memory leaks) are classic memory corruption vulnerabilities that can lead to severe consequences, including remote code execution. This reinforces the need for memory-safe programming practices, rigorous code reviews, and the continued use of sanitizers like Address Sanitizer (ASAN) during development and testing. The fact that LLMs can now systematically trigger these complex sequences means that these issues are no longer hidden by the difficulty of reaching them.

In essence, LLM-guided fuzzing represents a new frontier in automated vulnerability discovery, challenging developers and defenders to re-evaluate their current security testing paradigms and embrace more intelligent, specification-aware approaches to secure the critical components of our networked world.

Key Takeaways

• LLMs Revolutionize Protocol Fuzzing: Large Language Models are highly effective in guiding protocol fuzzing, overcoming critical limitations of traditional mutation-based fuzzers by providing structural, contextual, and state-aware intelligence.
• CHATAFL Outperforms State-of-the-Art: The developed tool, CHATAFL, significantly surpasses baseline fuzzers like AFLNET and NSFUZZ-v in both state space coverage (47.60% more transitions, 29.55% more states) and code coverage (5.81% more branches), achieving these results substantially faster.
• Discovery of Critical Zero-Days: CHATAFL successfully discovered nine distinct and previously unknown zero-day vulnerabilities, including use-after-free and buffer overflows, in widely-used protocol implementations, demonstrating its superior bug-finding capability compared to baselines.
• Three Synergistic LLM-Guided Strategies: The effectiveness of CHATAFL stems from the combined application of three LLM-guided strategies: grammar extraction for structure-aware mutation, seed enrichment for diversity and context, and intelligent message generation to surpass coverage plateaus.
• Minimal Manual Effort Post-Setup: Once initial prompt engineering for grammar extraction is complete (a one-time effort), CHATAFL operates with minimal manual intervention, automatically generating prompts for seed enrichment and plateau breaking.
• Optimal for Text-Based Protocols with Public RFCs: While highly promising, the current approach is most effective for text-based protocols with publicly available RFCs, as these are typically part of the LLM's training data.

About the Speaker(s)

Ruijie Meng is the presenter of this work on Large Language Model guided Protocol Fuzzing. The talk highlights his research into leveraging advanced AI techniques to address long-standing challenges in cybersecurity, specifically in the realm of automated vulnerability discovery for network protocols.