DeGPT: Optimizing Decompiler Output with LLM

DeGPT: Optimizing Decompiler Output with LLM

Speakers: Peiwei Hu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=G4MniivEvGo

Overview

Reverse engineering stands as a cornerstone in cybersecurity, indispensable for critical tasks such as malware analysis, vulnerability discovery, and understanding closed-source software. At its heart, this discipline seeks to unravel the intricate program logic and algorithms embedded within target binaries. Decompilers, tools like Hex-Rays and Ghidra, are pivotal in this process, transforming low-level assembly code into more human-readable, high-level programming languages. Despite their widespread use and significant advancements, state-of-the-art decompilers grapple with a fundamental challenge: the profound loss of semantic information during compilation. Crucial details, including meaningful variable names and descriptive comments, are typically stripped away, leaving reverse engineers with outputs that are difficult to parse and comprehend.

This talk introduces DeGPT, an innovative, end-to-end framework designed to address these long-standing limitations by leveraging the power of Large Language Models (LLMs). DeGPT aims to significantly enhance the readability and simplicity of decompiler output, thereby fundamentally assisting reverse engineers in their complex tasks. The framework confronts two primary challenges inherent in applying LLMs to this domain: the suboptimal performance of one-shot LLM prompts for multi-task optimization (C1) and the inherent uncertainty of generative LLM outputs, which could potentially compromise the semantic fidelity of the decompiled code (C2).

DeGPT proposes a novel three-role mechanism and a Micro Snippet Semantic Calculation (MSSC) component to overcome these hurdles. By strategically orchestrating LLM interactions and rigorously validating semantic preservation, DeGPT offers a comprehensive solution for decompiler output enhancement. This groundbreaking work promises to streamline reverse engineering workflows, making binary analysis more efficient, accurate, and accessible, ultimately bolstering capabilities in threat intelligence and defensive security.

Background

▶ Watch: Introduction to reverse engineering and decompiler challenges (0:00)

The necessity of reverse engineering permeates various facets of modern computing, from analyzing sophisticated malware to identifying zero-day vulnerabilities in proprietary software. Decompilers are the primary interface through which human analysts interact with the high-level representation of a binary's logic. Tools such as Hex-Rays and Ghidra perform the arduous task of translating machine-level instructions into C-like pseudocode, drastically reducing the cognitive burden compared to direct assembly analysis. However, this translation process is inherently lossy, leading to several significant drawbacks that DeGPT specifically targets.

One of the most prominent issues is the presence of redundant structures. During compilation, especially with optimization levels like -O0 (no optimization), compilers might introduce local variables or control flow constructs that, while functionally equivalent to the original source, are less concise and increase cognitive load. For instance, in a simple Fibonacci function, a decompiler might introduce variables like iVar1 and iVar2 that have no direct counterpart in the original C code. Decompilers rely on fixed algorithms to lift assembly to high-level constructs, often lacking the human-like ability to simplify code based on broader context.

Secondly, decompiler output is plagued by meaningless identifiers. Original source code employs descriptive variable and function names (e.g., number for an input parameter) that convey crucial semantic information. Post-compilation, these names are typically stripped, leaving decompilers to generate generic placeholders like param_1, iVar1, or local_4h. While param_1 might correctly identify a function parameter, it lacks the specificity needed for true comprehension. This issue stems from the irreversible loss of identifier information and the decompiler's difficulty in inferring meaningful names from its internal, often restructured, representations.

Finally, a complete lacking of comments severely hampers readability. Programmers routinely embed comments to explain complex logic, design choices, or non-obvious functionalities. These comments are entirely discarded during compilation, making their recovery by decompilers impossible. Although some research has explored auto-generating comments, these efforts often falter when faced with the meaningless identifiers and convoluted structures prevalent in decompiler output. Without comments, analysts are forced to painstakingly reconstruct the code's intent solely from its opaque logic.

Previous research has attempted to refine decompiler output readability by reconstructing semantic information. These efforts generally fall into two categories: (i) approaches that embed information from disassembled code, such as mnemonic sequences and control flow graphs, often employing encoder-decoder architectures; and (ii) methods that directly enhance the decompiler output, recognizing its richer semantic context compared to raw assembly. However, these prior studies faced significant limitations: their performance degraded severely when encountering unseen binaries due to restricted training dataset sizes and model capabilities, and they offered a narrow scope of optimizations, typically focusing only on variable renaming and type inference, neglecting broader structural simplification and comment generation. DeGPT aims to transcend these limitations by harnessing the advanced capabilities of Large Language Models.

Key Findings

▶ Watch: Detailing decompiler output drawbacks: redundancy, poor names, no comments (2:00)

DeGPT represents a significant advancement in decompiler output optimization, delivering a comprehensive, LLM-powered framework that tackles long-standing challenges in reverse engineering. The core innovation lies in its novel three-role mechanism—Referee (R_ref), Advisor (R_adv), and Operator (R_ope)—which orchestrates LLM interactions to perform fine-grained, step-by-step optimizations. This mechanism is complemented by the Micro Snippet Semantic Calculation (MSSC) component, crucial for ensuring the semantic fidelity of the optimized code, thereby mitigating the inherent uncertainty of generative LLM outputs.

DeGPT focuses on three primary optimization types: Variable Renaming, Comment Appending, and Structure Simplification. Its effectiveness was rigorously evaluated across a diverse dataset comprising functions from LeetCode, Coreutils, Mirai, and AudioFlux, compiled with -O2 and tested on both non-stripped and stripped binaries using Ghidra 10.2.3.

The framework's overall effectiveness is demonstrated through several key metrics:

• Meaningful Variable Ratio (MVR): DeGPT achieved an average MVR of 30.3% for non-stripped binaries and 24.2% for stripped binaries. This significantly outperforms state-of-the-art variable renaming frameworks like DIRTY, which only manages an 11.1% MVR on non-stripped binaries. This highlights DeGPT's superior ability to infer and assign meaningful names.
• Effort Ratio (ER): The average ER was 75.6% for non-stripped and 76.3% for stripped binaries, indicating a substantial 24.4% reduction in the cognitive burden associated with understanding decompiler outputs. This metric directly reflects the success of structure simplification.
• Correct Rate of Comments (CR): DeGPT demonstrated exceptional reliability in comment generation, achieving an average CR of 99.2% for both non-stripped and stripped binaries, meaning it rarely produces incorrect comments.
• Non-trivial Rate of Comments (NR): For non-stripped binaries, the average NR was 62.9%, signifying that a majority of generated comments provide valuable, non-trivial semantic information. For stripped binaries, NR dropped to 38.1%, as expected due to reduced semantic context.

A critical finding is DeGPT's strong generalization capability. Even when tested on AudioFlux, a codebase released in 2023 (after ChatGPT's training cutoff of September 2021), DeGPT maintained decent performance. This showcases a significant advantage over previous model-based approaches that often struggle with unseen binaries.

The three-role mechanism proved instrumental. Compared to a one-shot optimization approach, it led to 2.5 times more structural simplification and 3.1 times more variable renaming optimizations. It also boosted the Optimized Name Ratio (ONR) for variable renaming from a mere 57.0% (one-shot) to DeGPT's impressive 92.4%, indicating nearly all variables were renamed. Furthermore, the R_ref component alone contributed to a 21.9% reduction in API token consumption by intelligently determining the necessity of optimizations.

The MSSC component, vital for semantic preservation, demonstrated an 84% accuracy and 85% recall rate in detecting function semantic changes. It successfully identified and rejected 21.9% of LLM-proposed simplification suggestions that would have altered the original function's behavior.

User studies corroborated DeGPT's practical value, showing a 126% improvement in task accuracy and 47% time savings for participants attempting to understand code snippets. Overall, users reported a generally positive attitude, with scores ranging from 7.0 to 9.5 for various aspects of readability.

Finally, DeGPT exhibits remarkable robustness and versatility. Its performance metrics remained stable across different compiler optimization levels (O0 to O3) and it showed effectiveness when applied to outputs from other mainstream decompilers like Hex-Rays and Binary Ninja, not just Ghidra. While GPT-4 was explored, gpt-3.5-turbo proved sufficient for current objectives without an overwhelming performance advantage.

Technical Deep Dive

▶ Watch: DeGPT's novel three-role mechanism for LLM optimization (4:00)

DeGPT’s architecture is an end-to-end framework meticulously designed to optimize decompiler output for enhanced readability, conciseness, and semantic preservation. Its core strength lies in a sophisticated interaction model with Large Language Models, addressing the inherent challenges of one-shot LLM use and semantic fidelity.

The framework's cornerstone is the three-role mechanism, which meticulously decomposes the complex optimization task into manageable, sequential steps, guiding the LLM at each stage. This mechanism consists of three distinct roles:

1. Referee (R_ref): This initial role acts as a gatekeeper, determining which of the three target optimizations—variable renaming, comment appending, and structure simplification—are necessary for a given decompiler output. By querying the LLM with a concise prompt ("Do you think the following C code needs structure simplification, comment appending, and variable renaming? Answer three yes or no in order. No explanation."), R_ref efficiently identifies required optimizations. This pre-screening process is critical for cost-effectiveness, as it avoids unnecessary API token consumption, demonstrating savings of up to 21.9% in evaluations.

1. Advisor (R_adv): Upon receiving the optimization scheme from R_ref, the Advisor's role is to generate specific rectification measures to implement these optimizations. Recognizing the generative nature of LLMs, R_adv employs carefully designed, specific prompts for each optimization type (as detailed in Table I of the paper) to ensure predictable and analyzable LLM outputs. For instance, variable renaming prompts explicitly request old and new names in a JSON format (e.g., 'old name': 'new name') and forbid explanations to control verbosity. A crucial aspect of R_adv is its optimization sorting principle. Structure-related optimizations (e.g., removing redundant variables) are prioritized before semantic-related ones (renaming, comments). This is because structural changes can significantly alter the context for subsequent semantic modifications. Furthermore, comment appending is performed before variable renaming, as descriptive comments can provide valuable contextual cues, assisting the LLM in selecting more accurate variable names. R_adv iteratively applies these sorted optimizations, sending the current decompiler output and the appropriate prompt to the LLM, and then passing the modified code to R_ope for semantic validation.

1. Operator (R_ope): This is the crucial final arbiter, inspecting the rectification measures proposed by R_adv. Its primary responsibility is to ensure that optimizations do not alter the original function's semantics, thereby upholding the fidelity of the decompiler output. Correctness is defined as maintaining behavioral equivalence between the original function F and the optimized function F', satisfying two conditions: F(i) = F'(i) (identical return values for any input i) and SideEffect(F(i)) = SideEffect(F'(i)) (identical side effects for any input i). To achieve this, R_ope integrates the Micro Snippet Semantic Calculation (MSSC) component.

Micro Snippet Semantic Calculation (MSSC)

MSSC is a novel static analysis component designed to check for function semantic changes without requiring actual execution of the decompiler output, which is generally not compilable. It addresses the challenges of simulating code semantics and tracing side effects by:

• Simulating Function Execution: MSSC assigns unique random numbers to all symbols (variables, memory locations accessed by pointers, callee arguments) and updates their values along all identified execution paths. These random numbers are shared between the original and optimized code snippets to allow for direct comparison.
• Handling Call Chains: To avoid the prohibitive overhead of tracing deep call chains (inter-procedural analysis), MSSC represents function invocations with unique random numbers during simulation. This effectively converts inter-procedural analysis into intra-procedural analysis, where the "return value" or "side effect" of a callee is abstractly represented by its unique ID. While theoretically, different random numbers might coincidentally lead to the same values, the probability is negligible in practice.

MSSC operates in two phases:

1. Calculation Phase (Algorithm 1):

• Input: A code snippet (original or optimized).
• Output: SymTables (a set of symbol value tables for each execution path) and CallLogs (a set of invocation logs for each path).
• Steps:

1. Initialize empty SymTables and CallLogs.
2. Identify all possible Paths within the code snippet using getExecutionPath(Code).
3. Collect all Syms (variables, pointer-accessed memory) in the code.
4. Assign a unique random number to each Sym in an initial SymTable.
5. For each Path:

• Create a copy of the initial SymTable and a new CallLog.
• Iterate through each Statement in the Path:
• UpdateSymTable based on the statement's effect on symbols.
• If the statement involves a function invocation, LogInvocation with its arguments and assign a unique random number for its return value if it participates in further computations.
• Add the final SymTable and CallLog for this path to their respective sets.

1. Comparison Phase:

• This phase takes the SymTables and CallLogs generated for both the original and optimized code snippets.
• Invocation Checking: Compares the CallLogs. If they differ (e.g., an invocation was removed or arguments changed), DeGPT concludes that function semantics have been altered, as the LLM has no basis to change callee behavior. This efficiently detects changes in side effects.
• Variable Checking: Compares the values of return variables and non-local variables in the SymTables for each corresponding execution path.

If both invocation and variable checks pass, R_ope accepts the optimized code. MSSC has been evaluated to detect function semantic changes with an accuracy of 84% and a recall rate of 85%.

Implementation Details

DeGPT is implemented using ChatGPT (gpt-3.5-turbo, temperature 0.2) as the LLM backend, chosen for its performance and user-friendly API. The LLM interaction components are approximately 900 lines of Python. For program analysis, DeGPT uses cinspector, a tree-sitter-based code analysis framework, as decompiler output, while C-like, is not directly compilable by standard compilers. The analysis components comprise about 3,100 lines of Python. To mitigate path explosion issues in complex functions, loops are transformed into branch statements (similar to AURC), and a time limit is imposed on analyses. The average execution path length in the dataset was 6.2, with a maximum of 106, demonstrating the feasibility of this approach.

Demo / Proof of Concept

▶ Watch: DeGPT's three main optimization types: renaming, comments, simplification (4:30)

While DeGPT doesn't involve a live, interactive software demonstration in the traditional sense, the talk vividly illustrates its workflow and efficacy through a conceptual example using the Fibonacci function (as shown in Figure 3 of the paper). This illustration serves as a clear proof of concept for how the framework processes and optimizes decompiler output.

The demonstration begins with a typical Ghidra decompiler output for a simple Fibonacci function, compiled without debug information. This original output exemplifies the challenges DeGPT aims to solve: redundant local variables (e.g., iVar1, iVar2), meaningless parameter names (param_1), and a complete absence of comments.

DeGPT’s workflow then unfolds through the three-role mechanism:

1. R_ref's Assessment: The Referee (R_ref) first queries the LLM to determine the necessity of the three optimization types. For the Fibonacci example, R_ref receives "Yes" for structure simplification, comment appending, and variable renaming, indicating all are required.
2. R_adv's Guidance: The Advisor (R_adv) receives this scheme. It prioritizes optimizations, performing structure simplification first, followed by comment appending, and then variable renaming. R_adv selects the appropriate prompt for structure simplification and sends the decompiler output to the LLM for specific editing suggestions.
3. LLM's Suggestion and R_ope's Validation: The LLM generates a simplified version of the code, potentially removing redundant variables or restructuring control flow. This new, simplified code is then sent to the Operator (R_ope). R_ope's embedded Micro Snippet Semantic Calculation (MSSC) component rigorously compares the semantic behavior of the simplified code against the original. If MSSC confirms that no semantic changes have occurred, the simplified version is accepted, and the process continues to the next optimization (e.g., adding comments). If a semantic change is detected, the LLM's suggestion is rejected, preventing the introduction of incorrect code.
4. Iterative Refinement: This iterative process continues for comment appending and variable renaming. For comments, the LLM is prompted to add descriptive comments. For variable renaming, it's asked to provide meaningful names, often in a structured format like JSON. Each step is validated by MSSC to ensure fidelity.

The final output, as depicted in Figure 3-4 of the talk, is a dramatically improved version of the Fibonacci function. It features a more concise structure, meaningful variable names (e.g., number instead of param_1), and helpful comments explaining the function's purpose and logic. This transformation clearly demonstrates DeGPT's ability to turn opaque decompiler output into easily understandable C-like code.

Beyond this illustrative example, the effectiveness of DeGPT is further demonstrated by the comprehensive user studies. In task-based evaluations, participants achieved a 126% improvement in accuracy and 47% time savings when analyzing DeGPT-optimized code compared to raw decompiler output. This practical, measurable impact on reverse engineering tasks serves as a strong proof of concept for DeGPT's real-world utility.

Defensive Implications

▶ Watch: MSSC component: ensuring semantic fidelity in decompiler output (4:50)

DeGPT's advancements have profound implications for defensive security, directly enhancing the capabilities of reverse engineers, security analysts, and tool developers. By transforming unintelligible decompiler output into readable, semantically preserved code, DeGPT empowers defenders in several critical areas:

1. Accelerated Malware Analysis: Malware often employs obfuscation techniques and is typically stripped of debug information. DeGPT's ability to simplify structures, rename variables meaningfully, and add relevant comments can drastically reduce the time and effort required to understand malicious binaries. This faster analysis translates into quicker threat intelligence generation, more rapid signature development, and improved incident response times.

1. Enhanced Vulnerability Discovery: Identifying vulnerabilities in closed-source software or complex binaries is a painstaking process. DeGPT's optimizations, particularly the 24.4% reduction in cognitive burden (measured by Halstead's Effort), allow security researchers to grasp complex program logic more efficiently. This clarity can help in pinpointing logical flaws, identifying insecure API usages, and discovering memory corruption vulnerabilities with greater speed and accuracy.

1. Improved Closed-Source Software Comprehension: For organizations needing to understand the behavior of third-party libraries, firmware, or proprietary applications for compliance, interoperability, or security auditing, DeGPT offers a powerful aid. The framework provides a clearer view into the internal workings of such software, facilitating better security assessments and risk management without access to source code.

1. Higher Analyst Efficiency and Accuracy: The user studies unequivocally demonstrate DeGPT's direct impact on analyst performance, showing a 126% improvement in task accuracy and 47% time savings for understanding code snippets. This means security teams can analyze more binaries in less time, with fewer errors, leading to a more robust defensive posture.

1. Foundation for Next-Generation Security Tools: For developers of security analysis platforms (e.g., IDA Pro, Ghidra, Binary Ninja plugins, or automated vulnerability scanners), DeGPT provides a blueprint for integrating LLM-powered optimization. The three-role mechanism and MSSC offer a robust, semantics-preserving approach that can be adopted to enhance existing tools or build new ones. The findings on DeGPT's stability across compiler optimization levels and versatility across different decompilers (Ghidra, Hex-Rays, Binary Ninja) ensure broad applicability.

1. Better Training and Onboarding for Reverse Engineers: The simplified and commented output from DeGPT can serve as an invaluable educational tool for aspiring reverse engineers. It lowers the barrier to entry for understanding complex binaries, accelerating the learning curve for new analysts and making specialized knowledge more accessible.

In essence, DeGPT empowers defenders by making the opaque world of binary code more transparent. By reducing the complexity and ambiguity inherent in decompiled output, it directly contributes to more effective and efficient cybersecurity operations, allowing organizations to better understand, detect, and respond to threats.

Key Takeaways

• DeGPT is a pioneering end-to-end framework that leverages Large Language Models (LLMs) to comprehensively optimize decompiler output. It significantly enhances readability and simplicity, directly addressing long-standing challenges in reverse engineering such as meaningless variable names, redundant structures, and missing comments.
• The novel three-role mechanism (Referee, Advisor, Operator) is crucial for maximizing LLM potential and efficiency. This step-by-step orchestration of LLM interactions leads to significantly more optimizations (e.g., 2.5x more structural simplification, 3.1x more variable renaming) and saves up to 21.9% in API token costs compared to one-shot approaches.
• Micro Snippet Semantic Calculation (MSSC) is a critical component ensuring the semantic fidelity of optimized code. Integrated within the Operator role, MSSC achieves 84% accuracy and 85% recall in detecting harmful semantic changes, preventing LLMs from introducing incorrect modifications that could mislead reverse engineers.
• DeGPT demonstrably improves reverse engineering efficiency and accuracy. User studies revealed a 24.4% reduction in cognitive burden (Effort Ratio), a 126% improvement in task accuracy, and 47% time savings when analysts worked with DeGPT-optimized code.
• The framework exhibits strong generalization capabilities and robustness. It performs effectively on unseen codebases (like AudioFlux, released after LLM training cutoffs), maintains stability across various compiler optimization levels (O0-O3), and is versatile enough to work with outputs from different mainstream decompilers (Ghidra, Hex-Rays, Binary Ninja).
• DeGPT marks a significant step forward in assisting reverse engineers. By making binary analysis more accessible, efficient, and less error-prone, it offers substantial benefits for malware analysis, vulnerability discovery, and understanding closed-source software, ultimately bolstering defensive security capabilities.

About the Speaker(s)

The talk "DeGPT: Optimizing Decompiler Output with LLM" was presented by Peiwei Hu. The provided transcript and metadata do not offer additional biographical details about the speaker, such as their title or company affiliation.