No Symbols When Reversing No Problem Bring Your Own Max ‘Libra’ Kersten

No Symbols When Reversing No Problem Bring Your Own

Speakers: Max 'Libra' Kersten, Advanced Research Team, Trellix

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=-re_QOQoILY

Overview

In the intricate world of binary analysis, the absence of symbolic information presents a significant hurdle for reverse engineers. Max 'Libra' Kersten's talk, "No Symbols When Reversing No Problem Bring Your Own," addresses this pervasive challenge by introducing a robust methodology and automated tools to recover function names and identify code overlap in stripped binaries. This presentation is a crucial contribution for malware researchers, vulnerability analysts, and anyone dealing with compiled code where original debugging symbols have been intentionally removed or were never present.

The core of Kersten's work revolves around leveraging and extending Ghidra's powerful analysis capabilities, specifically its Function ID and Behavioral Similarity (BSim) features. By automating the creation and application of signature databases for common runtimes and libraries, his approach dramatically reduces the manual effort required to understand complex binaries. This talk not only highlights the technical specifics of these Ghidra features but also provides a practical, scalable methodology for integrating them into a streamlined reverse engineering workflow, enabling analysts to focus on the unique, user-defined logic within a binary rather than re-identifying known components.

The importance of this work cannot be overstated in the current threat landscape. Malware authors frequently strip binaries to hinder analysis, and many legitimate applications ship without symbols to reduce size or protect intellectual property. Kersten’s techniques empower analysts to overcome these obfuscation tactics, accelerating the identification of malicious functionality, vulnerable library versions, and shared code across different samples, thereby enhancing threat intelligence and defensive capabilities.

Background

▶ Watch: Introduction to symbol recovery challenges in binaries (0:00)

To appreciate the significance of recovering symbols, it's essential to understand what symbols are and why their absence poses such a challenge. From a reverse engineering perspective, a symbol is typically a human-readable function name (e.g., main, memcpy, CreateWindowExA). Ideally, these names should be recognizable, self-explanatory, and unambiguous, providing immediate insight into a function's purpose. However, many binaries, especially those compiled for release or those used in malicious contexts, are "stripped," meaning these valuable symbols are removed to reduce file size or complicate analysis. When symbols are absent, reverse engineers are left with generic names like FUN_00401234, requiring extensive manual effort to divine the function's true role.

Binaries generally consist of three conceptual types of code, each with different implications for symbol recovery:

1. Runtime Code: This refers to the code provided by the programming language's runtime environment. For example, in a Golang binary, a significant portion of the runtime is statically compiled into the executable. This means that multiple Golang binaries, even if unrelated in their primary function, will share a substantial amount of identical runtime code if compiled with the same Golang version. Identifying this common runtime code is crucial to filter out noise and focus on the unique application logic.
2. Library Code: This encompasses functions from external libraries linked into the binary (e.g., Boost, OpenSSL, lib XML, lib G crypt). These libraries often provide well-defined functionalities, and their presence can indicate specific capabilities or vulnerabilities. For instance, identifying a particular version of a library like OpenSSL might immediately flag known security flaws (e.g., a specific CVE). Tracking malware based on its use of specific library versions is also a common practice.
3. User Code: This is the unique code written by the developer or malware author, representing the core logic of the application. From a malware researcher's perspective, this is often the most interesting part, as it dictates the specific actions and capabilities of the malicious software. Identifying shared user code across different binaries can reveal connections between malware families or campaigns.

The problem, then, is that without symbols, differentiating between these code types and understanding their purpose becomes a daunting task. While tools might identify code overlap, without names, it's hard to interpret what that overlap signifies. This is where symbol recovery techniques become indispensable, allowing analysts to quickly label known runtime and library functions, thereby isolating and focusing on the critical user code.

Prior work in this area includes tools like IDA Pro's FLIRT (Fast Library Identification and Recognition Technology) signatures, which have long been used to identify common library functions. Ghidra, too, has its own equivalent features, which Kersten's work builds upon and automates to provide a more comprehensive and efficient solution for modern reverse engineering challenges.

Key Findings

▶ Watch: Overview of Ghidra's Function ID and BSim features (3:50)

Max Kersten's presentation highlights several key findings and contributions that significantly advance the state of symbol recovery in binary analysis:

First, the talk underscores the immense utility of two specific Ghidra features for symbol recovery: Function ID and Behavioral Similarity (BSim). Function ID, Ghidra's equivalent to IDA's FLIRT, operates by hashing functions and comparing these hashes against a database of known functions and their corresponding names. This allows for rapid identification and renaming of functions based on their unique structural characteristics. BSim, a feature recently released by the NSA, takes a more sophisticated approach. It leverages Ghidra's intermediate language, P-code, to compute "high P-code vectors" that represent the behavioral semantics of a function. This allows BSim to find functions that exhibit similar behavior, even if their compiled instruction sequences differ slightly due to compiler optimizations or obfuscation techniques.

Second, Kersten's primary contribution is the development of an automated methodology and accompanying scripts that streamline the entire symbol recovery process. Recognizing the manual overhead involved in creating and applying these signatures, he devised a system to run Ghidra in a headless mode. This automation allows for the batch ingestion of binaries and the generation of Function ID databases for various architectures and Ghidra Language IDs. This is a critical step towards making symbol recovery a routine, rather than an ad-hoc, part of the analysis pipeline.

Third, the research involved the creation of substantial Function ID databases for widely used runtimes and libraries. Kersten specifically mentions generating signatures for:

• Golang runtimes: Constituting 60% of his generated signatures, covering various available runtime versions. This is particularly valuable given Golang's static compilation model, which often results in large binaries with significant runtime code.
• Boost C++ library: Representing 35% of the signatures, covering roughly the last 10 years of Boost versions. Boost is a ubiquitous C++ library, and identifying its components is essential for analyzing many complex applications.
• Other common libraries: Making up the remaining 5%, including lib XML, OpenSSL, and lib G crypt. These libraries are frequently encountered in network-aware or cryptographic applications, and their rapid identification can pinpoint critical functionalities or potential vulnerabilities.

These pre-built databases represent a significant time-saver for reverse engineers, allowing them to instantly apply known function names to a large portion of a new binary. By offloading the identification of common, well-understood code, analysts can immediately direct their attention to the unique user-defined logic, which is often the most critical for understanding a program's purpose or malicious intent. This shift in focus drastically improves efficiency and analytical depth.

Technical Deep Dive

▶ Watch: Script for headless Ghidra Function ID database generation (5:50)

The technical core of Max Kersten's approach relies heavily on Ghidra's advanced analysis capabilities, specifically its Function ID and BSim features, augmented by custom automation scripts. Understanding these components is key to appreciating the methodology's power.

Ghidra's Function ID (FuncID):

Function ID serves as Ghidra's signature-based identification mechanism, conceptually similar to IDA Pro's FLIRT. The fundamental principle is to create a unique fingerprint or hash for each function. This hash is derived from various aspects of the function's structure, such as its instruction patterns, control flow graph, and call sequences, but often excludes variable data (like string literals) that might change between compilations.

1. Signature Generation: When a database is built, Ghidra analyzes a set of known, symbol-rich binaries. For each function in these binaries, it computes a hash and stores a mapping of this hash to the function's original, meaningful name.
2. Signature Application: When analyzing a new, stripped binary, Ghidra computes the Function ID hash for each unidentified function. It then queries its loaded Function ID databases. If a match is found, the generic function name (e.g., FUN_00401234) is automatically replaced with the recognized name from the database (e.g., _ZN5boost6system10error_codeC1Ev for a Boost constructor).

Kersten's automation script for Function ID databases is a critical enhancement. It operates as follows:

• Headless Ingestion: The script leverages Ghidra's headless mode, allowing it to import a large number of binaries without requiring the graphical user interface. This is crucial for batch processing.
• Database Partitioning: For each imported binary, the script identifies its unique architecture (e.g., x86, ARM) and Ghidra Language ID. A Language ID is a more specific identifier that includes details like the processor, bitness (32-bit, 64-bit), endianness, and even compiler specifics (e.g., "x86:LE:32:default:vs2015" for a 32-bit Windows PE file compiled with Visual Studio 2015). The script then creates a separate Function ID database for each unique architecture-Language ID combination found. This partitioning ensures that signatures are applied only to compatible binaries, preventing incorrect matches and improving accuracy.
• Shareability: The resulting Function ID databases are standalone files that can be easily shared among colleagues or within a team. This means that once a comprehensive database for a specific runtime or library is built, it can be reused by anyone, saving immense amounts of duplicated effort.

Ghidra's Behavioral Similarity (BSim):

BSim offers a more advanced, behavioral approach to function identification, complementing the structural matching of Function ID. It was released by the NSA and operates on Ghidra's P-code, an intermediate language designed to be architecture-agnostic.

1. P-code Generation: When Ghidra decompiles a function, it first translates the machine instructions into P-code. P-code represents operations in a standardized, high-level manner, abstracting away architecture-specific details.
2. Vector Calculation: BSim analyzes the P-code of functions to calculate "high P-code vectors" and other metrics. These vectors essentially capture the semantic behavior and data flow patterns of a function. Functions that perform similar operations, even if implemented with different instructions or compiled by different compilers, will tend to have similar P-code vectors.
3. Similarity Matching: BSim compares these vectors to find functions that exhibit behavioral similarity. This is particularly powerful for identifying code overlap when structural hashes (like those used by Function ID) might fail due to minor instruction differences, or for finding heavily optimized or slightly obfuscated code.
4. Application: In Ghidra's interactive environment, BSim is often used for patch diffing, where an analyst can compare the decompiled P-code of two functions side-by-side to highlight differences. Kersten's methodology "shoehorns" BSim for automated symbol recovery by focusing on finding overlaps without necessarily requiring the full interactive project context for the matching binary. This suggests a more streamlined, programmatic querying of BSim results to identify and rename functions based on behavioral similarity to known library/runtime functions.

Integrated Methodology:

The power of Kersten's approach lies in combining and automating these tools. The proposed methodology is:

1. Metadata Identification: First, check for any existing metadata within the binary (e.g., Golang's PCLN tab), which can immediately provide some function names.
2. Runtime and Library Identification: Utilize the automated Function ID and BSim processes to identify known runtime and library functions. This involves:

• Applying pre-built Function ID databases (e.g., for Golang, Boost, OpenSSL).
• Running BSim to find functions with behavioral similarities to known components.
• If a new, recurring library is identified, generate new Function ID signatures for it.

1. Further Logic Application: Once common functions are identified and renamed, apply additional analysis scripts or tools. Kersten mentions the potential use of Large Language Models (LLMs) to assist in renaming remaining unidentified functions based on their context, arguments, and internal logic.
2. Headless Automation: The entire pipeline—from importing a binary, running Function ID, applying BSim, to executing custom renaming scripts—can be performed in a headless Ghidra environment. This allows analysts to set up nightly jobs, processing multiple binaries while they sleep, and waking up to a significantly pre-analyzed and well-labeled project.

This comprehensive approach significantly reduces the initial "blank slate" problem in reverse engineering, allowing analysts to quickly move past boilerplate code and focus on the unique and often malicious aspects of a binary.

Demo / Proof of Concept

▶ Watch: Examples of libraries covered by Function ID databases (6:40)

While the talk did not feature a live, step-by-step demonstration of the scripts in action during the allotted 20 minutes, the speaker outlined the practical application of his automated methodology as a robust proof of concept for efficient symbol recovery. The core of this conceptual demonstration lies in the ability to process a large corpus of binaries and emerge with significantly more readable and annotated Ghidra projects.

The described "demo" is the successful creation and application of Function ID databases for complex and widely used software components. For instance, the generation of databases containing:

• 60% Golang-based signatures: This demonstrates the ability to identify a vast majority of runtime functions in Golang binaries, which are notoriously challenging due to their static compilation and often extensive internal runtime code. The utility here is immediately apparent: an analyst can load a stripped Golang malware sample, apply these signatures, and instantly see functions like runtime.mallocgc, fmt.Println, or net.Dial, allowing them to quickly understand the binary's basic operations without manual effort.
• 35% Boost C++ library signatures: This highlights the capability to tackle complex C++ binaries that heavily rely on Boost. Identifying functions such as boost::system::error_code::message or boost::filesystem::path::operator/= would immediately tell an analyst about error handling or file system interaction, guiding their subsequent analysis.
• 5% Other critical libraries: Signatures for lib XML, OpenSSL, and lib G crypt prove the method's versatility across different types of external dependencies. For example, recognizing SSL_read or XML_Parse functions in a stripped binary would instantly flag network communication or XML parsing capabilities, which are crucial for understanding a binary's interaction with external data or services.

The practical proof of concept is therefore the result of the automated process: a binary that would typically present as a sea of FUN_ addresses is transformed into one populated with hundreds or thousands of meaningful function names. This transformation, achieved through headless execution of Ghidra, Function ID, and BSim, effectively demonstrates that the "no symbols, no problem" claim holds true, enabling analysts to significantly accelerate their initial assessment and deep dive into unknown code.

Defensive Implications

▶ Watch: Step-by-step workflow for effective symbol recovery (7:50)

Max Kersten's methodology for automated symbol recovery has profound implications for defensive security operations, enhancing capabilities across malware analysis, vulnerability research, and threat intelligence.

1. Accelerated Malware Analysis: For malware analysts, the ability to quickly recover symbols for common runtime and library functions is a game-changer. Instead of spending hours or days manually identifying boilerplate code, analysts can apply pre-built signatures and immediately focus on the user code—the unique, often malicious logic implemented by the threat actor. This allows for faster understanding of malware capabilities, identification of command-and-control (C2) mechanisms, encryption routines, or persistence methods. Rapid analysis leads to quicker incident response and more timely deployment of detection rules.
2. Enhanced Vulnerability Research and Patch Diffing: When analyzing proprietary software or seeking zero-day vulnerabilities, symbols are rarely available. This methodology allows researchers to:

• Identify vulnerable library versions: By recovering names for libraries like OpenSSL or lib XML, analysts can quickly determine if a binary links to known vulnerable versions, even if stripped. This is crucial for proactive vulnerability assessments.
• Streamline patch diffing: When comparing a patched and unpatched version of a binary, applying symbol recovery first can label all common, unchanged functions. This highlights the differences (the actual patch) more clearly, allowing researchers to quickly pinpoint security fixes or new vulnerabilities introduced. BSim, in particular, is powerful here for finding behaviorally similar (but slightly changed) functions.

1. Improved Threat Intelligence: The ability to consistently and automatically identify shared code across different malware samples strengthens threat intelligence efforts.

• Malware Family Attribution: If multiple samples share the same unique user code (after common runtime/library functions are filtered out), it's a strong indicator they belong to the same malware family or campaign.
• Tracking Evolving Threats: By tracking the use of specific runtime versions or library components, analysts can observe how threat actors adapt their toolchains, gaining insights into their development practices and operational security.
• Signature Generation: The refined analysis enables the creation of more accurate and specific YARA rules or network signatures that target the unique aspects of malware, rather than generic patterns that might lead to false positives.

1. Increased Efficiency and Reduced Burnout: Reverse engineering is a mentally taxing job. Automating the tedious and repetitive task of symbol identification frees up highly skilled analysts to perform higher-level cognitive tasks. This not only boosts overall team productivity but also helps reduce analyst burnout, allowing security professionals to focus on the truly challenging and novel aspects of their work.
2. Democratization of Advanced Analysis: By providing automated tools and pre-built databases, the methodology lowers the barrier to entry for performing advanced binary analysis. Less experienced analysts can leverage these resources to gain insights that would otherwise require extensive manual effort and expertise.

In essence, Kersten's work transforms a significant reverse engineering bottleneck into an automated, scalable process, directly contributing to more effective and efficient defensive security operations in a world increasingly challenged by sophisticated, stripped binaries.

Key Takeaways

• Symbol recovery is paramount for efficient binary analysis: The absence of function names in stripped binaries drastically hinders understanding, making symbol recovery a critical first step for malware analysis and vulnerability research.
• Ghidra's Function ID and BSim are powerful tools: Function ID provides structural signature matching, while BSim offers advanced behavioral similarity detection using P-code, allowing for robust identification of known code.
• Automation is key to scalability: Max Kersten's scripts enable headless Ghidra execution for batch processing, automating the ingestion of binaries and the generation/application of Function ID databases, significantly reducing manual effort.
• Pre-built databases accelerate analysis: Comprehensive Function ID databases for common runtimes (60% Golang) and libraries (35% Boost, 5% other like OpenSSL) save immense time by immediately labeling known components.
• Focus shifts to unique user code: By automating the identification of boilerplate runtime and library functions, analysts can efficiently isolate and concentrate on the unique, often malicious, user-defined logic within a binary.
• Defensive capabilities are significantly enhanced: This methodology leads to faster malware analysis, more effective vulnerability research (e.g., patch diffing, vulnerable library identification), and improved threat intelligence for attributing and tracking threats.

About the Speaker(s)

Max Kersten, who goes by the nickname 'Libra', is a distinguished member of the Advanced Research team at Trellix. His work primarily involves deep dives into various aspects of malware, encompassing new threats, emerging trends, and any intriguing developments in the cybersecurity landscape. Beyond his research, Kersten is committed to community outreach, actively sharing his findings and methodologies with the broader security community, as exemplified by his presentation at DEF CON 32.