RE-Thinking: Modernizing the Malware Analyst

RE-Thinking: Modernizing the Malware Analyst

Speakers: Joseph Edwards, Incident Responder / Forensic Tools Developer, SentinelOne

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=YkU9m2aYKJ4

Overview

Joseph Edwards' talk, "RE-Thinking: Modernizing the Malware Analyst," delves into the evolving landscape of malware analysis, highlighting critical gaps in current educational approaches and professional practices. Edwards, an Incident Responder and Forensic Tools Developer at SentinelOne, draws upon his five to six years of experience to advocate for a significant shift in how analysts are trained and equipped to handle sophisticated threats. The core of his presentation centers on the need for greater flexibility, customizability, and deep system introspection in malware analysis tools, moving beyond traditional, GUI-centric methods.

The talk addresses the increasing challenges posed by modern malware, which often employs advanced evasion techniques, targets diverse platforms, and utilizes complex programming languages. Edwards argues that relying solely on commercial sandboxes or static analysis tools is no longer sufficient. Instead, he champions the adoption of malleable sandboxes and dynamic binary instrumentation (DBI) frameworks, empowering analysts to adapt their environments and tools to counter specific evasion tactics. This modernization is crucial for improving the speed and accuracy of incident response, enabling analysts to effectively dissect and understand even the most elusive samples.

Ultimately, Edwards' presentation serves as a call to action for the security community to rethink its approach to malware analysis education and tooling. By fostering a deeper understanding of underlying system interactions and promoting hands-on customization, the industry can better prepare the next generation of analysts to tackle the sophisticated threats of today and tomorrow. The insights shared are particularly relevant for incident responders, forensic investigators, and security researchers who regularly encounter novel and evasive malware.

Background

▶ Watch: Introduction, talk motivation, and speaker background. (0:00)

The foundation of modern malware analysis education, as Edwards points out, often rests on seminal works like Practical Malware Analysis (2012) and Malware Analyst's Cookbook and DVD (2010). While Practical Malware Analysis is widely lauded for its accessibility, its emphasis on GUI-based tools like Redshot, Autoruns, and ProcMon can inadvertently foster a mindset that struggles with programmatic, flexible analysis. In contrast, the Cookbook promotes Python scripting and framework modification, an approach Edwards believes is essential for contemporary analysts. The difficulty, he notes, lies in breaking out of the GUI-centric mindset once it's established, making the transition to professional tools like IDA Pro or Binary Ninja, which often require scripting for advanced tasks, less seamless.

Professionally, the demands placed on malware analysts typically fall into three tiers:

1. Indicators of Compromise (IOCs): Basic information like IP addresses, persistence mechanisms, or injected processes, often obtainable from well-configured sandboxes.
2. Behavioral Inspection: Required when malware detects and evades sandboxes, targets specific environments (e.g., domain-joined machines), or exhibits conditional execution. This demands a deeper look into its interactions.
3. Capability Analysis: The highest tier, involving comprehensive understanding of all malware functions, often requiring disassemblers and decompilers. Tools like Mandiant's CAPA can assist by scanning for API patterns and constants indicative of specific capabilities (e.g., RC4 encryption, networking).

Edwards focuses on behavioral inspection as a key area for rapid learning, noting that while capability analysis is critical, it demands extensive reverse engineering skills that take considerable time to develop. The existing landscape of tools, including commercial sandboxes, often falls short in providing the necessary customizability and comprehensive system interaction visibility to effectively handle evasive malware. Enterprise sandboxes, while offering excellent file type support and broad monitoring, are typically not customizable enough to bypass specific sandbox detection checks, forcing analysts to wait for vendor updates or resort to more manual, time-consuming methods during critical incident response scenarios.

Key Findings

▶ Watch: Foundational books and their distinct approaches to learning. (2:20)

Edwards' talk underscores several critical findings regarding the current state and future direction of malware analysis:

• Outdated Educational Paradigms: Current educational approaches, heavily reliant on older, GUI-based tools, fail to adequately prepare analysts for the complexities of modern malware and the need for programmatic, flexible analysis. This creates a disconnect between foundational learning and professional practice.
• Limitations of Emulation for Full Binaries: While emulators are highly effective for analyzing isolated functions or shellcode due to their safety and reset capabilities, they generally struggle with full Windows binaries. This is primarily because many emulation frameworks lack comprehensive implementations of the vast array of Windows API calls and low-level system calls (syscalls), leading to incomplete execution and analysis gaps.
• Power of Dynamic Binary Instrumentation (DBI): DBI frameworks offer a more practical and efficient approach for behavioral inspection of complex binaries. They provide extensive system interaction coverage and are highly customizable, allowing analysts to monitor and even modify a program's execution on the fly to bypass sandbox detections or deobfuscate scripts with less effort than implementing full OS emulation.
• The Ascendancy of Malleable Sandboxes: For production environments and effective incident response, malleable sandboxes are presented as the ideal solution. These user-controlled, highly customizable sandboxes allow analysts to spoof environmental characteristics (e.g., memory, CPU details, graphics cards) to trick evasive malware into executing fully, providing critical insights that commercial, black-box sandboxes often miss.
• Challenges of Modern Malware Development: New programming languages like Go and Rust, coupled with increasingly sophisticated obfuscation techniques (e.g., PyArmor, virtualized binaries like VMProtect), pose significant hurdles for traditional static analysis. These modern binaries are often large, statically linked, and designed to thwart easy reverse engineering, necessitating dynamic analysis and instrumentation.
• Prioritizing a Holistic Pipeline: An ideal malware analysis pipeline integrates initial triage, static or emulated unpacking for known families, and then conditional sandbox detonation or deep behavioral inspection using DBI for evasive or unknown samples. This tiered approach maximizes efficiency and ensures comprehensive coverage.

Technical Deep Dive

▶ Watch: The challenge of transitioning from GUI to flexible tooling. (4:25)

Edwards outlines three crucial criteria for evaluating tools suitable for behavioral inspection: comprehensive system interactions, broad file type support, and high customizability. He then explores various tool categories, detailing their strengths, weaknesses, and how they address these criteria.

Emulation Frameworks

Emulation involves imitating another program or device, faking responses to malware as if a real operating system were present, without actually executing the code. This makes it inherently safer and easier to reset for analysis. Edwards discusses several prominent emulators:

• Jilin: Supports Windows, Linux, and UEFI binaries. However, its Windows support is noted as insufficient for complex binaries, often failing to emulate them fully.
• Speakeasy: Developed by Mandiant, this framework excels at shellcode emulation and can successfully emulate some simpler binaries. Its primary strength lies in its ability to portably decrypt data from known encrypted/decrypted function pairs without needing to understand the underlying encryption algorithm.
• Limitations of Emulation: A significant hurdle for emulators is the sheer volume of Windows API calls and low-level syscalls that need to be implemented. Edwards illustrates this with an example where a Sliver implant stager (a Go binary) fails to fully emulate in Jilin because core syscalls like ZW allocate virtual memory are not implemented by default. This necessitates extensive custom code to "fake" every API call, making full binary emulation a monumental task. The framework Dumpulator, by the author of x64dbg, attempts to circumvent this by focusing on implementing syscalls, which are fewer in number than user-mode APIs, thereby achieving broader coverage. However, syscall arguments are more complex and vary across Windows versions, presenting an ongoing challenge.

Dynamic Binary Instrumentation (DBI)

DBI involves modifying a program's instructions as it executes. This allows for deep monitoring of behavior, removal of analysis obstacles, or bypassing detections on the fly.

• Dynamo Rio / DR Memory: An older but robust tool for instrumenting and tracing syscalls. A notable caveat is its frequent requests for symbols from the internet, which can be problematic in secure analysis environments.
• Frida: Highlighted as a powerful and promising tool, despite requiring JavaScript for injection. Edwards' coworker, Adam, successfully uses Frida to trace and deobfuscate scripts (e.g., VBScript executed by Wscript.exe or installers handled by msiexec.exe) by hooking functions dynamically. This capability is crucial for addressing obfuscated scripts that are difficult to analyze statically.
• Advantages of DBI: DBI offers high customizability and excellent coverage of system interactions. While it requires executing the malware (necessitating VM resets), it provides quick solutions to specific problems with relatively less effort compared to full emulation. Edwards cites Go Function Trace (GF trace) as an example, which hooks a specific gateway function in Go binaries to trace all Windows API calls, efficiently revealing critical information like command-and-control (C2) addresses.

Malleable Sandboxes

Edwards posits that for production environments, malleable sandboxes are the most ideal solution. These are user-controlled sandboxes that allow for deep customization to counter evasive malware.

• Cape V2: A robust and well-established framework, though it requires some involvement to set up.
• Drakvuf Sandbox: Developed by CERT Polska, this framework is highly modifiable and is also associated with tools for automated unpacking and configuration extraction.

These sandboxes address the customizability gap of enterprise solutions, allowing analysts to adjust parameters like reported memory size or CPU names to bypass malware's environmental checks.

Challenges in Modern Malware Analysis

Edwards also details several significant challenges facing analysts:

• Golang Binaries: These are exceptionally large due to static linking of numerous libraries, making manual reverse engineering inefficient as analysts often spend 90% of their time in library code.
• Unfriendly Compilers: Compilers for languages like Rust and Go are constantly evolving, leading to control flow patterns that can deviate significantly from traditional C/C++ binaries, complicating analysis.
• Obfuscation and Interpreted Languages: Malware frequently uses obfuscated VBScript, JavaScript, JScript, or Python (e.g., PyArmor), often chained together or relying on interpreters present on target systems. Statically deobfuscating these is often inefficient; dynamic instrumentation is more effective.
• Virtualized Binaries: Tools like VMProtect create custom virtual machines within binaries, interpreting assembly rather than executing it directly. This effectively neutralizes traditional reverse engineering skills until the sample is de-virtualized, a task that often requires specialized expertise and significant time, which most incident responders lack.
• MSI Installers: These can contain complex logic and sandbox evasion techniques.
• Linux Malware: There's a notable lack of sophisticated instrumentation and debugging tools for Linux malware compared to Windows. Edwards recounts an incident where a bug in Evans Debugger (part of REMnux) hampered memory dumping, highlighting the limited options available.

Demo / Proof of Concept

▶ Watch: Categorizing professional malware analysis tasks and expectations. (6:00)

While Joseph Edwards did not conduct a live demonstration during his talk, he effectively illustrated his points through two contrasting screenshots that served as a powerful proof of concept. These visual aids highlighted the limitations of current emulation frameworks versus the practical utility of dynamic instrumentation for specific analysis tasks.

The first screenshot depicted an attempt to emulate a packed Sliver implant stager, a Go binary, using Jilin. The image, though difficult to read in detail, visually represented the output from Jilin's emulation process. Edwards explained that this specific stager relied on the VirtualAllocEx API, which in turn calls the low-level Windows syscall ZW allocate virtual memory, for unpacking. Critically, Jilin, by default, did not implement these fundamental syscalls. As a result, the emulation failed to fully execute or provide meaningful arguments for the API calls, preventing a complete understanding of the malware's unpacking behavior. This demonstrated a core limitation of many emulation frameworks: their inability to comprehensively cover all necessary operating system interactions, especially for complex, real-world binaries.

In stark contrast, the second screenshot showcased the successful analysis of the same Sliver implant stager using Go Function Trace (GF trace), a tool based on dynamic binary instrumentation. This trace clearly displayed a sequence of API calls, culminating in the revelation of the command-and-control (C2) address used by the Sliver implant. Edwards emphasized that GF trace achieved this by hooking a specific "gate" function that all Windows API calls in Go binaries pass through. This demonstrated that with targeted instrumentation, significant value could be gained with considerably less effort than attempting full-system emulation. Although DBI involves executing the malware in a controlled environment (requiring VM resets), its ability to quickly provide critical insights for specific problems makes it a highly practical approach for incident responders.

These examples collectively reinforced Edwards' argument that while emulation has its place for single functions or shellcode, malleable sandboxes integrated with robust DBI capabilities are essential for scalable and effective analysis of complex, evasive malware in a production setting.

Defensive Implications

▶ Watch: Behavioral inspection: the best way to learn quickly. (7:40)

The insights shared by Joseph Edwards provide several critical defensive implications for organizations and malware analysts:

1. Invest in Malleable Sandbox Capabilities: Relying solely on enterprise cloud sandboxes is insufficient against modern evasive malware. Organizations should establish or invest in malleable sandboxes (e.g., Cape V2, Drakvuf Sandbox) within their own infrastructure. This allows for granular customization of the analysis environment, enabling analysts to spoof system parameters (memory, CPU, graphics card details) that malware often checks to detect sandboxes, ensuring full execution and comprehensive behavioral data.
2. Embrace Dynamic Binary Instrumentation (DBI): Defenders should integrate DBI frameworks like Frida into their analysis toolkit. These tools offer unparalleled flexibility to hook functions, deobfuscate scripts (VBScript, JavaScript, PyArmor), and observe malware behavior at a deep level. Learning to leverage DBI can significantly reduce the time spent on static analysis of complex, obfuscated code and bypass sandbox detection mechanisms on the fly.
3. Modernize Malware Analysis Education: The industry needs to shift its educational focus from purely GUI-based tools to hands-on scripting, framework development, and deep system introspection. Training programs should emphasize how to modify and extend tools, understand low-level API and syscall interactions, and develop custom analysis scripts to adapt to evolving threats. Open security training resources, such as those for assembly and reverse engineering, should be regularly referenced.
4. Prioritize Contextual Triage: Before detonating any sample, thorough triage is essential. This involves identifying all related files, understanding the execution context, and determining necessary command-line arguments. For EDR/antivirus companies, this context might be readily available, but for others, it may require initial reverse engineering to ensure successful and complete execution within the sandbox.
5. Automate Unpacking and Configuration Extraction: For known malware families or repeatedly encountered packers (like UPX), prioritize static or emulated unpacking. Developing custom Python scripts or leveraging tools that can emulate unpacking functions to dump decrypted code from memory is significantly faster than full sandbox detonation and can quickly yield IOCs.
6. Understand and Prepare for Evasion Techniques: Defenders must stay abreast of common and emerging malware evasion techniques, including checks for virtualized environments, specific hardware configurations, or domain membership. Knowing these helps in configuring malleable sandboxes effectively and developing targeted DBI scripts to circumvent them.
7. Address Linux Malware Tooling Gaps: For organizations with significant Linux infrastructure, there's a need to bridge the tooling gap for Linux malware analysis. This may involve investing in specialized Linux-focused sandboxes, exploring cross-platform DBI tools, or contributing to open-source debugging and instrumentation projects to enhance capabilities.

By adopting these defensive strategies, organizations can build a more resilient and adaptable malware analysis capability, better equipped to counter the sophisticated and evasive threats prevalent in today's landscape.

Key Takeaways

• Modern malware analysis demands a shift from static, GUI-centric tools to flexible, programmable approaches that leverage scripting and frameworks.
• Malleable sandboxes, controllable and customizable by analysts, are crucial for effectively handling increasingly evasive malware in production environments.
• Dynamic Binary Instrumentation (DBI) frameworks like Frida offer powerful, customizable methods to observe, manipulate, and deobfuscate malware behavior with less effort for specific problems.
• Traditional emulation frameworks often struggle with full Windows binaries due to incomplete implementations of the vast number of OS APIs and syscalls, limiting their utility beyond isolated functions or shellcode.
• Newer programming languages (Go, Rust), advanced obfuscation techniques (PyArmor), and virtualized binaries (VMProtect) pose significant challenges to conventional static analysis, necessitating dynamic methods.
• A holistic malware analysis pipeline should prioritize initial triage, efficient static/emulated unpacking, and then conditional, deep behavioral inspection via malleable sandboxes or DBI for evasive samples.

About the Speaker(s)

Joseph Edwards is an Incident Responder and Forensic Tools Developer at SentinelOne. With over five years of experience in malware analysis, he has spent the last two years as an incident responder, during which he gained extensive experience supporting teams by analyzing various types of malware, including those that tamper with EDR products or utilize vulnerable drivers. He is currently transitioning into a role focused on forensic tools and automation, drawing on frameworks like Dissect. Edwards holds the GCFE certification, crediting the SANS Forensic 500 class taught by Rob Lee for drawing him into this line of work. His expertise lies in developing flexible, programmable solutions to enhance malware analysis capabilities.