Using Volatility 3 to Detect Sophisticated Malware

Using Volatility 3 to Detect Sophisticated Malware

Speakers: Andrew Case, Director of Research, Vexity

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=cpjP2wius-k

Overview

In this insightful talk from BSides NYC, Andrew Case, Director of Research at Vexity and a core developer of the Volatility memory forensics framework, delved into the critical role of memory forensics in detecting highly sophisticated malware. The presentation highlighted how modern, advanced persistent threats (APTs) and criminal ransomware groups increasingly employ "memory-only" techniques, allowing their payloads to execute without ever touching disk. This evasion strategy renders traditional endpoint detection and response (EDR) solutions and disk-based forensics largely ineffective, leaving organizations blind to active infections.

Case demonstrated how Volatility 3, the latest iteration of the open-source memory forensics framework, provides an unparalleled capability to uncover these elusive threats. By analyzing the full state of a compromised system's RAM, Volatility 3 can identify malicious code injection, process hollowing, and other advanced evasion tactics that leave no persistent artifacts on the hard drive. The talk underscored the urgent need for organizations to integrate memory forensics into their incident response and threat hunting workflows, especially given directives from agencies like CISA that increasingly rely on memory-based indicators of compromise (IOCs).

The significance of this discussion cannot be overstated. As adversaries continue to innovate, security teams must evolve their detection capabilities beyond the file system. Andrew Case's presentation served as a powerful reminder that memory forensics, powered by tools like Volatility 3, is not merely an advanced technique but a fundamental requirement for uncovering and understanding the most dangerous threats operating silently within targeted networks today.

Background

▶ Watch: Introduction: Andrew Case and Volatility's role (0:00)

The landscape of cyber threats has dramatically shifted, with sophisticated actors moving away from traditional disk-based malware. Andrew Case emphasized that many high-impact attacks, including those from nation-state-backed groups and advanced criminal enterprises, now leverage memory-only malware. This class of malware is designed to reside exclusively in a system's volatile memory (RAM), never writing a single byte to disk. Consequently, if an infected machine is rebooted or powered off, the malware is irrevocably lost, along with all forensic evidence. This stark reality underscores why traditional disk forensics, which relies on analyzing persistent storage, is often insufficient for detecting and investigating these advanced threats.

Case drew a crucial distinction between "memory-only" malware and the often-misleading term "fileless malware." While "fileless malware" might avoid traditional executables (.exe or .dll), it often leaves artifacts on disk, such as shortcut files or registry entries, allowing it to persist across reboots. Memory-only malware, however, truly lives only in RAM, making memory forensics the only viable method for detection. This trend is driven by attackers' understanding that if they have stolen credentials, persistence on disk becomes unnecessary; they can simply re-establish access post-reboot using compromised legitimate accounts.

To illustrate the real-world impact of memory-only payloads, Case cited two significant reports:

1. Microsoft's "Cororlump" Report (2022): This report detailed an Austrian mercenary hacker-for-hire group whose main payloads resided "exclusively in memory." Microsoft's deep dive into their tactics, techniques, and procedures (TTPs) led to the group's complete dismantling, highlighting the effectiveness of memory forensics in exposing such operations. The malware facilitated extensive espionage, including microphone and webcam recording, keylogging, and data exfiltration.
2. Google's "Innovative Tactics from AP41" Report (May 2023): Google's Cloud Security team (formerly Google TAG) identified that the nation-state-backed group AP41 was employing "highly innovative" memory-only payloads, with a particular focus on process hollowing. This demonstrated that even cutting-edge attack groups continue to rely on memory-based evasions.

Furthermore, Andrew Case highlighted the increasing relevance of memory forensics in official incident response directives. Since the SolarWinds incident, the Cybersecurity and Infrastructure Security Agency (CISA) has consistently issued emergency directives to federal agencies, often mandating memory capture and forensics as the first step in incident response. Many of the Indicators of Compromise (IOCs) provided in these CISA reports are memory-only, rendering compliance impossible without dedicated memory analysis capabilities.

The Volatility Framework, first released in 2007, has long been the industry standard for open-source memory forensics. However, the original Volatility 2 architecture, designed in a different era of malware, struggled to keep pace with modern evasion techniques. This necessitated a complete rewrite, leading to the development and release of Volatility 3. Volatility 3 has now fully replaced Volatility 2, which was officially deprecated in May 2023. The research motivation behind Volatility 3's development, particularly at Vexity, was to conduct extensive deep dives into the latest Windows, Linux, and Mac threats to ensure the framework could automatically detect all prevalent code injection and memory-only malware in the wild. This was crucial as many techniques detectable by Volatility 2 were subsequently incorporated into EDRs, forcing malware authors to develop even more sophisticated evasions.

The core challenge addressed by Volatility 3 is detecting code injection – the practice of introducing malicious code into a legitimate, long-running process (like svchost.exe or lsass.exe). This technique allows malware to hide its network activity and execution within the noise of a normal system process, making it virtually invisible to live system tools like Task Manager, TCPView, or even many EDRs. For instance, if svchost.exe normally makes dozens of network connections, one extra malicious connection generated by injected code is unlikely to raise alarms without deep memory analysis.

To understand how Volatility 3 uncovers these hidden threats, it's essential to grasp how memory regions within a process are structured and protected. Windows uses Virtual Address Descriptors (VADs) to manage memory regions for each process, ensuring isolation. Each VAD entry provides crucial metadata, including the starting and ending addresses, protection (what access rights apply to the memory), and whether the region maps a file on disk. For legitimate code regions (e.g., a loaded DLL or executable), two attributes are critical:

1. Protection: It must be PAGE_EXECUTE_WRITECOPY. This protection allows the region to be executed, written to (for copy-on-write semantics), but not directly modified by arbitrary writes.
2. File Mapping: It must map a file on disk. If code is running, it should originate from a file (an executable or DLL).

Andrew Case emphasized that these VAD attributes, particularly the protection flags, are extremely difficult for malware to manipulate without causing a system crash (blue screen). While malware can manipulate live system APIs (the data that Task Manager or EDRs see), it cannot reliably alter these low-level kernel data structures without detection or system instability. This fundamental immutability of VADs at the kernel level forms the bedrock of Volatility's robust detection capabilities.

Key Findings

▶ Watch: Understanding Memory Forensics and its importance (2:00)

The talk presented several key findings regarding the detection of sophisticated, memory-only malware using Volatility 3:

• Memory-Only Malware Dominance: Sophisticated adversaries (APTs, ransomware groups) heavily rely on memory-only payloads, which completely bypass disk-based forensics and traditional EDRs. Examples like Microsoft's Cororlump and Google's AP41 reports underscore this prevalent trend.
• Volatility 3 as the Essential Detection Tool: Volatility 3 has been completely re-architected to address modern evasion techniques. It is now the primary open-source framework for detecting these threats, with Volatility 2 officially deprecated.
• VAD-Based Anomaly Detection: The core principle of detection revolves around analyzing Virtual Address Descriptors (VADs). Legitimate code regions in memory must have PAGE_EXECUTE_WRITECOPY protection and be mapped to a file on disk. Any deviation from these characteristics is a strong indicator of malicious code injection or manipulation. Malware cannot reliably manipulate these kernel-level VAD properties without crashing the system.
• Automated Detection of Traditional Injection: Volatility 3's maldetection plugin can automatically detect older, yet still common, code injection techniques such as shellcode injection and reflective DLL injection (e.g., Emotet), which involve executable code or full DLLs loaded directly into memory without disk presence.
• Comprehensive Process Hollowing Detection: The hollow.processes plugin in Volatility 3, built upon years of research, effectively identifies various process hollowing techniques. This includes "V1" hollowing (where a legitimate executable's memory region is overwritten, as seen in Stuxnet and Cobalt Strike) and techniques that manipulate the Image Base Address. It flags processes where the main executable region has incorrect protection (e.g., PAGE_EXECUTE_READWRITE instead of PAGE_EXECUTE_WRITECOPY) or lacks a file mapping.
• Evasion of PE Header Overwriting: Malware can overwrite the Portable Executable (PE) headers (e.g., the MZ signature) to prevent memory scanners from identifying injected code as a PE file. Volatility 3 counters this with the suspicious.threads plugin, which cross-references active threads with their corresponding memory regions. It identifies threads executing in regions with abnormal VAD protections or no file backing, regardless of PE headers.
• Detection of Advanced File System Evasions: Volatility 3 introduces plugins like ghost.processes to detect extremely stealthy techniques such as process ghosting and transaction tampering. These methods allow malware to run from files marked as "deleted" or to present different content to live system tools versus what's actually executing in memory, completely blinding EDRs.
• Zero False Positives for Key Detections: Many of Volatility 3's advanced detection plugins, such as hollow.processes and suspended.threads, have demonstrated a high degree of accuracy with zero false positives when tested against large datasets of both clean and infected memory samples.
• Destruction of Evidence: Rebooting or blindly terminating suspicious processes can lead to the permanent loss of critical evidence, especially with malware that uses "delete on close" mechanisms (e.g., Cobalt Strike). Memory acquisition before any live system interaction is paramount.

Technical Deep Dive

▶ Watch: Distinguishing Memory-Only from Fileless Malware (2:45)

Andrew Case's presentation provided a granular look into the technical mechanisms Volatility 3 employs to detect advanced memory-only malware. The foundation of these detections lies in understanding how Windows manages process memory and how malware attempts to subvert these mechanisms.

Understanding Memory Regions with VAD info

The initial step in memory forensics is to analyze the Virtual Address Descriptors (VADs), which are kernel data structures describing the memory regions allocated to a process. The VAD info plugin in Volatility 3 exposes this critical information. For each memory region within a process (identified by its Process ID, or PID), VAD info outputs its starting and ending addresses, its protection (e.g., PAGE_EXECUTE_READ, PAGE_READWRITE, PAGE_EXECUTE_WRITECOPY), and whether it maps a file on disk.

Crucially, for any legitimate code (executables or DLLs) loaded by the operating system, the associated memory region will consistently exhibit two characteristics:

1. Protection: PAGE_EXECUTE_WRITECOPY. This specific protection allows the kernel to manage copy-on-write semantics for shared pages, ensuring efficiency and integrity.
2. File Mapping: The region will be mapped to a corresponding file on disk. Code, by definition, originates from a binary file.

Andrew Case emphasized that malware cannot arbitrarily set the PAGE_EXECUTE_WRITECOPY protection. Microsoft intentionally restricts this, making it a robust indicator for legitimate code. Any executable memory region that does not possess PAGE_EXECUTE_WRITECOPY protection, or is not backed by a file, is highly suspicious and a strong candidate for malicious injection.

Traditional Code Injection Techniques

Before delving into the most sophisticated evasions, Case reviewed older, but still prevalent, code injection methods:

1. Remote DLL Injection: This technique involves forcing a victim process to load a malicious DLL that exists on disk. While not memory-only, it's a precursor to more advanced methods. Volatility 3's dlllist plugin can enumerate all loaded DLLs, including those injected this way, as they use legitimate loading APIs.
2. Shellcode Injection: Malware injects raw machine instructions (shellcode) directly into a memory region, then executes it. This shellcode is typically not backed by a file and often resides in memory regions with unusual permissions (e.g., PAGE_EXECUTE_READWRITE). Volatility 3's maldetection plugin is designed to identify such anomalous executable regions, flagging them immediately.
3. Reflective DLL Injection: This is a more advanced technique that allows attackers to inject a full-blown DLL into memory without it ever touching disk. A loader component first downloads an encrypted DLL, decrypts it in memory, and then "reflectively loads" it using in-memory parsing rather than standard Windows API calls. This results in a full DLL existing only in RAM, not backed by a disk file. maldetection also catches these. Andrew Case cited Emotet as a prime example, which extensively uses reflective DLL injection into numerous processes, leading to hundreds of maldetection findings with zero false positives on an infected machine. These reflectively loaded DLLs will show PAGE_EXECUTE_READWRITE and no file mapping, directly contradicting legitimate code characteristics.

Process Hollowing

Process hollowing is a sophisticated technique where a legitimate process is started, its original code is "hollowed out," and a malicious payload is injected in its place. Google's TAG group recently highlighted this as "innovative" in AP41 attacks, though variants have existed for over a decade (e.g., Stuxnet).

The typical "V1" process hollowing workflow involves:

1. Creating a Suspended Process: Malware calls CreateProcess (or similar APIs) to start a legitimate executable (e.g., lsass.exe) in a suspended state. Windows creates the process, allocates basic memory regions for the executable, the Process Environment Block (PEB), and critical DLLs (ntdll.dll, kernel32.dll), but then pauses execution.
2. Hollowing Out: The malware gains control, unmaps the legitimate executable's memory region, and then writes its own malicious payload into that same memory space.
3. Resuming Execution: The malware resumes the suspended process. The operating system, unaware of the memory manipulation, believes the legitimate executable is still running, while in reality, the malicious payload is executing.

Live system tools (Task Manager, EDRs) will only see the legitimate process name (lsass.exe) and its expected path, completely missing the malicious payload. Volatility 3, however, detects this via the hollow.processes plugin. This plugin looks for the main executable's memory region (as identified by VADs) and flags it if:

• Its protection is PAGE_EXECUTE_READWRITE (or anything other than PAGE_EXECUTE_WRITECOPY).
• It does not map a file (or maps an incorrect file).

Case showed examples of Stuxnet (from 2011) and a recent Cobalt Strike sample both being detected by hollow.processes for these exact anomalies.

A variation of process hollowing involves Image Base Address Manipulation. Instead of directly overwriting the original executable's memory region, malware might create a new memory region for its payload and then update the process's Image Base Address to point to this new malicious region. The detection for this is remarkably simple in Volatility 3: the kernel maintains a backup copy of the original Image Base Address. If the current Image Base Address (what the process thinks its executable's base is) does not match the kernel's backup (the Section Base Address), it is a 100% certain indicator of process hollowing. The hollow.processes plugin incorporates this check.

Bypassing Protection Checks

Case also addressed potential malware bypasses for VAD-based detections:

• Setting PAGE_EXECUTE_WRITECOPY: As mentioned, Microsoft prevents user-mode applications from directly setting this specific protection, making it a reliable indicator.
• CreateSection / MapViewOfSection: Malware can use these APIs to create memory sections that can have PAGE_EXECUTE_WRITECOPY protection. However, these sections still cannot be associated with a file on disk. Thus, Volatility 3's plugins will still flag them as suspicious due to the lack of file mapping.

PE Header Overwriting and suspicious.threads

A significant challenge for memory scanners is when malware overwrites the PE (Portable Executable) header (specifically the MZ signature) of its injected payload. This makes it impossible for tools scanning for PE file signatures in memory to identify the malicious executable.

To counter this, Volatility 3 leverages its threads plugin, which enumerates all active threads within a process and their current execution addresses. The suspicious.threads plugin, a new addition in Volatility 3, then cross-references these execution addresses with their corresponding VAD entries. It flags threads that are executing in memory regions with:

• Abnormal VAD protections (e.g., PAGE_NOACCESS, PAGE_EXECUTE_READ, PAGE_READWRITE).
• No file backing.

This approach effectively detects injected code even when PE headers are tampered with, as it focuses on where code is executing and the underlying memory protections, rather than relying on file signatures. Case presented an example where suspicious.threads flagged a thread executing in a PAGE_NOACCESS region, a clear sign of manipulation.

Process Ghosting and Transaction Tampering

These are arguably the most advanced evasions discussed, demonstrating deep abuse of Windows internals:

• Process Ghosting: Malware creates a malicious executable on disk, but immediately marks it as deleted using specific Windows APIs. It then starts a process from this "deleted" file. From a live system perspective, an EDR trying to read the file path will be told by Windows that the file is deleted, even though a process is actively running from it. The ghost.processes plugin in Volatility 3 detects this by identifying processes where the underlying executable file has the delete pending bit set. Case showed an example flagging a kelp.exe process with a faked path and the delete pending flag.
• Transaction Tampering: This technique involves manipulating file system transactions to ensure that the contents of an executable mapped into memory are entirely different from what an EDR would read from the disk. This creates a deceptive façade where the on-disk file appears benign, while the in-memory executable is malicious.

A critical implication of these techniques, especially for incident responders, is the use of "delete on close" functionality. Malware like Cobalt Strike can configure its executable to be deleted from disk automatically when its process terminates. If an analyst blindly shuts down or terminates a suspicious process (e.g., via Task Manager), they inadvertently instruct Windows to delete the primary evidence, leaving nothing on disk to analyze. Volatility 3's ghost.processes plugin can identify such processes before termination, allowing the analyst to dump the malicious executable from memory and preserve the evidence.

Demo / Proof of Concept

▶ Watch: Case Study: Microsoft Cororlump's memory-only payloads (4:10)

While the talk did not feature a live demo, Andrew Case effectively illustrated Volatility 3's capabilities through numerous screenshots of its plugin outputs against real-world and simulated malware samples. These visual proofs demonstrated how specific plugins automatically identify the anomalies discussed in the technical deep dive:

• maldetection: Screenshots showed maldetection output clearly identifying reflectively injected DLLs (e.g., from Emotet) and raw shellcode blocks by highlighting memory regions with PAGE_EXECUTE_READWRITE permissions and no file backing.
• hollow.processes: Examples for Stuxnet (from 2011) and recent Cobalt Strike samples displayed hollow.processes flagging legitimate-looking processes (lsass.exe) where the main executable region had incorrect VAD protections (PAGE_EXECUTE_READWRITE instead of PAGE_EXECUTE_WRITECOPY) and lacked a proper file mapping. It also showed detection of Image Base Address manipulation where the section base address did not match the image base address.
• suspicious.threads: Output demonstrated the plugin successfully identifying threads executing in memory regions with highly abnormal protections like PAGE_NOACCESS or PAGE_EXECUTE_READ that were not backed by files, proving its efficacy even against PE header overwriting.
• ghost.processes: A screenshot revealed ghost.processes detecting a process (kelp.exe) running from a file with the delete pending bit set, illustrating the detection of process ghosting. This was explicitly linked to how Cobalt Strike can use "delete on close" to remove its executable upon process termination.

Case also recommended the Offensive Panda GitHub repository (specifically their process-injection-techniques repo) as an excellent resource for security researchers and defenders. This repository contains fully functional, open-source code for dozens of process injection techniques, allowing users to compile and test these methods against their own EDRs and memory forensics tools. This resource serves as a practical "proof of concept" environment for those wishing to explore these techniques hands-on and validate Volatility 3's detection capabilities. The Forinet blog post, "A Deep Dive into a Dumped Malware without a PE Header," was also cited as an example of manual analysis tackling the same problem that suspicious.threads automates.

Defensive Implications

▶ Watch: Case Study: Google's APT41 and process hollowing (6:05)

The insights shared by Andrew Case carry profound implications for defensive security strategies, particularly for organizations facing sophisticated and targeted attacks:

1. Prioritize Memory Acquisition: The most crucial takeaway is the absolute necessity of integrating memory acquisition into incident response plans. Given that sophisticated malware is increasingly memory-only and can employ "delete on close" tactics, rebooting or terminating a suspicious process without first capturing memory guarantees the loss of critical evidence. CISA's directives already mandate this, and all organizations should follow suit.
2. Adopt Volatility 3 Immediately: With Volatility 2 officially deprecated, organizations must migrate to Volatility 3. It is the only open-source framework equipped with the advanced plugins and architecture required to detect modern, evasive memory-only threats. Its availability on GitHub means it can be deployed and utilized by any security team.
3. Automate Memory Analysis: Manual analysis of memory dumps is time-consuming. As discussed in the Q&A, Volatility 3 supports various output formats (JSON, CSV, JSONL), making it highly suitable for automation. Security Operations Centers (SOCs) and Managed Security Service Providers (MSSPs) should develop pipelines to automatically ingest memory samples (potentially terabytes of data), process them with Volatility 3 plugins, and feed the structured output into SIEMs, SOAR platforms, or custom detection engineering systems. This allows for scalable threat hunting and rapid detection.
4. Enhance Threat Hunting Capabilities: Volatility 3's specialized plugins (hollow.processes, maldetection, suspicious.threads, ghost.processes, suspended.threads) are powerful tools for proactive threat hunting. Regularly analyzing memory dumps from critical systems (e.g., domain controllers, high-value endpoints) can uncover active infections that have bypassed other security layers.
5. Understand EDR Limitations: Recognize that EDRs, while essential, have inherent limitations against kernel-level evasions and memory-only payloads. EDRs often rely on live system APIs and disk-based telemetry, which malware can manipulate or avoid entirely. Memory forensics complements EDRs by providing visibility into the "blind spots" where advanced threats operate.
6. Invest in Training and Expertise: Effectively utilizing Volatility 3 requires specialized knowledge of Windows internals, malware techniques, and memory forensics principles. Organizations should invest in training for their incident responders and threat hunters to build this critical expertise. Resources like the Offensive Panda GitHub repo can facilitate hands-on learning.
7. Do Not Terminate Blindly: Emphasize to incident responders that terminating a suspicious process, especially one identified as using advanced evasions, should never be the first step. Always acquire a memory dump first to preserve the executable, configuration, and other artifacts of the malicious payload.

By integrating these defensive strategies, organizations can significantly enhance their capability to detect, investigate, and respond to the most sophisticated and stealthy cyber threats that leverage memory-only techniques.

Key Takeaways

• Memory-only malware is a prevalent and critical threat that bypasses traditional disk-based forensics and many EDR solutions, making memory forensics indispensable for modern incident response.
• Volatility 3 is the essential open-source framework for detecting these advanced threats, offering robust and automated analysis capabilities for complex evasion techniques, with Volatility 2 now deprecated.
• Detection fundamentally relies on analyzing Virtual Address Descriptors (VADs) for anomalous memory region protections (e.g., anything other than PAGE_EXECUTE_WRITECOPY for code) and the absence of legitimate file mappings.
• Specialized Volatility 3 plugins such as hollow.processes, maldetection, suspicious.threads, and ghost.processes automate the identification of code injection, process hollowing, PE header overwriting, and advanced file system evasions like process ghosting.
• Live system tools and EDRs are often blind to sophisticated memory-only techniques, as malware manipulates or avoids the APIs they rely on, highlighting the unique value of deep memory analysis.
• Always perform memory acquisition before any system interaction (rebooting, terminating processes) to prevent the irreversible loss of critical evidence, especially when dealing with malware that employs "delete on close" mechanisms.

About the Speaker(s)

Andrew Case is the Director of Research at Vexity, a company focused on monitoring, incident response, and in-depth analysis of highly targeted networks. In this role, he has extensive experience with the latest sophisticated malware, toolkits, and ransomware used by advanced threat actors. Beyond his work at Vexity, Andrew is a long-standing core developer of the Volatility Framework, having been involved with the project since approximately 2010 or 2011. His unique blend of real-world incident response experience and deep technical expertise in memory forensics positions him as a leading authority on detecting advanced cyber threats.