Anti-Forensics - You are doing it wrong (Believe me, I'm an IR consultant)

Anti-Forensics - You are doing it wrong (Believe me, I'm an IR consultant)

Speakers: Stephan Berger, IR Team Lead, InfoGuard

Conference: Security Fest

YouTube: https://www.youtube.com/watch?v=Znl7TBFAz9E

Overview

In this insightful talk from Security Fest, Stephan Berger, an Incident Response Team Lead at InfoGuard, challenges the common assumptions surrounding anti-forensic techniques. Titled "Anti-Forensics - You are doing it wrong (Believe me, I'm an IR consultant)," Berger dismantles popular methods used by attackers to erase their tracks, revealing how often these attempts leave behind a wealth of forensic evidence for skilled defenders. The presentation serves as a critical wake-up call for red teamers and a powerful affirmation for blue teamers, demonstrating that even sophisticated adversaries frequently fail to achieve true stealth.

Berger's core message is that while attackers strive to cover their tracks, a deep understanding of operating system internals and advanced forensic tools can uncover even the most determined anti-forensic efforts. He illustrates this through a series of practical examples across both Windows and Linux environments, highlighting the persistent nature of digital evidence. This talk is crucial for anyone involved in cybersecurity, from penetration testers seeking to refine their stealth capabilities to incident responders aiming to enhance their detection and recovery strategies.

The presentation is structured to show a continuous cat-and-mouse game, with sections dedicated to common anti-forensic techniques, how incident responders detect them, and then how attackers might attempt to improve their methods. Berger's real-world experience as an IR consultant lends significant credibility to his assertions, providing actionable intelligence on how to both identify and thwart anti-forensic maneuvers.

Background

▶ Watch: Introduction and talk structure (0:00)

The concept of anti-forensics is as old as digital forensics itself, with early discussions dating back decades. Stephan Berger references a paper titled "How to Cover Your Tracks," published in 1995, noting that some techniques described within it, such as clearing log files on Linux, are still surprisingly prevalent among attackers today. This historical context underscores a fundamental challenge in cybersecurity: defenders often lag behind attackers, and outdated or easily detectable methods persist in the wild, even among advanced persistent threats (APTs).

The problem, as Berger articulates, is compounded by the public availability of tools designed for anti-forensics. He cites Forenzia, an open-source tool on GitHub, which purports to cover or delete many tracks incident responders use. However, Berger asserts that such tools offer only a subset of necessary techniques and are often insufficient against skilled forensicators. He embraces a quote often attributed to Rob Lee, a renowned SANS instructor, suggesting that anti-forensic tools can "test the capabilities of your incident response forensic team," a challenge Berger readily accepts throughout his presentation.

Common anti-forensic techniques frequently discussed include deleting registry keys, time stomping, and file deletion. Berger's talk aims to dissect these methods, demonstrating their inherent weaknesses and how blue teams can effectively reverse or circumvent them. This battleground between hiding and finding traces forms the bedrock of modern incident response, making a detailed understanding of both sides essential for effective defense.

Key Findings

▶ Watch: Public anti-forensics tools like Forenzia (2:00)

Stephan Berger's talk delivers several critical findings that challenge conventional wisdom regarding anti-forensics and empower incident responders:

1. File Deletion is Rarely Absolute: Simply deleting a file, even from the recycle bin, does not remove it from the file system. Traces persist in the NTFS Master File Table (MFT) on Windows, allowing for recovery of critical data like configuration files and even plain-text passwords.
2. Specialized Deletion Tools Leave Traces: Tools like sdelete, designed to securely delete files by overwriting them, paradoxically leave their own distinct forensic artifacts, making their usage detectable. Attackers attempting to replicate this functionality with custom PowerShell scripts may avoid antivirus detection but still generate other identifiable behaviors.
3. Operating Systems Log Everything: Windows' Update Sequence Number (USN) journal and System Resource Usage Monitor (SRAMM), along with Linux's journald and various log files, create a comprehensive record of system activity. Even when attackers attempt to disable or clear these logs, the act of tampering often generates its own log entries, creating a new forensic artifact.
4. Memory Forensics is a Powerful Countermeasure: Techniques like disabling PowerShell console history or clearing bash history are ineffective against memory forensics. Attackers' commands and activities often reside in system memory, accessible to tools like Volatility or custom scripts, even after on-disk traces are purged.
5. Time Stomping is Detectable: The manipulation of file timestamps, a classic anti-forensic technique, can be identified by inconsistencies between different timestamp types (e.g., MFT timestamps vs. file name timestamps) or by specific patterns like all-zero fraction seconds. Tools like Velociraptor have specific artifacts to detect this.
6. Advanced Linux Anti-Forensics Still Fails: Even a seemingly clever technique involving the Linux dynamic loader (LD_PRELOAD) to hide malicious executables in /proc directories can be countered by dumping and reconstructing memory regions, revealing the true binary.
7. ChatGPT is Not a Substitute for Research: Berger demonstrated that relying solely on AI tools like ChatGPT for anti-forensic code can lead to incorrect assumptions about system internals (e.g., database table names and data formats in SRAMM), highlighting the continued need for deep technical expertise.

These findings collectively underscore the difficulty for attackers to achieve true stealth and emphasize the robust capabilities available to incident responders armed with a thorough understanding of system forensics and advanced tooling.

Technical Deep Dive

▶ Watch: Rob Lee's take on anti-forensics effectiveness (3:00)

Stephan Berger meticulously dissects various anti-forensic techniques across Windows and Linux, demonstrating how they are often flawed and detectable. He structures his analysis by first presenting an attacker's anti-forensic method, then showing how defenders can detect it, and finally, sometimes, how attackers might attempt to improve their technique.

Windows Anti-Forensics and Countermeasures

1. File Deletion and the NTFS Master File Table (MFT)

• Anti-Forensic Technique: Attackers simply delete files, often from the recycle bin, believing them to be permanently gone. Berger uses the example of arclone.com, a configuration file for rclone, a tool frequently used for data exfiltration.
• Defensive Countermeasure: Windows does not immediately overwrite deleted files. Instead, it marks their entries in the NTFS MFT as free. Forensic tools like Velociraptor can query the MFT for these marked entries. Berger demonstrates recovering the arclone.com file, which contained a clear-text password to an attacker's server, enabling law enforcement to intervene. The MFT ID acts as a unique identifier for every file.
• Improved Attacker Technique: Attackers might use tools like sdelete (System Internal's Secure Delete) to overwrite file contents multiple times and rename files, making recovery more difficult. However, sdelete itself leaves traces in the MFT and the journal, and is often flagged by antivirus.
• Stealthier Attacker Technique: Berger shows a simple PowerShell script that manually renames a file multiple times and pumps random content into it. This mimics sdelete's functionality without using a flagged tool, but still leaves traces and is not foolproof.

1. The USN Journal (Update Sequence Number Journal)

• Purpose: The USN journal tracks every change on the file system, including file creation, renaming, and service creation. Berger presents a real-world example where a malicious PNG file was used to create a service, and the USN journal precisely pinpointed the file's creation time, even if the file was later deleted.
• Anti-Forensic Technique: An attacker can disable the USN journal using fsutil usn delete journal. This command effectively removes all traces of file system changes from the journal.
• Defensive Countermeasure: While this command clears the journal, it's a significant system modification. Berger suggests monitoring for this specific command via EDR or SIEM, noting that Elastic Security provides a detection rule for this behavior.

1. System Resource Usage Monitor (SRAMM)

• Purpose: The SRAMM database tracks system and application resource usage, providing evidence of execution even when MFT and USN journal entries for a file are absent. Berger shows an instance where a malicious right86.exe file left no MFT or USN traces but was clearly recorded in SRAMM.
• Anti-Forensic Technique: Attackers can stop the SRAMM service (sc stop DPS). A more sophisticated attacker might attempt to selectively clean records from the SRAMM database.
• Attacker's Challenge (and ChatGPT's Failure): Berger illustrates the difficulty of tampering with SRAMM by showing how ChatGPT's suggested PowerShell code to delete specific records failed. ChatGPT incorrectly assumed table names and plain-text storage within the database, whereas SRAMM's internal structure uses different names and hexadecimal encoding. This highlights that AI assistance still requires deep technical understanding for effective anti-forensics.

1. PowerShell Logging

• Purpose: PowerShell console history logs every command entered by a user in a PowerShell console, providing a rich forensic artifact. Berger presents a real incident where an attacker's entire data exfiltration sequence was recovered from this history.
• Anti-Forensic Technique: Attackers can change the path of the log file, disable logging (seps red line option save nothing), or simply delete the console history file. Another technique, used by the Russian APT group TLA, involves disabling Windows Event Tracing entirely with a few lines of PowerShell.
• Defensive Countermeasure: Monitoring for commands that disable logging or delete history files is crucial. Elastic Security offers detection rules for the deletion of console_history files. Additionally, PowerShell transcripts and other logging mechanisms may still capture activity.

1. Time Stomping

• Concept: Rob Lee identified two tell-tale signs of time stomping: misaligned file timestamps (e.g., MFT timestamps not matching file name timestamps) and all-zero fraction seconds in timestamps. Windows maintains multiple timestamps for a file (creation, modification, access, entry modified in MFT). Attackers typically manipulate only the file name timestamps via API calls.
• Defensive Countermeasure: Tools like Velociraptor have a "suspect time stomping" feature that automatically flags files where these timestamps don't align. Berger shows a test case where a file's birth time (MFT metadata) was two years older than its time-stomped file time, triggering Velociraptor's detection.
• Attacker's "Trick": Simply renaming a file and then renaming it back can update the file name timestamp, making it appear more consistent, but this still doesn't affect the underlying MFT timestamps.

Linux Anti-Forensics and Countermeasures

1. Kernel Ring Buffer (dmesg)

• Anti-Forensic Technique: Attackers clear the kernel ring buffer using dmesg -c. Mandiant identified this as a technique used by APTs, specifically after loading malicious kernel modules.
• Defensive Countermeasure: Even if dmesg is cleared, the same information is often persisted in /var/log/kern.log.
• Improved Attacker Technique: Attackers might then delete /var/log/kern.log.
• Further Defensive Countermeasure: Modern Linux systems use journald, which logs system messages persistently. Even if /var/log/kern.log is deleted, the information remains in the journal.

1. Clearing Log Files (auth.log, syslog, etc.)

• Anti-Forensic Technique: Mandiant's "Steel Corgi" APT group used a tool called bleach to clear various log files like auth.log, which tracks user logins. They believed this made incident response difficult.
• Defensive Countermeasure: Again, journald often retains these login records (IP address, username, acceptance/denial) even after the traditional log files are purged.

1. Clearing the Journal (journalctl)

• Anti-Forensic Technique: A more advanced attacker might attempt to clear the journal using journalctl --rotate.
• Defensive Countermeasure: The act of rotating or clearing the journal itself generates a log entry within the journal, indicating that tampering occurred. This becomes a new red flag for defenders. Berger notes that Elastic Security currently lacks a specific rule for this, highlighting a potential gap.

1. Bash History (.bash_history)

• Anti-Forensic Technique: Attackers can disable bash history logging by unsetting HISTFILE or redirecting it to /dev/null (unset HISTFILE, export HISTFILE=/dev/null). This prevents commands from being written to disk.
• Defensive Countermeasure: Despite these measures, commands entered into the shell are still stored in memory until the user logs out. Memory forensics tools, such as Volatility with its bash plugin, can dump server memory and extract the entire command history. Berger proudly recounts how a new team member successfully used this technique to find his "hidden" commands during a challenge. Tools like prof (originally designed by attackers to find clear-text passwords in memory) can also be repurposed by defenders to dump processes and find attacker strings.
• Stealthier Attacker Technique: Using an older, simpler shell like sh instead of bash, as sh typically doesn't have a history feature, and Volatility might not have a parser for it, could potentially evade this specific memory forensic technique.

1. Dynamic Loader Trick (Bypassing /proc Visibility)

• Anti-Forensic Technique: A highly sophisticated Linux anti-forensic technique involves using the dynamic loader. By passing an argument directly to the loader (ld-linux.so) to execute a malicious program (./root-anti-loader), the actual malicious executable itself does not appear in the /proc/<PID>/exe link. Instead, /proc/<PID>/exe points to the loader, making it appear as if only the legitimate loader is running. This makes it harder for blue teamers to simply copy the malicious binary from /proc.
• Defensive Countermeasure: Even with this trick, the malicious executable must reside in memory to run. Defenders can inspect the memory regions of the process (/proc/<PID>/maps and /proc/<PID>/mem). With a Python script, they can dump these memory regions, reconstruct the executable, and obtain the original ELF file, thus recovering the malicious binary. Berger humorously calls this "Titan magic."

Demo / Proof of Concept

▶ Watch: Real case: recovering exfiltration passwords via MFT (6:00)

Stephan Berger's presentation is rich with practical demonstrations and real-world examples, serving as a continuous proof of concept for the efficacy of advanced forensic techniques against common anti-forensic maneuvers. While there isn't a single, continuous "demo" in the traditional sense, Berger consistently illustrates his points with screenshots and specific command-line examples from actual incidents or controlled lab environments.

For instance, in the discussion of Windows file deletion, he showcases Velociraptor's graphical user interface (GUI) to query the NTFS Master File Table (MFT). He provides a screenshot of Velociraptor returning four hits for a deleted arclone.com file, demonstrating how its plain-text content (including an attacker's password) was recovered. This directly proves that simply deleting a file is insufficient.

Similarly, when discussing the USN journal and SRAMM, Berger presents screenshots from Velociraptor and Elastic Security. He highlights how specific entries in the USN journal pinpoint the exact time a malicious service was created, and how SRAMM can reveal evidence of execution even when other artifacts are absent. He also details the failed attempt to use ChatGPT to tamper with the SRAMM database, showing the actual database structure versus ChatGPT's incorrect assumptions, effectively demonstrating the limitations of AI without human expertise.

On the Linux side, Berger visually demonstrates the clearing of the dmesg kernel ring buffer and then shows the identical, persistent information found in /var/log/kern.log and later in journald. He provides command-line outputs for clearing the journal (journalctl --rotate) and emphasizes that this action itself is logged, showcasing the self-defeating nature of many anti-forensic efforts.

The most compelling "proof of concept" is arguably the memory forensics section. Berger narrates how a new team member successfully used Volatility to extract his "hidden" bash history from memory, despite his attempts to disable on-disk logging. He further illustrates the advanced Linux dynamic loader trick and immediately counters it by explaining how memory dumping and ELF reconstruction can recover the "hidden" malicious executable.

These numerous, specific examples and tool interactions collectively serve as a powerful demonstration that anti-forensics is significantly more challenging than many attackers realize, and that skilled incident responders possess a robust arsenal to uncover their tracks.

Defensive Implications

▶ Watch: SDE limitations and forensic traces (8:00)

Stephan Berger's talk provides critical insights for defenders, emphasizing that a multi-layered and deep understanding of system forensics is paramount to countering anti-forensic techniques. The key defensive implications include:

1. Embrace Multi-Layered Logging and Artifact Collection: Do not rely on a single log source. As demonstrated, attackers might clear dmesg, but the data persists in /var/log/kern.log and journald. Similarly, Windows has the NTFS MFT, USN journal, SRAMM, and various PowerShell logs. A comprehensive collection strategy across these artifacts significantly increases the chances of detection and recovery.
2. Implement Robust EDR/SIEM Detection Rules: Many anti-forensic techniques involve specific commands or system calls. Defenders should develop or leverage existing rules (like those provided by Elastic Security) to detect actions such as:

• fsutil usn delete journal (disabling USN journal)
• seps red line option save nothing or remove item for PowerShell history files.
• dmesg -c (clearing kernel ring buffer).
• journalctl --rotate (clearing journald, even though the action itself is logged).
• Disabling Windows Event Tracing.
• File deletion events, especially for critical system files or known attacker tools.

1. Prioritize Memory Forensics: The talk repeatedly highlights memory as a persistent source of evidence, even when on-disk artifacts are purged. Investing in memory forensics capabilities, including tools like Volatility and custom scripting, is crucial. This allows defenders to recover command history (e.g., bash history), running processes, and even "hidden" executables that evade traditional file system analysis.
2. Understand OS Internals Deeply: Attackers often exploit superficial knowledge of system behavior. Defenders need a deep understanding of how operating systems handle files, logs, and processes (e.g., the intricacies of the NTFS MFT, the USN journal, SRAMM database structures, Linux's /proc filesystem, and journald). This knowledge is essential for identifying subtle traces and for developing effective countermeasures.
3. Validate Anti-Forensic Claims and AI Outputs: Berger's ChatGPT example underscores the danger of blindly trusting tools or AI for anti-forensic advice without thorough validation against actual system behavior. Defenders should encourage red teams to test their anti-forensic techniques rigorously, helping to identify gaps in both offensive and defensive strategies.
4. Monitor for Tampering with Forensic Tools/Services: The act of an attacker trying to stop the SRAMM service or clear the USN journal is itself a strong indicator of malicious activity. Implementing alerts for such actions is a high-value defensive measure.
5. Share and Learn from the Community: Berger frequently references open-source detection rules from Elastic and encourages browsing their repositories to learn new techniques and identify detection gaps. Collaborative knowledge sharing within the security community is vital for staying ahead.

By adopting these defensive strategies, organizations can significantly improve their ability to detect, investigate, and respond to incidents, even when faced with determined adversaries attempting to cover their tracks.

Key Takeaways

• Anti-forensics is harder than it looks: Many common anti-forensic techniques, even those used by APTs, are easily detectable by skilled incident responders with deep OS knowledge.
• Operating systems are persistent by design: Windows' NTFS MFT, USN journal, and SRAMM, along with Linux's journald and memory, log system activity extensively, making true stealth extremely difficult.
• Memory forensics is a critical capability: Even when attackers clear on-disk logs or hide executables, their actions often remain in memory, recoverable via tools like Volatility.
• Multi-layered logging and detection are essential: Relying on a single log source is insufficient. Defenders must collect and analyze data from various artifacts, including those that record tampering attempts.
• Validate assumptions, especially with AI: Do not blindly trust public tools or AI-generated anti-forensic code; a deep understanding of OS internals is crucial for both effective offense and defense.
• The act of anti-forensics often leaves its own trace: Clearing logs or disabling forensic services frequently generates new log entries, providing fresh evidence of tampering.

About the Speaker(s)

Stephan Berger is an IR Team Lead at InfoGuard, a prominent cybersecurity firm. With his extensive experience in incident response, Berger provides a practical, real-world perspective on the challenges and successes of digital forensics. He is an active and responsive member of the cybersecurity community, maintaining a presence on social media and frequently blogging about his insights. Berger is also an avid outdoors enthusiast, proudly embracing his Swiss heritage with a love for mountains and hiking. This talk marks his second appearance at Security Fest, where he clearly feels at home, sharing his expertise and challenging the community to continuously improve.