The Hidden Treasure of Crash Reports?

The Hidden Treasure of Crash Reports?

Speakers: Patrick Wardle, CEO and Co-founder of Objective-See, Founder of the Objective-C Foundation

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=35mdBRhulTQ

Overview

Patrick Wardle's talk, "The Hidden Treasure of Crash Reports?", challenges the conventional perception of crash reports as mere debugging tools for developers. Wardle, a prominent figure in macOS security, passionately argues that these often-overlooked diagnostic files are, in fact, rich repositories of critical information for a much broader audience, including users, security teams, and even adversaries. The core premise is that by understanding, collecting, and analyzing crash reports, organizations and individuals can uncover significant insights into software vulnerabilities, detect sophisticated malware, identify failed exploit attempts, and gain an "absolute truth" about system stability and security incidents.

The presentation primarily focuses on macOS crash reports, but the principles extend to other operating systems like Windows and iOS. Wardle aims to shift the audience's perspective, encouraging them to "fall in love" with crash reports due to their immense potential to enhance software reliability and bolster security postures. He illustrates this potential through compelling real-world case studies, demonstrating how crash reports have been instrumental in identifying zero-day vulnerabilities and pinpointing the root cause of widespread system failures, often contradicting initial assumptions.

This talk is crucial for anyone involved in software development, cybersecurity, or IT operations. It highlights an underutilized data source that, when properly harnessed, can provide early warnings against advanced threats, accelerate vulnerability patching, and improve incident response capabilities. By demystifying the structure and utility of crash reports, Wardle empowers attendees to leverage this "hidden treasure" to proactively defend against an evolving threat landscape.

Background

▶ Watch: Introduction to the hidden treasures of crash reports (0:00)

Traditionally, crash reports have been viewed almost exclusively through the lens of software development. When an application or system component fails, a crash report is generated, providing developers with a snapshot of the system state at the moment of failure. This data, which typically includes a stack trace, register values, and memory dumps, is invaluable for diagnosing bugs, identifying logical errors, and ultimately improving software quality. For the average user, a crash report is often an annoying consequence of unstable software, something to be reluctantly forwarded to support or simply ignored.

However, the landscape of software security has evolved significantly, introducing new dimensions to the utility of crash reports. Modern operating systems are equipped with robust anti-exploitation mechanisms designed to mitigate the impact of vulnerabilities, particularly those involving memory corruption. These defenses, such as Address Space Layout Randomization (ASLR), Data Execution Prevention (DEP), and Control-Flow Integrity (CFI), make it increasingly difficult for exploits to achieve 100% reliability. Consequently, many exploit attempts, especially against sophisticated targets, result in a crash rather than successful code execution. This unreliability transforms what was once a developer's headache into a security team's potential goldmine.

Before Wardle's insights, the security community often focused on active indicators of compromise (IOCs) or static analysis of binaries. The idea of routinely ingesting and analyzing system-generated crash reports for security purposes was not as widely adopted, particularly outside of large operating system vendors like Microsoft. While some specialized security companies, especially in the iOS ecosystem, had begun to recognize this potential, the broader cybersecurity community, particularly those defending macOS environments, had not fully embraced crash reports as a primary source of threat intelligence. This talk seeks to bridge that gap, highlighting why this problem exists—a lack of awareness and appreciation for the security data embedded within these diagnostic files—and how it can be addressed.

Key Findings

▶ Watch: Developer benefit: fixing software (Hopper example) (2:00)

The central and most compelling finding presented by Patrick Wardle is that crash reports are an "absolute truth" and an underappreciated source of intelligence for a diverse range of stakeholders beyond traditional developers. This fundamental shift in perspective underpins all other discoveries highlighted in the talk.

Firstly, Wardle emphasizes the immediate value for users and developers. Using his personal experience with the Hopper disassembler as an example, he illustrates how sending crash reports to developers, accompanied by reproduction steps, enables rapid bug fixes. This creates a "win-win" scenario: developers release less buggy software, and users experience greater stability.

Secondly, and most significantly for the security community, crash reports are identified as a critical mechanism for uncovering failed exploitation attempts and zero-day vulnerabilities. Because modern anti-exploitation mechanisms often cause exploits to crash, these failures generate reports that, when analyzed by security teams or OS vendors, can reveal previously unknown flaws. Wardle cites a powerful case study from Microsoft, where out of "millions of crash reports" ingested, they identified two that revealed a zero-day vulnerability affecting "every version of Windows," allowing adversaries to remotely infect systems. This discovery led to a patch, demonstrating the profound impact crash report analysis can have on global security.

Thirdly, the talk highlights the relevance of crash reports for malware detection products. Wardle argues that malware is inherently "buggy" for two main reasons:

1. Incompetent Authorship: Malware authors are often less skilled and rigorous than legitimate software engineers, leading to poorly tested and written code that crashes frequently.
2. Edge-Pushing Behavior: Malware often utilizes private APIs or performs unusual operations that are sensitive to operating system updates, leading to crashes when the OS changes.

By ingesting and analyzing crash reports, security tools can identify crashes originating from unauthorized or malicious software, providing a novel detection vector. Companies like ZecOps (acquired by Jamf) and iVerify are presented as examples of firms already leveraging crash report analysis, predominantly on iOS, to identify sophisticated threats.

Finally, Wardle underscores the unique "absolute truth" nature of crash reports, contrasting them with speculative claims or misinterpretations during incidents. He references the CrowdStrike incident, where initial system crashes led many to incorrectly blame Microsoft. However, a proper analysis of the kernel driver crash reports quickly revealed that CrowdStrike's driver was the culprit. This example powerfully illustrates that understanding crash reports can cut through misinformation, accurately attribute fault, and accelerate effective remediation, proving their indispensable role in incident response and post-mortem analysis.

Technical Deep Dive

▶ Watch: Microsoft using crash reports to find zero-day vulnerabilities (3:05)

While the talk emphasizes the value of crash reports rather than an exhaustive breakdown of their internal structure, it provides crucial insights into how these reports can be programmatically monitored and utilized, particularly on macOS.

A typical crash report, regardless of the operating system, contains a wealth of diagnostic information. Although not explicitly detailed in the transcript, this generally includes:

• Crash Type and Reason: The specific signal or exception that caused the crash (e.g., EXC_BAD_ACCESS for memory access violations).
• Process Information: Details about the crashing process, including its name, Process ID (PID), and parent process.
• Thread States: The state of all threads within the crashing process, including their stack traces (the sequence of function calls leading to the crash). The stack trace is often the most critical piece of information for debugging.
• Register Values: The contents of the CPU registers at the time of the crash.
• Loaded Modules/Libraries: A list of all dynamically linked libraries and executables loaded into the process's memory space, along with their base addresses. This helps in identifying the specific code responsible for the crash.
• System Information: Details about the operating system version, hardware, and uptime.

The core technical contribution presented by Wardle is a methodology for monitoring crash report directories on macOS. The general workflow is articulated as: enumerate, monitor, collect, process.

1. Enumerate: Identify the standard directories where crash reports are stored. On macOS, these typically include:

• /Library/Logs/DiagnosticReports/ (for system-wide and some user-level crashes)
• ~/Library/Logs/DiagnosticReports/ (for user-specific application crashes)
• /var/log/DiagnosticReports/ (for some daemon crashes)
• /Library/Application Support/CrashReporter/ (legacy location, still used by some apps)

1. Monitor: Continuously watch these directories for the creation of new crash report files. Wardle highlights the use of Endpoint Security (ES) framework, a powerful and privileged API provided by Apple for system monitoring. ES allows security tools to subscribe to a wide range of system events, including file system operations.

• Specifically, Wardle mentions using mute inversion with Endpoint Security. The ES framework allows muting (ignoring) specific events to reduce noise. By "inverting" this logic, a security product can tell the operating system to only deliver very specific events, such as file creations (ES_EVENT_TYPE_NOTIFY_CREATE) within the designated crash report directories. This highly targeted approach ensures that the monitoring agent is only notified when a new crash report file appears, minimizing overhead and maximizing efficiency.

1. Collect: Once a new crash report is detected, the monitoring agent collects the file. This typically involves reading its contents, which are usually plain text or a structured format like plist or JSON (though macOS reports are generally text-based).
2. Process: The collected crash report is then subjected to analysis. This processing can involve:

• Parsing: Extracting key fields like the crashing process name, stack trace, and crash reason.
• Symbolication: Converting raw memory addresses in the stack trace into human-readable function names and line numbers. This requires access to the original binaries and their debug symbols.
• Fingerprinting: Creating unique identifiers for specific crash patterns to track recurrence or identify known vulnerabilities.
• Correlation: Linking crash reports to known malware samples, exploit attempts, or system updates.
• Alerting: Generating alerts for security teams if a crash report indicates a suspicious event (e.g., a crash from an unknown process, a crash pattern matching a known exploit signature, or a kernel panic).

The technical implementation using Endpoint Security and mute inversion provides a robust and efficient way for security products to passively monitor system health and detect anomalies that might otherwise go unnoticed. This method allows for real-time detection of crashes, enabling swift response to potential security incidents or critical system instabilities.

Demo / Proof of Concept

▶ Watch: Crash reports as "absolute truth" (CrowdStrike incident) (5:58)

In the context of the talk, Patrick Wardle referenced the underlying code for monitoring crash report directories using Endpoint Security and mute inversion. He explicitly stated, "I'm not going to really walk through all this code, it's more for reference," indicating that while the technical mechanism was presented, a live, step-by-step demonstration of the code execution was not the primary focus.

Instead, the code snippet served as a conceptual proof of concept, illustrating how security teams could implement such monitoring. It highlighted the specific Endpoint Security events and filtering techniques necessary to efficiently detect new crash report file creations on macOS. This approach effectively conveyed the feasibility and technical pathway for ingesting crash reports into a security analysis pipeline without diverting the talk's core message into a detailed coding tutorial. The implication was that the underlying mechanisms are readily available within the macOS framework for those with the technical expertise to leverage them.

Defensive Implications

▶ Watch: Monitoring crash reports in an enterprise (7:15)

The insights presented by Patrick Wardle offer profound defensive implications for organizations and individuals alike, transforming crash reports from mere diagnostic files into powerful security intelligence assets. Defenders should adopt a proactive stance by integrating crash report analysis into their security operations.

1. Proactive Zero-Day Detection: Security teams should establish systems to ingest and analyze crash reports from all endpoints. As demonstrated by the Microsoft case study, failed exploit attempts against zero-day vulnerabilities often result in crashes. By meticulously analyzing these reports, organizations can potentially uncover previously unknown flaws in their software stack or operating system before they are successfully exploited. This requires advanced analytics capabilities to identify unusual crash patterns or crashes originating from sensitive processes.

1. Enhanced Malware Detection: Implement monitoring for crashes originating from unknown or suspicious processes. Malware is frequently buggy, especially when pushing the boundaries of the operating system or when OS updates break its reliance on private APIs. A crash from an unexpected application or a system utility that doesn't typically crash could indicate a malicious process attempting an illicit operation. Integrating crash report data into an Endpoint Detection and Response (EDR) system can provide a valuable, passive detection vector for evasive or poorly written malware.

1. Accurate Incident Response and Root Cause Analysis: Train security analysts and incident responders on how to properly interpret crash reports. The CrowdStrike incident highlighted the danger of misattributing system failures based on superficial observations. Understanding the stack trace, the crashing kernel driver, and the context provided by a crash report is crucial for quickly and accurately identifying the true cause of system instability or security incidents, preventing wasted effort and enabling targeted remediation.

1. Leveraging macOS Endpoint Security: For macOS environments, defenders should explore implementing crash report monitoring using the Endpoint Security (ES) framework, specifically employing the mute inversion technique described. This allows for efficient and targeted detection of new crash report files, minimizing system overhead while maximizing visibility into potential security events. This capability can be integrated into custom security tools or leveraged by commercial EDR solutions that support ES.

1. Vulnerability Management and Patch Prioritization: Analyze recurring crash patterns to identify underlying software bugs or vulnerabilities. Even if not directly malicious, frequent crashes can indicate stability issues that might be exploited by attackers. Prioritizing patches for software components that frequently generate crash reports can improve overall system resilience.

1. User Empowerment: Educate users on the importance of submitting crash reports, especially for problematic applications like the Hopper disassembler example. While direct security benefits for individual users might be less immediate, fostering a culture of reporting helps developers fix bugs, leading to more stable software ecosystems that are inherently harder to exploit.

By embracing crash reports as a vital source of security intelligence, defenders can move beyond reactive measures, gaining a more comprehensive understanding of their systems' health and potential threats, ultimately building more resilient and secure environments.

Key Takeaways

• Crash reports are an underutilized security asset: Beyond debugging, they offer critical intelligence for detecting malware, identifying zero-day vulnerabilities, and understanding system failures.
• Exploit failures generate valuable data: Modern anti-exploitation mechanisms often cause exploits to crash, creating crash reports that can reveal sophisticated attack attempts and unknown flaws.
• Malware often crashes: Due to poor coding or reliance on private APIs, malware frequently generates crash reports, offering a passive detection mechanism for security tools.
• Crash reports provide "absolute truth": They offer objective evidence for incident response, helping to accurately attribute fault and understand the root cause of system issues, as seen in the CrowdStrike example.
• Monitoring crash reports is feasible: On macOS, the Endpoint Security framework with mute inversion allows for efficient and targeted monitoring of crash report directories.
• Defenders should ingest and analyze: Organizations should integrate crash report collection and analysis into their security operations to proactively detect threats and improve incident response.

About the Speaker(s)

Patrick Wardle is a highly respected figure in the macOS security community. He is the CEO and Co-founder of Objective-See, a company dedicated to creating free, open-source security tools for macOS users. In addition to his work at Objective-See, Wardle is also the founder of the Objective-C Foundation, an organization focused on supporting and advancing the Objective-C programming language and its ecosystem. Known for his deep technical expertise and passion for educating the community, Wardle regularly presents at major security conferences, sharing insights into macOS vulnerabilities, exploitation techniques, and defensive strategies. His work has significantly contributed to raising awareness about macOS security and providing practical solutions for users and security professionals.