Closing the Gap vs Adversaries With Community Resources

Closing the Gap vs Adversaries With Community Resources

Speakers: Ian Davila, Lead Adversary Emulation, Scythe Cyber

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=Mxc7Ju8xK60

Overview

In this insightful talk from BSides NYC, Ian Davila, Lead Adversary Emulation at Scythe Cyber, delves into the critical importance of threat-informed defense and how organizations can leverage readily available community resources to bolster their security posture against evolving threats. Davila, a former member of the MITRE ATT&CK team, highlights the common pitfalls in applying security frameworks and presents a pragmatic approach to understanding and defending against adversary tradecraft. The core message emphasizes moving beyond reactive, signature-based detections to a proactive, behavior-centric strategy, powered by collective intelligence.

The presentation outlines a structured methodology for organizations to focus their limited resources effectively. It stresses the necessity of profiling adversaries relevant to one's specific industry and operational context, rather than attempting to defend against every conceivable threat. Davila introduces the Scythe Cyber Community Edition as a pivotal platform that aggregates open-source intelligence, detection logic, and testing frameworks, empowering defenders to identify gaps in their security controls and validate their effectiveness against real-world Tactics, Techniques, and Procedures (TTPs).

This talk is particularly relevant for security professionals struggling with resource constraints and the overwhelming volume of threat intelligence. It provides a clear roadmap for operationalizing MITRE ATT&CK beyond a mere checklist, transforming it into an actionable framework for building resilient defenses. By advocating for a community-driven approach, Davila inspires a shift towards collaborative security, where shared knowledge and tools are instrumental in closing the persistent gap between defenders and sophisticated adversaries.

Background

▶ Watch: Introduction, speaker background, and talk overview (0:00)

The concept of threat-informed defense underpins Davila's entire presentation. It's defined as the ability to understand adversary tradecraft—their Tactics, Techniques, and Procedures (TTPs)—to effectively defend against them. Davila argues that in an environment saturated with diverse threats, no single company can defend against all of them. This necessitates a focused approach: organizations must identify and prioritize the threats most likely to impact them, a process known as threat profiling. This targeted strategy prevents resource exhaustion and ensures security investments deliver maximum value.

A crucial distinction is drawn between TTPs and traditional indicators like CVEs (Common Vulnerabilities and Exposures) and IOCs (Indicators of Compromise). While CVEs focus on initial access vulnerabilities and IOCs (like hash values, IP addresses, domain names) are signature-based detections for known malicious artifacts, TTPs describe the behaviors an adversary exhibits after initial access or when traditional signatures fail. TTPs evolve at a much slower pace than CVEs or IOCs, making them more durable and valuable indicators for long-term defense. Davila references David Bianco's Pyramid of Pain, a foundational concept in threat intelligence, to illustrate this point. At the base are easily changed indicators like hash values, while at the apex are TTPs, which are the most difficult for adversaries to alter, making detections based on them far more impactful. He clarifies that tools themselves are not behaviors; rather, tools implement procedures, which then generate observable behaviors. This distinction is critical for understanding Indicators of Attack (IOAs), which focus on what an adversary is doing rather than merely what they are using or where they are coming from.

This context naturally leads to MITRE ATT&CK, a framework created in 2013, around the same time as the Pyramid of Pain's inception. ATT&CK provides a globally accessible, common language and taxonomy for describing adversary TTPs based on real-world observations. It structures adversary behavior into Tactics (the adversary's goals, e.g., "Execution"), Techniques (how they achieve those goals, e.g., "Command and Scripting Interpreter"), and Procedures (specific implementations by groups or software). Davila emphasizes ATT&CK's power in reducing friction between security teams (blue, red, purple) by providing a shared understanding of adversary actions, thereby enabling more efficient and coordinated defense efforts.

However, Davila is also candid about common pitfalls associated with ATT&CK. Firstly, ATT&CK is entirely dependent on real-world observations and community contributions; if behaviors aren't reported, they won't be in the framework. Secondly, it's not a "silver bullet," and claiming "100% coverage" is misleading. Instead, organizations should focus on confidence levels in detecting particular techniques implemented by specific groups or software. This requires diving deep into the actual procedures and crafting broad yet accurate detections that minimize false positives. The discussion also touches upon the challenge of opportunistic threats versus targeted APTs, acknowledging that many organizations are more likely to be hit by widespread campaigns like ransomware than by highly sophisticated state-sponsored actors. For these organizations, basic hygiene like patching and focusing on prevalent opportunistic TTPs (e.g., common ransomware behaviors) becomes paramount, even within a threat-informed framework.

Key Findings

▶ Watch: Defining threat-informed defense and its importance (2:18)

Ian Davila's talk reveals several key findings and contributions that empower defenders to enhance their security posture through community-driven resources:

1. Community is the Cornerstone of Effective Threat-Informed Defense: The efficacy of frameworks like MITRE ATT&CK is directly proportional to community contributions. Real-world observations, analyses, and shared intelligence from security professionals are what make ATT&CK a living, breathing, and valuable resource. Without this collective input, the knowledge base stagnates, limiting its usefulness.
2. Threat Profiling is Non-Negotiable for Resource Optimization: Given the overwhelming landscape of threats, organizations cannot afford to defend against everything. The talk emphasizes that effective defense hinges on threat profiling, where organizations identify and prioritize adversaries most likely to target them (direct threats, industry threats) alongside an understanding of common opportunistic threats. This focused approach ensures scarce security resources are allocated efficiently.
3. Beyond Raw ATT&CK Data: Enriched, Actionable Intelligence: While MITRE ATT&CK provides the foundational language, Davila demonstrates that its true power is unlocked when enriched with additional metadata and integrated with practical defensive and testing solutions. The Scythe Cyber Community Edition serves as a prime example, adding layers of context like adversary motivations, suspect attribution, and targeted sectors to ATT&CK groups, making threat profiling more intuitive and actionable.
4. Shift from "Coverage" to "Confidence" in Detections: The notion of achieving "100% ATT&CK coverage" is dismissed as a myth. Instead, defenders should strive for high confidence in their ability to detect specific adversary procedures. This requires a granular understanding of how particular groups or software implement techniques and ensuring detections are tailored to these nuances, aiming for broad detection capabilities without excessive false positives.
5. Integration of Open-Source Tools Creates a Powerful Ecosystem: The Scythe Cyber Community Edition showcases how integrating diverse open-source tools and resources—such as Sigma rules for detection logic, Olaf Hartong's Sysmon Modular configuration for endpoint telemetry, Elastic detections, and Atomic Red Team for adversary emulation—creates a synergistic environment. This ecosystem allows defenders to identify gaps, source detection logic, and validate their controls all within a unified platform.
6. Personalized Knowledge Bases Bridge Gaps in Public Intelligence: Recognizing that even comprehensive frameworks like ATT&CK might not contain every observed adversary procedure, the talk highlights the capability to customize and expand one's knowledge base. By allowing users to duplicate and modify adversary profiles and add specific, newly discovered procedures (like a unique rundll32 usage by Wizard Spider), the platform empowers organizations to tailor their threat intelligence to their unique findings and operational needs.
7. Continuous Validation Through Testing is Paramount: A critical finding is that detections, however well-crafted, are only truly valuable if they are regularly tested against real-world adversary behaviors. Tools like Atomic Red Team provide a mechanism for adversary emulation, allowing organizations to validate their security controls and ensure their detections function as intended before an actual attack occurs. This proactive testing prevents unpleasant surprises during an incident.

Technical Deep Dive

▶ Watch: Explaining why TTPs are more valuable than CVEs or IOCs (3:40)

The technical core of Davila's talk revolves around the Scythe Cyber Community Edition, a free platform designed to aggregate and operationalize community-driven threat intelligence and defensive resources. This platform directly addresses the common pitfalls of MITRE ATT&CK by providing an enriched, interactive environment for threat profiling, gap analysis, and control validation.

Threat Profiling and Enrichment

The platform begins by enhancing the standard MITRE ATT&CK group data with additional metadata. This enrichment includes critical context such as:

• Motivation: Financial, espionage, destructive, etc.
• Suspect Attribution: Countries or regions believed to sponsor or originate the group.
• Observed Sectors: Industries commonly targeted (e.g., Financial Services, Infrastructure, Manufacturing).
• Observed Countries: Geographic regions where the group has operated.

Users can leverage these metadata fields to perform sophisticated filtering. For instance, Davila demonstrates how to narrow down the 133+ ATT&CK groups to a manageable subset by filtering for groups that have targeted the United States, operate within the Financial Services sector, and are financially motivated. This process helps organizations identify their "direct threats" and prioritize their focus. The filtered groups can then be added to a Matrix, which serves as a persistent "label bar" to maintain the research context throughout the platform. This matrix visually highlights overlapping techniques across the selected groups, providing immediate insights into common adversary behaviors.

Identifying Defensive Gaps

A significant technical contribution is the platform's ability to overlay threat intelligence with defensive capabilities to reveal gaps. This is achieved through the Product Registry, which hosts configurations and detection logic from various community resources. Davila showcases this by:

1. Selecting the previously profiled threat groups.
2. Adding Olaf Hartong's Sysmon Modular configuration from the product registry. Sysmon Modular is an open-source, community-driven configuration for Sysmon, a powerful Windows system monitor that provides detailed logging for security analysis.
3. The platform then visually indicates where the selected Sysmon configuration provides detection coverage against the techniques used by the profiled threat groups.
4. Crucially, it highlights gaps—techniques for which no detection is available through the chosen defensive tool. For example, Davila identifies that "Scheduled Task" (T1053.005) lacked coverage in the Sysmon Modular configuration, presenting a clear gap.

To address these gaps, the platform integrates with analytics, specifically Sigma rules. Sigma is a generic and open signature format that allows security analysts to describe detection methods in a vendor-agnostic way. For a technique like "Scheduled Task" identified as a gap, users can browse available Sigma rules within the platform. These rules, though unimplemented, provide the detection logic that can be adapted and deployed into an organization's specific security solution (e.g., SIEM, EDR). This provides a "quick win" for defenders to source and implement detections for critical uncovered techniques.

Customizing and Expanding the Knowledge Base

Davila emphasizes that ATT&CK, while comprehensive, may not contain every single observed adversary procedure. To address this, the Scythe Cyber Community Edition allows users to personalize their knowledge base. The process is demonstrated through an example involving Wizard Spider, a prominent financially motivated threat group.

1. Using Google Dorking (e.g., site:crowdstrike.com "Wizard Spider"), Davila finds a CrowdStrike blog post detailing Wizard Spider's use of specific defensive evasion techniques.
2. The blog post reveals a procedure for rundll32 (specifically, the "Signed Binary Proxy Execution" technique, T1218.011) that wasn't explicitly tagged under Wizard Spider in the default ATT&CK data within the platform.
3. Users can duplicate an existing group (e.g., Wizard Spider) into their personal account, creating a custom "technique set."
4. Within this custom set, users can then add notes to existing techniques or even add new techniques/procedures not present in the original ATT&CK data. Davila shows adding the rundll32 procedure with its specific details and references.
5. This customized version of the group, marked as a "draft," then appears in the user's matrix, allowing for a tailored view of adversary tradecraft that reflects their unique research and intelligence. This feature is crucial for adapting to newly observed adversary behaviors that haven't yet been officially incorporated into ATT&CK.

Testing and Validation with Community Tools

After identifying gaps and potentially implementing new detections, testing is the final, critical step. The platform integrates with Atomic Red Team, an open-source adversary emulation project that provides atomic tests for hundreds of ATT&CK techniques.

1. For a technique like rundll32, the platform lists specific Atomic Red Team tests. These tests are small, highly focused executions designed to simulate specific adversary procedures (e.g., rundll32 with control.exe, rundll32 for persistence).
2. Defenders can use these tests to validate whether their newly implemented detections (e.g., from Elastic or custom Sigma rules) are effective against the actual adversary behavior. This direct feedback loop is essential for ensuring security controls are working as intended and provides confidence in the defensive posture.

Leveraging Other Community Detections

The talk also highlights other valuable community resources for detection:

• Elastic Detections: The Elastic team provides a vast array of open-source detection logic, often available on GitHub. Davila demonstrates how the Scythe platform links directly to these resources, allowing users to inspect the detailed detection logic for techniques like rundll32 (e.g., "Initial Child Process of rundll32.exe"). Even if an organization doesn't use Elastic products, this publicly available logic can serve as a template for implementing detections in other security solutions.

The technical deep dive underscores that the Scythe Cyber Community Edition acts as a central hub, making it easier for defenders to access, process, and apply the wealth of open-source security intelligence to build a more robust, threat-informed defense.

Demo / Proof of Concept

▶ Watch: Understanding the Pyramid of Pain and detection levels (4:40)

The core of Ian Davila's presentation was a compelling live demonstration (pre-recorded for reliability) of the Scythe Cyber Community Edition platform, showcasing how defenders can practically apply the principles of threat-informed defense using community resources.

The demo began by navigating to the "Groups" section within the Scythe Community Edition, which enriches MITRE ATT&CK adversary group data with additional metadata. Davila demonstrated filtering capabilities by applying several criteria:

1. Targeted Countries: Filtering for groups that have targeted the United States.
2. Observed Sectors: Specifying Financial Services as the target industry.
3. Motivation: Selecting Financially Motivated groups.

This process effectively narrowed down the vast number of ATT&CK groups (over 133) to a focused list of 21, then to an even smaller, more relevant set of financially motivated groups targeting US financial services. These filtered groups were then added to a "Matrix" – a persistent label bar at the top of the interface – allowing the user to maintain context throughout their research. The matrix visually represents the overlap of techniques used by these selected groups.

Next, Davila moved to the "Product Registry" to identify defensive gaps. He selected the filtered threat groups and then added Olaf Hartong's Sysmon Modular configuration as a defensive tool. This action overlaid the detection capabilities of Sysmon Modular onto the techniques used by the selected adversaries. The visual representation in the matrix immediately highlighted areas of detection coverage and, more importantly, gaps. A specific gap was identified for the "Scheduled Task" technique (T1053.005), indicating that the Sysmon Modular configuration, in this context, did not provide a direct detection. To address this, Davila then navigated to the "Analytics" section, which houses Sigma rules – generic, open-source detection logic. Here, he could find and review various Sigma rules relevant to "Scheduled Task," providing readily available, unimplemented detection logic that could be adapted and deployed by a security team.

The demo then transitioned to showcasing how to personalize the knowledge base. Davila took the example of Wizard Spider, a ransomware group. He explained how a defender might perform Google Dorking (e.g., searching CrowdStrike blogs for "Wizard Spider") to uncover new, specific procedures not yet in ATT&CK. He found a CrowdStrike report detailing Wizard Spider's use of rundll32 for defensive evasion (specifically, the "Signed Binary Proxy Execution" technique, T1218.011). To incorporate this, he demonstrated duplicating the existing Wizard Spider group into a personal "technique set" within the platform. He then added a new note to the rundll32 technique, including the specific procedure details and the link to the CrowdStrike reference. This customized group, marked as a "draft," then appeared in his matrix, demonstrating how users can enrich their threat intelligence with their own findings.

Finally, the demo highlighted testing and validation. For the rundll32 technique, Davila showed how the platform integrates with other community resources:

• Elastic Detections: The platform displayed available open-source detection logic from Elastic for rundll32, such as "Initial Child Process of rundll32.exe," providing concrete examples of how to detect this behavior.
• Atomic Red Team: The platform listed various Atomic Red Team tests specifically designed for rundll32, including scenarios like "RunDLL32 with Control.exe" or "RunDLL32 for Persistence." These tests provide actionable adversary emulation scenarios that defenders can use to validate their implemented detections against realistic adversary behaviors.

The demo effectively illustrated the platform's utility in streamlining threat profiling, identifying defensive gaps, sourcing detection logic, customizing intelligence, and validating security controls, all through the power of integrated community resources.

Defensive Implications

▶ Watch: Introducing the MITRE ATT&CK framework and its structure (6:20)

The insights shared by Ian Davila carry significant defensive implications for security teams aiming to mature their capabilities and efficiently allocate resources:

1. Prioritize and Focus with Threat Profiling: Defenders must move beyond a "defend against everything" mindset. By leveraging enriched MITRE ATT&CK data and metadata (motivation, sectors, attribution), organizations can create targeted threat profiles that prioritize direct threats, industry-specific adversaries, and common opportunistic attacks (like ransomware). This focused approach ensures scarce resources are directed where they will have the most impact. Don't waste time on APTs that will never target your small business, but don't ignore the opportunistic ransomware that will hit everyone.

1. Foundational Security Remains Critical: While the talk emphasizes TTPs, Davila explicitly states that basic security hygiene, such as patching CVEs and implementing IOC-based detections, remains 100% crucial. Threat-informed defense builds on top of these fundamentals, not in place of them. Organizations must first address known vulnerabilities and block known malicious artifacts before advancing to behavioral detections.

1. Shift from "Coverage" to "Confidence" in Detections: Security teams should evaluate their detections not merely by how many ATT&CK techniques they "cover," but by their confidence level in detecting specific adversary procedures. This means understanding the granular ways adversaries implement techniques and ensuring detections are robust enough to catch variations without excessive false positives. A single detection for a broad technique like PowerShell execution might not be enough; understanding how a specific threat group uses PowerShell for persistence allows for more precise and confident detection.

1. Leverage Community-Driven Detections and Intelligence: Platforms like the Scythe Cyber Community Edition, along with resources like Sigma rules, Elastic detections, and Sysmon Modular configurations, provide a wealth of readily available, open-source intelligence and detection logic. Defenders should actively explore and integrate these community resources to quickly address identified gaps in their security posture, rather than reinventing the wheel. The talk highlights how Elastic, for instance, provides free detection logic that can be adapted even if an organization doesn't use their products.

1. Proactive Testing is Non-Negotiable: Implementing detections without validating them is a critical oversight. Defenders must regularly use adversary emulation tools like Atomic Red Team to test their security controls against realistic adversary TTPs. This proactive testing validates detection efficacy, identifies control gaps, and ensures that security investments are truly effective before a real incident occurs. It helps answer the crucial question: "Do our detections actually work?"

1. Customize Your Threat Intelligence: No single public framework will contain every piece of intelligence relevant to an organization. Defenders should actively research specific adversary procedures (e.g., through CTI reports, malware analysis) and use tools that allow them to customize and enrich their threat intelligence knowledge base. Adding specific procedures, notes, and references to techniques within their own, personalized matrix ensures their defense is tailored to the most current and relevant threats.

1. Contribute Back to the Community: To strengthen collective defense, security professionals are encouraged to contribute their findings back to broader community efforts. This includes submitting new techniques or procedures to MITRE ATT&CK, sharing detection logic with projects like Sigma, and contributing to community spotlights. This collaborative spirit ensures that the knowledge base evolves faster and benefits everyone, ultimately making it harder for adversaries to succeed.

Key Takeaways

• Threat-informed defense is essential for efficient resource allocation: Organizations must prioritize defending against adversaries most relevant to their specific industry and operational context, rather than attempting to cover all possible threats.
• MITRE ATT&CK provides a common language but requires enrichment and granular understanding: While ATT&CK is a powerful framework, its true value comes from enriching it with specific adversary metadata and diving deep into the procedures used by threat groups, moving beyond generic technique coverage.
• Leverage community resources to bridge knowledge and capability gaps: Platforms like Scythe Cyber Community Edition, alongside open-source tools such as Sigma rules, Atomic Red Team, Elastic detections, and Sysmon Modular, offer invaluable intelligence, detection logic, and testing capabilities that empower defenders.
• **Focus on detection confidence by understanding adversary procedures and testing controls:** Instead of chasing "100% coverage," prioritize building high-confidence detections for specific, real-world adversary procedures, ensuring these detections are regularly validated against adversary emulation.
• Proactive testing against real adversary tradecraft is non-negotiable for validating controls: Regularly using tools like Atomic Red Team to simulate adversary behaviors is crucial to confirm that security controls and detections function as intended before an actual attack, preventing surprises during an incident.
• Contribute back to the security community to strengthen collective defense: Sharing discovered techniques, procedures, and detection logic with frameworks like MITRE ATT&CK and projects like Sigma enhances the collective knowledge base, making it harder for adversaries to operate effectively.

About the Speaker(s)

Ian Davila is the Lead Adversary Emulation at Scythe Cyber, a role that aligns perfectly with his passion for threat-informed defense and the MITRE ATT&CK framework. Prior to joining Scythe Cyber, Davila was an integral part of the MITRE ATT&CK team, where he focused on technique research, particularly for Windows environments. He also contributed to the MITRE ATT&CK Evaluations team, responsible for implementing the code used to test and evaluate various security vendors' detection capabilities against real-world adversary emulation scenarios. Originally from Puerto Rico, Davila brings a deep technical background and a community-oriented perspective to the field of cybersecurity.