A Blueprint for Detection Engineering: Tools, Processes, and Metrics

A Blueprint for Detection Engineering: Tools, Processes, and Metrics

Speakers: Jose and Eric, Splunk Threat Research

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=KfNggeXWDeY

Overview

This talk, presented by Jose and Eric from the Splunk Threat Research team at RSAC 2024, delves into the critical discipline of detection engineering. It addresses a fundamental challenge faced by security organizations: the ability to definitively prove an organization can detect a specific attack and to do so rapidly. The speakers articulate a comprehensive blueprint encompassing tools, processes, and metrics designed to mature an organization's detection capabilities from ad-hoc, reactive measures to a continuous, automated, and highly effective defensive posture.

The core premise of the presentation revolves around the question: "How do you know your organization can detect Attack X? Can you prove it? And can you answer us within 15 minutes?" If the answer is no, the speakers contend that a robust detection engineering practice is essential. This talk is not just theoretical; it shares the practical journey and open-source contributions of the Splunk Threat Research team, who transformed their own detection development and validation process, reducing investigation times from weeks to mere minutes.

The importance of this talk lies in its actionable guidance for security teams struggling with the efficacy and maintainability of their detection rules. By outlining a structured approach, leveraging version control, automated simulations, continuous integration/continuous deployment (CI/CD), and data-driven metrics, the speakers provide a clear roadmap for building confidence in detection capabilities, improving coverage against evolving threats, and ultimately, fighting back more effectively against adversaries.

Background

▶ Watch: Core questions behind detection engineering (0:40)

The genesis of Splunk's deep dive into detection engineering stemmed from a painful operational reality. Initially, the Splunk Threat Research team shipped detections as simple searches for their Splunk Enterprise Security product. While intended to provide out-of-the-box solutions, this approach quickly led to a deluge of escalations from professional services. Customers reported that these detections often "just didn't work," and the Splunk team struggled to diagnose why. Each such escalation consumed an arduous three days to two weeks of a small threat research team's time, highlighting a severe lack of transparency and provability in their detection content.

This challenge forced the team to confront core questions: How could they answer these questions faster? How could they internally prove that a specific detection effectively captured a given threat? And how could they communicate this efficacy to customers and internal teams in the most transparent way possible? This quest for efficiency and provability directly led them to develop the tooling and methodologies that now form their detection engineering blueprint.

The speakers define detection engineering as a process involving the development, testing, and simulation of attacks and Tactics, Techniques, and Procedures (TTPs) to create detections (which can be rules, models, or algorithms) that identify malicious activities. This entire cycle typically culminates in execution within a Security Information and Event Management (SIM) system. They emphasize that while AI might offer a generic definition, the practical application involves specific core components:

1. Git or a Git-driven workflow: Essential for managing detections at scale, providing version control, history, and collaborative capabilities.
2. Simulations: Crucial for testing detections against realistic attack scenarios, especially when live customer data is not available or suitable for testing.
3. Analytics engine: The platform where detections are executed, such as Splunk Enterprise Security, but conceptually, it could be any system capable of processing data for anomalies, even a simple SQL server.
4. Automation: To accelerate the SOC process, enabling faster triage and response to detected events.
5. Metrics: To measure the success and progress of detection efforts, allowing teams to track improvement and identify gaps.

Before adopting these principles, the Splunk team's process was largely ad hoc, relying on tools like Jira and Excel for tracking. Testing often occurred in production environments, and detection development was primarily reactive, driven by emergency threats rather than proactive coverage planning. This reactive, manual approach highlighted the urgent need for a more structured and automated methodology to validate and deploy detection content effectively.

Key Findings

▶ Watch: Defining detection engineering: process, detections, malicious activity (2:40)

The talk presents several key findings and contributions that collectively form a robust framework for effective detection engineering:

1. Detection Engineering is a Necessity for Modern Security Operations: The central finding is that a structured detection engineering practice is not merely beneficial but essential for proving detection efficacy, reducing response times, and building confidence in a security program. The Splunk team's journey from multi-week investigation times to answering detection questions in minutes serves as compelling evidence.
2. A Holistic Approach Requires Specific Core Components: Successful detection engineering integrates five critical elements: a Git-driven workflow for content management, automated simulations for testing, a capable analytics engine (SIM), automation for operational efficiency, and meaningful metrics for measuring success and guiding improvement.
3. Open-Source Tooling Accelerates Adoption: The Splunk Threat Research team has not only developed an internal process but has also open-sourced key tools that enable others to implement similar practices. Tools like Splunk Attack Range, Atomic Red Team (to which Splunk is a major contributor), Splunk Attack Data, and Content CTL provide a ready-to-use ecosystem for building, testing, and validating detections.
4. Content Goes Beyond Simple Searches: A detection is more than just the underlying query (e.g., Splunk SPL). It must include rich metadata such as descriptions, names, linked resources (blog posts, CVEs), MITRE ATT&CK IDs, risk scores, and potentially SOAR playbooks. This comprehensive documentation is crucial for maintainability, collaboration, and transparency.
5. Automated Testing and CI/CD are Transformative: Integrating automated validation, building, and testing into a CI/CD pipeline drastically improves the quality and deployability of detection content. This ensures rapid feedback to detection authors, catches errors early, and allows for frequent, confident releases.
6. Metrics Drive Strategic Improvement: High-level metrics, particularly MITRE ATT&CK coverage diagrams and asset-type coverage analysis, provide invaluable insights for SOC managers and CISOs. Tracking these metrics over time allows teams to identify gaps, prioritize development efforts, and demonstrate tangible improvements in security posture.
7. Detection Engineering Follows a Maturity Model: Organizations typically progress through stages: Ad hoc, Process, Automation, and finally, Continuous. Recognizing this path helps teams set realistic goals and build capabilities incrementally, emphasizing that achieving continuous maturity requires ongoing effort and adaptation.

Technical Deep Dive

▶ Watch: Introduction to core components of detection engineering (3:30)

The technical blueprint for detection engineering presented by Splunk is deeply rooted in a DevOps philosophy, applying software development best practices to security content. The core components identified are not merely theoretical concepts but are supported by specific open-source tools and a well-defined process.

At the foundation is a Git-driven workflow. The speakers stress that managing detections at scale is impossible without robust version control. Git provides not only a history of every change but also enables inline documentation alongside the detection content itself. This ensures that descriptions, context, and references are always co-located with the detection logic, avoiding the pitfalls of scattered documentation in Jira tickets or emails. Furthermore, Git facilitates CI/CD (Continuous Integration/Continuous Deployment), allowing automated checks and deployments on every commit, significantly accelerating the feedback loop for detection authors.

Simulations are paramount for testing detection efficacy. The talk highlights the importance of generating realistic test data to validate detection hypotheses. Splunk leverages and contributes significantly to Atomic Red Team, an open-source project by Red Canary. Atomic Red Team provides thousands of small, granular attack simulations, or "atomics," each tagged with specific MITRE ATT&CK IDs and descriptions. These atomics can range from simple PowerShell commands to executing complex executables or installing DLLs. This rich library allows for precise testing against known TTPs.

To execute these simulations and collect relevant logs, Splunk developed Splunk Attack Range, their most popular open-source project (github.com/splunk/attack_range). Attack Range is a powerful environment automation tool that can set up a full virtual machine-based attack range in about 10 minutes with just four commands. It provisions pre-configured virtual machines (Windows, Linux, optional Kali for simulation) and a fully configured Splunk server, which automatically ingests data sources from all endpoints. This capability allows security teams to:

1. Stand up a consistent, reproducible test environment.
2. Run Atomic Red Team simulations within this environment.
3. Collect the resulting logs directly into Splunk, properly configured for analysis.
4. Customize the environment via Ansible scripts to mirror production configurations, including adding other XDR or endpoint logging providers.

The generated logs from these simulations are critical. Splunk takes this a step further by curating and open-sourcing these logs in Splunk Attack Data. This repository provides raw, pre-generated test data, tagged with the original log source and MITRE ATT&CK ID, allowing users to download and replay attack data directly into their environments without having to run the simulations themselves. This resource is invaluable for teams wanting to test detections quickly or learn from real attack telemetry.

The process of developing and deploying detections is structured around a multi-stage automated pipeline:

1. Validation and Linting: The first step involves rapid, automated checks (taking mere seconds) to ensure content quality. This includes verifying that required fields are present (e.g., risk scores are numbers, reference links are valid), and that the Splunk Processing Language (SPL) or other detection logic is syntactically correct. This immediate feedback helps authors quickly iterate and fix issues.
2. Building: Once validated, the content is automatically packaged into deployable artifacts, such as JSON objects or Splunk apps. This automation prevents manual errors and ensures consistency.
3. Testing: This is where Content CTL, another open-source tool by Splunk (recently released v4.0), comes into play. Content CTL configures the test environment (often an Attack Range instance), replays the relevant test data from Splunk Attack Data, and then executes the detection searches. It verifies not only that the searches run without error but also that they produce the intended results, confirming the detection's efficacy against the simulated attack. The tool is designed to test thousands of detections efficiently.
4. Metrics Generation: The results from the testing phase are compiled into actionable metrics. Key metrics include:

• MITRE ATT&CK Coverage Diagrams: Visual representations showing which ATT&CK techniques are covered by existing detections. This helps identify gaps at a glance.
• Asset Type Coverage: Analyzing detection coverage across different asset types (cloud providers, specific endpoints, applications, network gear) to ensure a balanced defensive posture.
• Trend Analysis: Tracking coverage and test pass rates over time to measure progress, identify degradation, and justify resource allocation.

Deployment of validated content can either be fully automated via CI/CD, directly pushing updates to production environments, or involve a human-in-the-loop review for environments requiring additional oversight. The speakers emphasize that the choice depends on the organization's trust in its CI/CD systems and its risk tolerance.

Finally, the talk highlights the critical importance of tuning and maintenance for the full detection lifecycle. This involves:

• Analyst Feedback: Actively soliciting feedback from SOC analysts (both explicit, like true/false positive labels, and implicit, like detection enablement rates or trouble tickets) to refine and improve detections.
• Continuous Retesting: Regularly retesting detections in fresh environments with the latest versions of applications, operating systems, and SIM platforms to account for evolving data schemas and threat landscapes. This prevents detections from becoming stale or breaking due to environmental changes.

This comprehensive technical framework, supported by open-source tools, allows organizations to move from reactive, manual detection development to a proactive, automated, and continuously validated security posture.

Demo / Proof of Concept

▶ Watch: Essential components: Git, simulations, analytics, automation, metrics (3:40)

While a live, interactive demonstration of the tools in action was not performed during the presentation, the speakers extensively described and illustrated the functionality of their open-source tools and the practical implementation of their detection engineering blueprint. They showcased screenshots and conceptual diagrams to convey how these components work together.

For instance, the talk included a visual representation of the Splunk Attack Range setup, showing a network diagram with pre-configured virtual machines (Windows, Linux, Kali) and a Splunk server ingesting logs. They explained the ease of setting up this environment with just a few commands. Similarly, examples of Git commit messages were shown to illustrate the version control aspect and how CI/CD jobs are triggered.

The speakers also presented a MITRE ATT&CK coverage diagram, a key metric generated by their process, demonstrating how such visualizations provide an "at a thousand feet level" view of detection coverage and identify areas needing improvement. They discussed how this is derived from annotating attacks with MITRE ATT&CK IDs and testing them.

The detailed explanation of tools like Atomic Red Team (describing "atomics" and their tagging) and Content CTL (explaining its role in linting, building, and testing) served as a conceptual proof of concept. The audience was provided with the GitHub URLs for these tools, encouraging them to explore and implement these practices in their own environments. The emphasis was on sharing the methodology and the tools that enable it, rather than a step-by-step live walkthrough.

Defensive Implications

▶ Watch: Implementing detection engineering: specific tools and methodologies (4:50)

The blueprint for detection engineering outlined by Splunk Threat Research offers several profound implications and actionable recommendations for security defenders:

1. Embrace a Git-Driven Workflow for Detection Content: All detection rules, models, and associated metadata should be managed under version control (e.g., Git). This provides historical tracking, enables collaboration, facilitates code reviews, and is foundational for integrating with CI/CD pipelines. This moves detection content from disparate, undocumented locations into a centralized, auditable repository.
2. Implement Automated Simulation and Testing: Defenders must move beyond manual, ad-hoc testing. Leveraging tools like Splunk Attack Range and Atomic Red Team to create and execute realistic attack simulations in a controlled environment is crucial. This allows for rigorous validation of detection logic against known TTPs, ensuring detections work as intended before deployment. This proactive testing helps prove detection efficacy and reduces the risk of false positives or missed threats in production.
3. Integrate CI/CD for Detection Lifecycle Management: Adopt Continuous Integration/Continuous Deployment (CI/CD) principles for detection content. Automated pipelines should perform linting, building, testing, and potentially deployment of detections. This significantly accelerates the feedback loop for detection engineers, catches errors early, and enables rapid iteration and deployment of high-quality detection updates.
4. Prioritize Rich Metadata and Inline Documentation: Detections should be accompanied by comprehensive metadata, including clear descriptions, MITRE ATT&CK mappings, relevant CVEs, risk scores, and links to external resources. This inline documentation is vital for maintainability, analyst understanding, and knowledge transfer within the SOC, reducing reliance on individual "hero" knowledge.
5. Develop a Metrics-Driven Approach to Detection Coverage: Establish and consistently track key metrics such as MITRE ATT&CK coverage and asset-type coverage. These metrics provide a high-level view of an organization's defensive posture, highlight critical gaps, and enable data-driven prioritization of detection development efforts. Tracking trends over time helps demonstrate progress and justify security investments.
6. Foster a Culture of Continuous Feedback and Improvement: Actively solicit feedback from SOC analysts who are on the front lines consuming detection alerts. This explicit and implicit feedback is invaluable for tuning detections, reducing false positives, and ensuring their practical utility. Combine this with regular retesting in updated environments to ensure detections remain effective against evolving threats and changing infrastructure.
7. Leverage Open-Source Tools: Defenders don't need to build everything from scratch. The open-source tools shared by Splunk (Attack Range, Content CTL, Splunk Attack Data) provide a powerful foundation to jumpstart or mature detection engineering practices, allowing teams to focus on customizing and extending rather than reinvention.
8. Shift from Reactive to Proactive Detection: By implementing a robust detection engineering process, organizations can move away from a purely reactive stance (developing detections only after a breach or major vulnerability) towards a more proactive approach, systematically improving coverage and adapting to the threat landscape before incidents occur.

Key Takeaways

• Detection Engineering is Essential for Proving Efficacy: Organizations must adopt detection engineering to reliably prove their ability to detect specific attacks, drastically reducing the time it takes to validate and respond to threats from weeks to minutes.
• Five Core Components Drive Success: A robust detection engineering practice relies on a Git-driven workflow, automated simulations, a capable analytics engine, automation, and actionable metrics.
• Open-Source Tools Accelerate Implementation: Splunk's open-source contributions, including Splunk Attack Range, Atomic Red Team, Splunk Attack Data, and Content CTL, provide practical, ready-to-use solutions for building, testing, and managing detection content.
• Automated Testing and CI/CD are Transformative: Integrating automated validation, building, and testing into a CI/CD pipeline significantly enhances the quality, speed, and confidence of detection content deployment.
• Metrics Guide Strategic Improvement: High-level metrics like MITRE ATT&CK coverage diagrams and asset-type coverage are vital for identifying defensive gaps, prioritizing development, and demonstrating continuous improvement in security posture.
• Continuous Tuning and Feedback are Critical: Detections are not "set and forget." Regular retesting, incorporating analyst feedback, and adapting to evolving threats and data sources are crucial for maintaining detection efficacy over time.

About the Speaker(s)

The talk was presented by Jose and Eric, members of the Splunk Threat Research team. This team is responsible for developing and shipping all security content for Splunk products, including detection rules, playbooks, and security models. Their work involves in-depth study of emerging threats, reverse engineering malware and vulnerabilities (CVEs), and crafting effective detections. The Splunk Threat Research team is committed to transparency, actively open-sourcing their tools and methodologies to empower the broader security community.