Panel: Ctrl-Alt-Detected: Unraveling Threats with Detection Practitioners

Ctrl-Alt-Detected: Unraveling Threats with Detection Practitioners

Speakers: Julia Agnes Sparks (Moderator), Tammy Truong, Christina Devlin, Shannon McCormick

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=fz9zbiPEwu4

Overview

This panel discussion, "Ctrl-Alt-Detected: Unraveling Threats with Detection Practitioners," brought together leading voices in detection engineering from Snowflake, Salesforce, Twitch/NASA/Stripe alumni, and Datadog. Moderated by Julia Agnes Sparks, a Security Researcher at Datadog, the panel featured Tammy Truong (Threat Detection Engineer at Snowflake), Christina Devlin (Senior Security Engineering Manager, formerly at Twitch, NASA, and Stripe), and Shannon McCormick (Detection and Response Engineering at Salesforce). The discussion provided an invaluable deep dive into the practical realities, challenges, and evolving strategies within the field of threat detection and response.

The talk is crucial for anyone involved in cybersecurity, from aspiring detection engineers to seasoned security leaders. It illuminates the intricate balance required to build effective detection programs that not only identify genuine threats but also maintain operational efficiency and team morale. The panelists shared real-world anecdotes, discussed the nuances of data quality, threat modeling, and insider threats, and offered forward-looking insights into the future of the discipline. Their collective experience underscores the importance of a holistic, collaborative, and data-driven approach to safeguarding organizational assets in an increasingly complex threat landscape.

Background

▶ Watch: Panelist introductions and moderator (0:00)

The landscape of cybersecurity has shifted dramatically, moving beyond purely preventative measures to embrace robust detective capabilities. However, building and maintaining effective threat detection systems is fraught with challenges. Organizations grapple with an overwhelming volume of security telemetry, the constant evolution of attacker tactics, and the inherent difficulty in distinguishing malicious activity from benign, yet unusual, behavior. This often leads to alert fatigue among incident response (IR) teams, where a deluge of low-fidelity alerts can desensitize responders and obscure true threats.

Prior work in the field has largely focused on signature-based detections and rule engines, which, while effective against known threats, struggle with novel attacks. The problem is exacerbated by the sheer scale of modern enterprise environments, particularly in cloud infrastructure, where ephemeral assets and dynamic configurations make traditional asset management and logging strategies insufficient. The need for a more sophisticated, context-rich, and adaptable approach to detection engineering has become paramount, moving towards behavioral analytics and risk-based methodologies to overcome these limitations. The panel delved into these persistent problems, offering practitioner-level insights into how their respective organizations are tackling them.

Key Findings

▶ Watch: Shannon's story: unraveling a vague C-suite threat (2:50)

The panel discussion yielded several key findings and practical insights across various facets of detection engineering:

• Balancing Precision and Coverage: Panelists emphasized the tension between writing high-precision detections that minimize false positives and low-precision ones that offer broader coverage, especially for initial access. While high-fidelity alerts are crucial for preventing burnout, low-fidelity alerts can be valuable when combined through risk-based alerting to tell a more complete attack story. This allows for a balance between immediate, critical alerts and lower-severity indicators that contribute to a larger risk score.
• The Criticality of Data Quality and Logging Visibility: Effective detection hinges on comprehensive, high-quality, and well-indexed logging. Panelists highlighted the cost implications of data storage (e.g., Splunk's 30-day limit, terabytes of data) and the need for mindful data collection. Collaborative efforts with engineering teams are essential to define what data is collected, why, and how it's stored. Threat hunting was presented as an effective exercise not only for finding threats but also for validating the logging platform's capabilities, identifying gaps, and improving data fidelity.
• Structured Threat Modeling: For both cloud and on-premise environments, panelists advocated for a structured approach to threat modeling. Key steps include scoping the system or feature, identifying entities (users, services, processes, data stores), mapping data flows, and applying frameworks like STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) to identify threats. The discussion also highlighted the importance of leveraging prior incidents, bug bounty programs, and adversary simulation (red teaming) to inform threat models and prioritize detective and preventative controls.
• Identifying and Protecting Crown Jewels: Determining an organization's most critical assets—its "crown jewels"—is fundamental for prioritizing detection efforts. This requires deep collaboration with business units, IT partners, and even asset management teams to understand what data, systems, and intellectual property are most vital. The challenge lies in accurately mapping these assets, including custom scripts and configuration drift, and integrating this context into a security data lake for effective querying and analysis.
• Contextual Enrichment for Alerts: A recurring theme was the need for rich context within alerts to aid rapid incident response. Panelists envisioned an ideal state where detections are accompanied by comprehensive information, such as links to prior incidents, historical user behavior, and relevant entity details (job title, team, device registration status). This context should ideally be in-line with the logs or readily available in a "single pane of glass," reducing the need for responders to jump between multiple tools for data enrichment.
• Managing the Detection Lifecycle: An effective detection program requires a robust lifecycle that includes continuous feedback loops. This involves integrating incident response teams into detection reviews, enabling them to provide detection improvement requests (DIRs) for tuning or suppression. Automated mechanisms, such as auto-quarantine for alerts that fire too frequently (e.g., "20 times in one hour"), were discussed as critical for managing alert fatigue and ensuring detection engineers address noisy alerts promptly. Metrics on false positives and true positives are essential for demonstrating the value and impact of detection work.
• Nuance of Insider Threat: While many detections for external attackers can apply to insiders, the insider threat space presents unique challenges. Insiders already have legitimate access, making it harder to detect malicious intent. The focus shifts to data exfiltration techniques and deviations from baselined normal behavior. The panel discussed the complexity of distinguishing malicious insider activity from benign, but unusual, actions (e.g., remote work, travel, specific project activities) and the difficulty in scaling automated solutions for highly nuanced behavioral analysis.

Technical Deep Dive

▶ Watch: Panel discusses labor costs of high vs. low precision detections (3:36)

The panel highlighted several technical approaches and concepts crucial for modern detection engineering:

Risk-Based Alerting (RBA): This strategy moves beyond binary "alert/no-alert" decisions. Instead, individual low-fidelity signals are assigned a risk score or severity. When multiple low-severity events related to the same entity (user, host, IP address) occur within a defined timeframe, their risk scores aggregate. If the combined score crosses a threshold, a higher-severity alert is triggered. Tammy Truong specifically mentioned implementing RBA, allowing "lower fidelity detections to be labeled as a lower severity alert and they might not fire atomically but with another activity... can fire together and tell that story about... the attack story" (06:00). Christina Devlin echoed this, envisioning RBA as a way to make detection severity more granular, moving beyond "low, medium, and high" to provide a quantitative score that helps IR prioritize "which one do they look at first?" (50:00). This approach directly combats alert fatigue by reducing the volume of individual, low-value alerts while ensuring that correlated suspicious activities are still flagged.

Data Quality and Logging Pipelines: The foundation of any detection program is the underlying data. Shannon McCormick stressed the financial limitations of data storage, noting that "my Splunk only goes 30 days back. Or, like, I have, like, X terabytes of data I can store here" (10:00). This necessitates careful selection of what data to collect, prioritizing "what data has the highest value in terms of detection and response." The panelists emphasized the need for documented logging strategies, explaining why certain logs are collected or filtered out. Christina Devlin highlighted that detection response "ends up becoming almost a data engineering problem" (14:00) due to reliance on data availability, normalization, and quality from the source. The discussion also touched upon the challenges of cold storage (14:00) and the difficulty of retrieving data for IR purposes.

Advanced Querying and Analysis Platforms: The ability to flexibly analyze logs is paramount. Shannon McCormick mentioned the use of Jupyter Notebooks (11:00) for threat hunting and analysis, indicating a shift towards more programmatic and data-science-driven approaches. The panel also discussed the challenges of load-bearing queries (12:00) that become slow or time out due to backend changes or large data volumes. This underscores the need for optimized data storage as tables and caching (32:00) for frequently run alerts. The concept of inviting data platform engineers or authentication management teams to participate in threat hunts (12:00) was suggested as a way to improve the logging platform's fidelity and usability.

Contextual Enrichment and Single Pane of Glass: A significant technical goal articulated by the panelists is the enrichment of alert data. Tammy Truong expressed a desire for context "in line with the logs," enabling joins between "AWS config table and... CloudTrail event log" (29:00) to present a complete picture to IR without extra lookups. Christina Devlin expanded on this, envisioning "all of that information, like, in one pane of glass" (30:00), including details from IDPs (Identity Providers) and EDRs (Endpoint Detection and Response). This would provide crucial context like job title, team, device registration status, and historical user behavior, making alerts more actionable and reducing IR investigation time.

User and Entity Behavioral Analytics (UEBA) and Baseline Analysis: For detecting insider threats and subtle anomalies, UEBA and baseline analysis were highlighted as critical. Tammy Truong mentioned UEBA as a "passion project" that ties into risk-based alerting, using data science to "look at these like specific indicators and raising my risk score because I am doing like some sus activity that doesn't exactly warrant an alert" (49:00). Shannon McCormick emphasized baseline analysis as a mainstream approach, requiring skills "to take a large volume of data and kind of understand like what normal is and kind of like the outliers" (50:00). This involves understanding typical behavior for different roles (e.g., engineers vs. business analysts), teams, and even during specific projects or travel, to identify true deviations rather than just unusual but benign activity.

Demo / Proof of Concept

▶ Watch: Christina on balancing low/high precision with risk-based alerting (5:20)

As this was a panel discussion focused on sharing experiences and best practices, there was no live technical demonstration or proof of concept. However, the panelists effectively conveyed practical application through their "favorite threat detection stories" and real-world examples. Julia Agnes Sparks shared her "first true positive alert" where a detection for a same factor across multiple accounts successfully caught a red team exercise, validating the detection's value (00:00). Christina Devlin recounted a hunt that uncovered configuration gaps in a ticketing system, triggered by contractors emailing themselves ticket links (02:00). Tammy Truong described a challenging but ultimately satisfying investigation into an anonymous Twitter threat, piecing together evidence from security telemetry to identify an individual of interest (02:00). These anecdotes served as practical illustrations of how effective detection engineering translates into tangible security wins and improved posture.

Defensive Implications

▶ Watch: Shannon on noisy alerts' impact on partner team relationships (6:00)

The insights from this panel provide actionable guidance for defenders looking to enhance their detection capabilities:

1. Prioritize High-Fidelity Detections, but Leverage Risk-Based Alerting: Focus on building detections that minimize false positives to combat alert fatigue and build trust with incident response teams. For lower-fidelity but potentially valuable signals, implement risk-based alerting (RBA) to correlate multiple events and generate higher-severity alerts only when a cumulative risk threshold is met. This allows for broader coverage without overwhelming responders.
2. Invest in Data Quality, Visibility, and Cost Management: Proactively manage logging pipelines. Collaborate with engineering teams to define essential data sources, understand storage limitations and costs (e.g., "Splunk only goes 30 days back" at 10:00), and meticulously document what is logged and why. Regularly conduct threat hunts as a means to validate log completeness and identify gaps, ensuring that necessary telemetry is available when an incident occurs.
3. Implement a Robust Detection Lifecycle with Continuous Feedback: Integrate incident response (IR) teams into every stage of the detection lifecycle, from initial strategy to review and ongoing tuning. Establish formal feedback mechanisms, such as detection improvement requests (DIRs), and consider automated processes like auto-quarantine for noisy alerts (e.g., "20 times in one hour" at 43:00) to ensure timely resolution and prevent burnout. Regularly review detection metrics to track false positives and improve fidelity.
4. Adopt Structured Threat Modeling and Crown Jewel Identification: Systematically threat model critical systems and infrastructure, especially in cloud environments, using frameworks like STRIDE. Work closely with business and IT stakeholders to identify and tag crown jewels—the most critical assets and data. Leverage insights from prior incidents, bug bounty programs, and red team exercises to prioritize where detection efforts should be concentrated.
5. Strive for Contextual Alert Enrichment: Aim to provide as much context as possible directly within alerts or in a unified "single pane of glass." Integrate data from Identity Providers (IDPs), Endpoint Detection and Response (EDRs), and asset management systems to enrich logs with details about users, devices, and their normal behavior. This empowers IR teams to quickly understand the scope and severity of an alert, reducing investigation time.
6. Develop Baseline Analysis and UEBA Capabilities for Insider Threat: Recognize that insider threats require a nuanced approach. Invest in tools and skills for baseline analysis to understand "normal" user and entity behavior across different roles and teams. Focus on detecting data exfiltration techniques and significant deviations from established baselines, acknowledging the complexity of scaling these detections across large, dynamic organizations.
7. Leverage Threat Intelligence and Collaboration: Actively consume and integrate threat intelligence (TI), prioritizing information relevant to your industry and known threat actors targeting peer organizations. Foster collaboration within the security community and with internal partner teams (e.g., data platform, infrastructure, AppSec) to share knowledge, improve platforms, and collectively enhance defensive posture.

Key Takeaways

• Prioritize High-Fidelity Detections: While broad coverage is important, minimizing alert fatigue through high-precision detections and effective use of risk-based alerting is crucial for IR team effectiveness and morale.
• Data is Foundational: Comprehensive, high-quality, and well-managed logging is non-negotiable. Organizations must be mindful of data storage costs, document logging strategies, and use threat hunting to validate visibility and identify gaps.
• Context is King for Response: Enriching alerts with data from IDPs, EDRs, and asset management, ideally presented in a "single pane of glass," significantly speeds up incident response and provides crucial context.
• Implement a Robust Detection Lifecycle: Integrate IR feedback into detection development, utilize mechanisms like auto-quarantine for noisy alerts, and conduct regular reviews to continuously improve detection accuracy and relevance.
• Threat Modeling and Crown Jewels Guide Strategy: Systematically identify and prioritize critical assets ("crown jewels") through threat modeling (e.g., STRIDE) to ensure detection efforts are focused on the highest-impact areas.
• Nuance is Key for Insider Threat: Detecting insider threats requires sophisticated baseline analysis and User and Entity Behavioral Analytics (UEBA) to distinguish malicious deviations from normal, legitimate activities.

About the Speaker(s)

• Julia Agnes Sparks (Moderator): A Security Researcher at Datadog, Julia skillfully guided the panel discussion, posing insightful questions that delved into the practical challenges and future directions of detection engineering.
• Tammy Truong: A Threat Detection Engineer at Snowflake, Tammy shared her expertise on developing and implementing effective detection strategies, including her experiences with risk-based alerting and investigating vague threat reports.
• Christina Devlin: As a Senior Security Engineering Manager with a rich background at companies like Twitch, NASA, and Stripe, Christina provided valuable insights into managing security engineering teams and the broader challenges of data quality and threat modeling in diverse environments.
• Shannon McCormick: Working in Detection and Response Engineering at Salesforce, Shannon contributed perspectives on the operational aspects of detection, emphasizing the importance of data cost management, logging visibility, and fostering strong relationships with partner teams.