Please Don't Discard - Security Data

Please Don't Discard - Security Data

Speakers: Rishabh Gupta, Hrushikesh Paralikar

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=W9jLMqrItM0

Overview

This article delves into a novel approach for tackling security review challenges by systematically persisting, storing, and structuring security data. Presented by Rishabh Gupta, a Senior Security Engineer, and Hrushikesh Paralikar, a Senior Software Engineer, both from Snowflake and working on anti-abuse, the talk highlights how this methodology, combined with Natural Language Processing (NLP), can effectively identify and address complex security threats, particularly abuse scenarios. The core problem addressed is the difficulty in answering critical questions like "which systems are affected by abuse threats?" within large organizations, a task often daunting due given the multitude of systems and threat types.

The traditional landscape of security reviews is often characterized by unstructured, informal documentation, rarely stored in a centralized or queryable manner. Furthermore, threat modeling tends to be non-deterministic, relying heavily on individual security knowledge, making it challenging to scale and achieve consistent results, especially when aiming for a developer-driven security program. The speakers demonstrate how a structured approach to data collection, coupled with a repeatable threat modeling process and the judicious application of Large Language Models (LLMs), transforms security reviews from an art into a more scientific, scalable, and automated discipline.

The significance of this work lies in its ability to empower developers to conduct more effective security reviews with reduced friction, while simultaneously enabling security teams to gain deeper insights into their systems' vulnerabilities. By making security data machine-parsable and human-readable, and by leveraging modern AI techniques, the presented solution not only streamlines the identification of known threats but also uncovers latent and emerging threat classes, such as compute and network abuse, that might otherwise be overlooked. This shift towards data-driven security intelligence represents a crucial step in building more resilient and secure software development lifecycles.

Background

▶ Watch: Initial problem: Identifying systems affected by abuse threats. (01:00)

The impetus for this work stemmed from the inherent challenges in scaling security reviews, particularly within a developer-driven model. The speakers identified several key pain points. Firstly, security reviews often manifest as unstructured documents—such as Confluence pages or design documents—which are informal, have a short lifespan, and are rarely stored in a way that allows for easy retrieval or analysis. This makes it difficult for security engineers to track how specific developer-led security reviews impact the overall security posture of a system.

Secondly, threat modeling, a critical component of security reviews, is inherently non-deterministic. It often relies on the individual security knowledge and experience of the developer, making it more of an "art" than a "science." This subjectivity leads to inconsistent and irreproducible results, posing a significant hurdle when attempting to achieve a standardized security baseline across an organization. Developers, on their part, are often burdened by additional tasks and seek fast, low-friction processes, not activities that consume days or weeks of their time. This tension between the need for thorough security reviews and the desire for rapid development cycles necessitated a new approach.

To address these challenges, the team at Snowflake made several foundational decisions regarding data persistence and structuring. The first crucial choice was to store security review data in GitHub, consciously placing it alongside the source code it pertained to. This decision was driven by the observation that source code and test cases tend to "live forever," unlike design documents or Confluence pages that often get misplaced or forgotten. The data was structured using specific formats: Markdown for risk assessments and threat modeling data, XML for Data Flow Diagrams (DFDs) generated from tools like draw.io, and Gherkin test cases for the output of threat modeling—specifically, the identified threats and their corresponding mitigations. These formats were selected for their dual benefits: ease of machine parsing and querying, as well as reasonable human readability.

Initially, the team faced difficulties with the traditional STRIDE per element approach to threat modeling. STRIDE, an acronym for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privileges, proved to be too time-consuming and inefficient for developers. Asking developers to consider each STRIDE threat for every node or element in a DFD, and then determine applicability, was overly complex and highly subjective, depending on their individual security expertise. This led to the adoption of Rapid Threat Modeling Prototyping (RTMP). RTMP significantly simplifies the process by requiring only a DFD as input from developers. Based on this DFD, the RTMP approach can automatically generate approximately 80% of the applicable threats, making the process repeatable, reproducible, and yielding consistent output.

The DFDs, once created, are represented as directed graphs comprising nodes and edges. Both nodes and edges are enriched with properties, such as flow direction and trust scores. This graph data is then persisted into two tables—one for nodes and one for edges—along with all their associated properties. This structured representation of security review data, accumulated over years, became the fundamental basis for building the advanced abuse detection and analysis capabilities discussed in the talk.

Key Findings

▶ Watch: RTMP Introduction: Automating threat modeling for developers. (06:00)

The core contribution of this work lies in demonstrating how a systematic approach to security data management, combined with modern analytical techniques, can fundamentally transform security reviews. Several key findings emerged from this research:

Firstly, the structured persistence of security review data is paramount. By storing DFDs, risk assessments, threats, and mitigations in machine-parsable formats like Markdown, XML, and Gherkin, and representing them as graph data (nodes and edges with properties) in relational tables, the data becomes amenable to automated analysis. This move away from unstructured documents to a queryable dataset is a foundational enabler for advanced security insights.

Secondly, the adoption of Rapid Threat Modeling Prototyping (RTMP) significantly simplifies and standardizes the threat modeling process. By providing developers with a repeatable methodology that requires only a DFD as input and automatically generates a substantial portion of applicable threats (around 80%), RTMP reduces the non-determinism inherent in traditional threat modeling. This ensures more consistent and reproducible security review outcomes across different development teams and projects.

Thirdly, Large Language Models (LLMs) serve as a critical bridge between high-level security concepts and the granular, structured data. The ability of LLMs to interpret natural language descriptions—such as "nodes that can execute code" or "incoming edge indicates code execution"—and translate them into boolean conditions (yes/no, true/false) that can be integrated into SQL queries is a powerful innovation. This allows for the automated identification of complex threat scenarios that would be difficult to define purely through static rules or manual analysis.

Fourthly, the methodology enables the identification of new and complex threat classes, specifically compute and network abuse, by "retrofitting" existing threat models. By applying LLM-assisted queries to the structured graph data, the system can pinpoint specific nodes or flows vulnerable to these abuse types. This demonstrates the extensibility of the framework beyond traditional STRIDE threats to address evolving attack vectors.

Finally, a surprising and valuable finding was the tool's ability to validate the accuracy of existing DFDs. The speakers noted a strong correlation: if a node was identified as susceptible to compute abuse, it almost invariably also had an associated elevation of privilege threat. In cases where compute abuse was detected but elevation of privilege was absent, 100% of the time, the underlying DFD was found to be incorrect and required correction. This indicates that the analytical framework can serve as a powerful sanity check, improving the quality and accuracy of the foundational security documentation itself.

Technical Deep Dive

▶ Watch: SQL for Threats: Using SQL queries on persisted graph data to identify STRIDE... (11:00)

The technical foundation of this approach rests on two main pillars: the Rapid Threat Modeling Prototyping (RTMP) methodology for generating structured threat data, and the subsequent analysis of this data using SQL queries augmented by Large Language Models (LLMs) for identifying specific abuse threats.

The RTMP process begins with a developer providing a Data Flow Diagram (DFD). This DFD is then subjected to a structured analysis to assign trust zones and apply simplified STRIDE rules.

1. Trust Zone Assignment:

• Any element (node) that is untrusted or exposed to the open internet is assigned a trust score of zero (0).
• Any trusted node directly connected to a node with a trust score of zero is assigned a trust score of one (1).
• Data sinks, such as databases, are consistently assigned a trust score of nine (9).
• All other nodes in between receive a subjective trust score based on their position and importance within the DFD. Nodes closer to data sinks generally receive higher scores, while those closer to the internet edge receive lower scores.

1. Simplified STRIDE Rules Application: Once trust zones are established, a set of five simplified rules are applied to identify potential STRIDE threats:

• Elevation of Privilege (E): This threat is identified when data flows from a node in a lower trust zone to a node in a higher trust zone.
• Spoofing (S) and Denial of Service (D): If a node with a trust score of zero is connected to a node with a non-zero trust score (e.g., one), the destination node is assigned a Spoofing (S) threat, and the connecting edge is assigned a Denial of Service (D) threat.
• Tampering (T): When data flows from a lower trust zone to a higher trust zone, the connecting edge is assigned a Tampering (T) threat.
• Repudiation (R): If an element (node) has a Spoofing (S) threat and a connecting edge has a Tampering (T) threat, the destination node is assigned a Repudiation (R) threat.
• Information Disclosure (I): This threat occurs when data flows from a node in a higher trust zone to a node in a lower trust zone, which logically represents sensitive information moving to a less trusted environment.

The DFDs, with their assigned trust zones and identified STRIDE threats, are then represented as directed graphs. This graph data, comprising nodes and edges along with all their properties (e.g., trust scores, flow directions, identified threats), is persisted into two relational tables. This structured storage allows for efficient querying using standard SQL. For instance, identifying a Spoofing threat can be boiled down to a SQL query with WHERE conditions checking for data flowing from trust_zone = 0 to trust_zone = 1. Similarly, other STRIDE threats can be identified through specific SQL queries based on trust zone comparisons and flow directions.

The innovative aspect of this system comes into play when identifying more complex, emerging threats like compute abuse and network abuse through "retrofit modeling." The challenge here is that concepts like "a node that can execute code" or "an incoming edge indicates code execution" are not directly quantifiable in SQL. This is where LLMs are leveraged.

The process for identifying abuse threats involves:

1. Defining Abuse Scenarios: The talk specifically focuses on compute abuse (running malicious code on a compute node, e.g., a virtual warehouse at Snowflake) and network abuse (running code for network attacks or facilitating incoming attacks, e.g., a reverse shell).
2. LLM-Assisted Query Generation: Instead of trying to hardcode definitions for "code execution" or "shared code," LLMs are used to interpret the natural language descriptions of nodes and edges. Prompts are carefully crafted to elicit a "yes" or "no" (or true/false) response from the LLM. For example, a prompt might be: "Given the name of node X, answer only in yes or no: does this node indicate code execution?"
3. Integrating LLM Output into SQL: The boolean output from the LLM is then used directly within the WHERE clauses of SQL queries.

• For Compute Abuse: The system queries for threat models where node_can_execute_code (determined by LLM), an incoming_edge_indicates_code_execution (determined by LLM), and the code is shared (e.g., from a marketplace, also determined by LLM).
• For Network Abuse: The system queries for threat models where node_can_execute_code (determined by LLM) and there is either an outgoing_edge_to_the_open_internet or an incoming_edge_from_the_open_internet. The "open internet" condition is identified by checking for nodes with a trust_zone of zero.

This hybrid approach, combining structured graph data, SQL querying, and LLM-driven semantic understanding, allows the system to dynamically identify and flag complex abuse threats that would be difficult to detect using traditional, rule-based methods alone. The ability to boil down complex security questions into simple "yes/no" LLM responses that feed into SQL conditions is the key technical innovation enabling this "retrofit modeling."

Demo / Proof of Concept

▶ Watch: LLM for Abuse Detection: Leveraging LLMs to interpret node properties for abu... (13:00)

Hrushikesh Paralikar provided a compelling walkthrough from a developer's perspective, illustrating the practical application and benefits of the system. He began by outlining the common frustrations and challenges developers face when confronted with traditional security review processes. These include the need to constantly revise threat modeling knowledge, struggling to understand complex concepts like "repudiation," difficulty in determining appropriate mitigations, and the iterative nature of threat modeling often being neglected due to project pressures. Developers frequently express sentiments like "it works on my machine," "why do you need to review this small feature?", or the classic "we'll address this later," often stemming from a lack of security knowledge, difficulty in thinking about mitigations, and an absence of an attacker's perspective.

The demo then showcased how the LLM-assisted retrofit threat modeling approach directly addresses these pain points.

1. Initiating Abuse Risk Identification: A developer, working on a new feature, would select the type of abuse they want to check for, for instance, "compute abuse."
2. Automated Analysis: In the background, the system processes all existing property graphs (nodes, edges, and their properties) using a combination of directional flow analysis and NLP. This process generates candidate threats related to the selected abuse type.
3. Threat Visualization and Identification: The demo presented a sample threat model involving a provider, a consumer installing applications built by the provider, and a compute node (specifically, a virtual warehouse at Snowflake). The tool visually highlighted the compute node in red, indicating its vulnerability to compute abuse.
4. Contextual Threat Information: Alongside the visual identification, the tool generated a tooltip providing a clear explanation of what the identified threat actually means. For the provider/consumer scenario, it specifically suggested "Crypt Miner" as a relevant use case that needed to be addressed, demonstrating its ability to provide context-specific threat intelligence.
5. Mitigation Suggestions: Upon clicking "show mitigations," the tool presented a list of possible defensive measures. This directly tackles the developer's struggle with identifying appropriate mitigations, offering concrete guidance.
6. Validation and Resolution: The developer would then implement the suggested mitigations and, crucially, create Gherkin test cases for them. These Gherkin scenarios are designed to become executable test cases, allowing for automated validation. Once the issue is resolved and validated, the visual representation of the vulnerable node on the graph would change from red to blue, signifying that the threat has been addressed.

This demonstration effectively proved that by leveraging existing security data, the system can proactively identify latent abuse cases, suggest relevant risks, propose actionable mitigations, and provide a clear mechanism for validating their resolution. It transforms a complex, manual, and often overlooked process into an integrated, guided, and verifiable part of the development workflow.

Defensive Implications

▶ Watch: DFD Correction Insight: System's ability to identify and correct flawed DFDs. (25:00)

The methodology presented by Rishabh Gupta and Hrushikesh Paralikar offers several significant defensive implications for organizations seeking to enhance their security posture and streamline their development processes.

Firstly, the ability to proactively identify abuse threats (compute and network abuse) in both new and existing systems is a game-changer. By retrofitting existing threat models with LLM-assisted queries, security teams can uncover vulnerabilities that traditional, static analysis or manual reviews might miss. This moves organizations from a reactive to a more proactive stance against evolving threat landscapes.

Secondly, the approach empowers developers to take greater ownership of security. By providing a structured, repeatable, and less burdensome threat modeling process (RTMP), developers are more likely to engage with security reviews effectively. The reduction in friction, coupled with automated threat identification and mitigation suggestions, means security considerations are integrated earlier into the development lifecycle, leading to "shift-left" security benefits.

Thirdly, the system fosters a consistent security posture across the organization. By reducing the non-determinism inherent in traditional threat modeling, the RTMP approach ensures that security reviews yield more uniform and reproducible results, regardless of the individual developer's security expertise. This standardization is crucial for maintaining a baseline level of security across diverse projects and teams.

Fourthly, the automated mitigation suggestions and the generation of Gherkin test cases provide actionable intelligence for defenders. Developers are not just told about a threat but are guided on how to fix it, and the Gherkin tests ensure that the fixes are verifiable and remain effective over time. This closes the loop on vulnerability management, moving beyond identification to validated resolution.

Fifthly, the accumulation of structured security review data creates a data-driven security intelligence platform. This historical data can be analyzed to identify common vulnerabilities, track the effectiveness of mitigations, and refine threat models over time. It allows security teams to make informed decisions based on empirical evidence rather than anecdotal experience.

Finally, a particularly insightful defensive implication is the system's capacity for Data Flow Diagram (DFD) validation. The observation that a compute abuse threat without an accompanying elevation of privilege threat almost always indicates an error in the DFD provides a powerful self-correction mechanism. This means the tool not only identifies threats but also helps ensure the accuracy of the foundational architectural documentation, which is critical for effective security analysis.

Key Takeaways

• Structured Security Data is Gold: Persisting and structuring security review data (DFDs, threats, mitigations) in machine-readable formats (Markdown, XML, Gherkin) is crucial. This transforms unstructured information into a queryable asset, enabling advanced analysis and automation.
• RTMP Streamlines Threat Modeling: Rapid Threat Modeling Prototyping (RTMP) significantly simplifies and standardizes the threat modeling process for developers. By requiring only a DFD and automatically generating approximately 80% of applicable threats, it makes security reviews repeatable, reproducible, and less reliant on individual security expertise.
• LLMs Bridge Semantic Gaps: Large Language Models (LLMs) are instrumental in translating complex natural language security concepts (e.g., "node that can execute code") into actionable boolean conditions for querying structured security data. This innovation allows for the automated identification of nuanced and emerging threat scenarios.
• Retrofit Modeling for New Threats: Existing threat models can be "retrofitted" with LLM-assisted queries to identify new classes of threats, such as compute and network abuse. This extends the utility of historical security data to address evolving attack vectors without requiring a complete re-evaluation.
• Automated Mitigation and Validation: The system can suggest specific mitigations for identified threats and generate Gherkin test cases. This not only guides developers in addressing vulnerabilities but also enables automated validation, ensuring that security issues are effectively resolved and remain fixed.
• Empowering Developers: By reducing friction, providing clear guidance, and automating significant portions of the security review process, this approach empowers developers to integrate security into their workflow more effectively, leading to earlier and more consistent threat identification and resolution.

About the Speaker(s)

Rishabh Gupta is a Senior Security Engineer at Snowflake. His work focuses on anti-abuse, leveraging his expertise to build systems that protect against malicious activities.

Hrushikesh Paralikar is a Senior Software Engineer at Snowflake. He also works on anti-abuse initiatives, contributing his software engineering skills to develop robust security solutions.

Both speakers are actively involved in addressing security challenges at Snowflake, particularly in the domain of anti-abuse, and their presentation reflects their practical experience in building scalable and effective security programs.