Beyond Vibe Coding: Building Reliable AI AppSec Tools

Beyond Vibe Coding: Building Reliable AI AppSec Tools

Speakers: Emily Choi-Greene

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=iI5u099MrZY

Overview

In "Beyond Vibe Coding: Building Reliable AI AppSec Tools," Emily Choi-Greene delves into the critical intersection of artificial intelligence and application security, advocating for the strategic application of AI to solve pressing security challenges. While acknowledging the pervasive "AI hype," Choi-Greene emphasizes that Large Language Models (LLMs) are powerful tools when applied appropriately, particularly in the realm of AppSec. The talk addresses the burgeoning landscape of software development, where AI-driven "vibe coding" makes software creation more accessible than ever, paradoxically leading to a surge in potential security vulnerabilities.

The core thesis of the presentation is that security professionals must embrace AI not as a replacement for human judgment, but as an indispensable aid for scaling security efforts and tackling complex, context-rich problems. Choi-Greene highlights the limitations of naive AI approaches and outlines a robust, engineered methodology for building reliable, production-grade AI-powered AppSec tools. This talk is crucial for security practitioners, developers, and leadership seeking to leverage AI effectively to enhance their security posture in an increasingly AI-driven development world.

Background

▶ Watch: Introduction: Applying AI to AppSec challenges (0:00)

The rapid evolution of AI, particularly Large Language Models (LLMs), has dramatically lowered the barrier to entry for software development. This phenomenon, dubbed "vibe coding" by the speaker, allows individuals with varying levels of expertise to generate code quickly. While this accessibility is empowering, it introduces a significant challenge: a corresponding increase in software vulnerabilities. Choi-Greene invokes the Jevons Paradox, explaining that increased supply (more AI-generated code) leads to increased demand (more need for securing that AI-generated code). This escalating need necessitates a scalable, efficient approach to application security, positioning AI as a potential solution.

Security tasks possess several characteristics that make them uniquely suited for AI application. They often require understanding diverse context types, including codebases, documents, images, and rich content from multiple sources. LLMs excel at processing and comparing these varied inputs, such as reconciling initial design decisions with final implementation details. Furthermore, security workflows often involve transforming complex context into normalized, structured outputs—like converting threat findings into a vulnerability management system. LLMs, as computational engines, can handle the scale and repetition inherent in these tasks, consistently applying well-defined workflows and frameworks like STRIDE without fatigue or bias.

However, Choi-Greene also delineates crucial scenarios where AI is inappropriate. AI should not be used when goals and outcomes are unclear, as models require explicit definitions of "done." Similarly, situations demanding significant business judgment, major trade-offs with no clear right answer, or an absolute intolerance for probabilistic outcomes (i.e., requiring 100% correctness) are best left to human decision-makers or deterministic code. For security, where "multiple nines of reliability, accuracy, and consistency" are paramount, relying solely on probabilistic AI outputs without an engineered system is insufficient.

The talk then narrows its focus to threat modeling as a prime example for AI application. A naive approach of simply dumping an architecture diagram into a generic LLM like ChatGPT yields superficial results, failing to provide the structured, contextual, and methodology-driven analysis required for effective threat modeling within an enterprise environment. This highlights the necessity of building sophisticated AI systems that can integrate deep organizational context and adhere to established security processes.

Key Findings

▶ Watch: Naive approach: ChatGPT for threat modeling (insufficient) (4:40)

The talk's central findings revolve around overcoming the limitations of basic LLM interactions to build reliable, production-grade AI AppSec tools. Choi-Greene reveals that while simple LLM queries are insufficient, sophisticated techniques can unlock significant value.

Firstly, the presentation critiques the limitations of basic Retrieval Augmented Generation (RAG). Traditional RAG, which relies on chunking source material into a vector store and performing semantic searches, is highly dependent on the quality of chunking and the embedding model used. Generic embeddings can map specialized security concepts too closely, leading to irrelevant results. Critically, basic RAG requires prior knowledge of what information to search for, limiting its proactive contextualization capabilities.

This leads to the second key finding: the superiority of tool use and function calling, which Choi-Greene terms RAG 2.0. This approach allows LLMs to programmatically interact with diverse internal systems, web APIs, and other agents. By enabling the LLM to call specific tools, it can dynamically gather a much richer and more relevant context for security tasks, moving beyond mere semantic similarity to actual data retrieval from enterprise systems. This expanded context is vital for comprehensive threat modeling, allowing the AI to consider elements like related Jira tickets, organizational policies, and specific regulatory requirements (e.g., the EU Radio Equipment Directive).

Thirdly, the talk underscores the critical need for production-grade reliability in AI AppSec tools. Unlike a proof-of-concept (PoC) that might tolerate a 5% failure rate (e.g., for JSON output), security systems demand "multiple nines" of reliability, accuracy, and consistency. A 5% failure rate in opening findings or creating tickets is catastrophic for security operations. This means an engineered system must surround the LLM to handle its inherent probabilistic nature and ensure consistent, structured output.

Finally, Choi-Greene identifies specific strategies to build reliability and combat common LLM weaknesses like hallucinations and inconsistent output. These include:

• Chain of Thought reasoning: Encouraging the LLM to explain its steps, which improves accuracy and provides observability.
• Source Citation: Requiring the LLM to link its assertions directly back to original sources, verifying information and preventing fabrication.
• Prompt Engineering: Explicitly instructing the LLM to acknowledge when information is missing or unknown, rather than generating plausible but incorrect answers.
• LLM as Judge: Using one LLM to evaluate the output or reasoning of another, creating a self-correcting mechanism.

These findings collectively present a roadmap for moving beyond speculative AI applications to robust, trustworthy solutions for critical security functions.

Technical Deep Dive

▶ Watch: Building context with traditional Retrieval Augmented Generation (RAG) (5:40)

The technical foundation of building reliable AI AppSec tools, as presented, extends beyond simple LLM prompting to sophisticated Retrieval Augmented Generation (RAG) techniques and robust reliability engineering.

At its core, RAG combines the generative power of LLMs with a retrieval mechanism to fetch relevant information from a knowledge base. The traditional RAG pipeline involves a query to an LLM, followed by a retrieval component (often a vector store containing embeddings of chunked source material), a generative component that synthesizes the retrieved information, and finally, the response.

However, Choi-Greene highlights significant limitations of this basic RAG approach. The efficacy of the retrieval is heavily dependent on how source materials are chunked and the quality of the embeddings model. If chunks are too large or too small, or if the embedding model is generic, the semantic search might return irrelevant or incomplete snippets. For specialized domains like security, where terms might cluster closely in a generic vector space, the results can be highly imprecise. Furthermore, basic RAG requires the system to know what to search for in advance, necessitating extensive pre-processing and limiting dynamic context discovery.

To overcome these challenges, the talk advocates for Tool Use and Function Calling, which it characterizes as RAG 2.0. This advanced approach transforms the LLM from a passive information synthesizer into an active agent capable of interacting with external systems. Instead of merely searching a static vector store, the LLM can programmatically invoke APIs to:

• Search internal enterprise systems (e.g., a Jira instance for related development tickets).
• Query the web for external information.
• Call other specialized AI agents.
• Access specific databases containing organizational policies, compliance requirements (such as the EU Radio Equipment Directive for devices launched in Europe), or design documents.

This dynamic interaction allows the LLM to build a far richer and more pertinent context for a given security task, such as threat modeling. For instance, when analyzing an architecture diagram, an LLM equipped with tool-use capabilities can pull in not only system components but also:

• Relevant development tickets detailing ongoing work or known issues.
• Information about the submitting team or individual, their organizational context.
• Company-wide security policies and procedures.
• Specific regulatory requirements applicable to the system or its deployment environment.

This comprehensive, dynamically generated context enables the AI to produce a more accurate and actionable threat model, including contextual diagrams, data flow analysis, severity ratings based on internal policies, and documentation of existing mitigations.

A critical technical challenge in moving these AI applications to production is ensuring structured output. LLMs are probabilistic models; even when instructed to output JSON, they might fail to do so perfectly in a small percentage of cases (e.g., 5%). While this might be acceptable for a casual chatbot, it is disastrous for a security system where findings must be reliably ingested into a threat database or vulnerability management system. Therefore, an engineered system must wrap the LLM, implementing robust parsing, validation, and error handling mechanisms to guarantee the integrity and consistency of the output. This involves potentially re-prompting, using multiple LLMs, or implementing deterministic fallback logic.

To further bolster reliability and combat LLM hallucinations—where the model fabricates information—Choi-Greene details several key techniques:

1. Chain of Thought Reasoning: This technique involves instructing the LLM to articulate its reasoning process step-by-step before arriving at a final answer. By generating intermediate thoughts, the LLM is more likely to arrive at a correct conclusion, similar to a student showing their work in mathematics. This also enhances observability, allowing human operators to understand the AI's decision-making.
2. Source Citation: The LLM is explicitly prompted to cite the exact source material for every piece of information it provides. This verifiable linkage back to the original context (e.g., a specific document, policy, or code snippet) directly prevents hallucinations and ensures factual accuracy.
3. Prompt Engineering for "I Don't Know": Baseline LLMs are often reinforced to be "helpful assistants," leading them to generate answers even when they lack sufficient information. Effective prompt engineering involves granting the LLM explicit permission to state when it "does not know" or when information is "not present." This prevents the model from fabricating plausible but incorrect details, crucial for identifying gaps in security posture (e.g., "encryption at rest isn't specified").
4. LLM as Judge: This advanced technique involves employing a second LLM to evaluate the output or reasoning of the primary LLM. The "judge" LLM can assess correctness, completeness, adherence to instructions, or even identify potential hallucinations, providing a layer of self-correction and quality control within the AI system.

These techniques collectively form the bedrock of building AI AppSec tools that are not just intelligent but also dependable and trustworthy in high-stakes security environments.

Demo / Proof of Concept

▶ Watch: Key limitations of basic RAG for security applications (6:20)

Emily Choi-Greene mentioned an intention to demonstrate the concept of LLM as Judge. She indicated that setting up this particular demo was time-consuming, suggesting its technical complexity. The transcript concludes precisely as she prepares to initiate this demonstration, stating, "I actually have a demo for this and that was what took a long time to try to set up properly. So hopefully this works. Um but I'm going to basically show you how we do LM as Judge at Clearly um to give you some ideas of how you can use LMS to actually help make your LM processes better." While the specifics of the demo's execution are not captured in the provided transcript, the goal was to illustrate how one LLM could be leveraged to evaluate and improve the output or reasoning of another LLM, thereby enhancing the overall reliability and accuracy of AI-driven security processes. This would have visually reinforced the practical application of advanced reliability strategies discussed in the talk.

Defensive Implications

▶ Watch: Moving beyond RAG: embracing tool use and AI agents (8:00)

The insights presented in this talk have profound implications for security defenders grappling with the escalating volume and complexity of software development, especially that driven by AI.

Firstly, defenders must embrace AI as a force multiplier for AppSec. With the "Jevons Paradox" in full effect—more software producers and code leading to more vulnerabilities—human security experts alone cannot keep pace. AI tools, when properly engineered, offer the scalability and consistency needed to address this growing demand, allowing human experts to focus on the most critical and nuanced issues.

Secondly, the talk serves as a critical warning against naive adoption of LLMs for security tasks. Simply dumping sensitive architecture diagrams or code into a generic LLM like ChatGPT will yield inadequate, unstructured, and potentially inaccurate results. Defenders should be highly skeptical of "vibe coding" approaches to security and instead demand robust, context-aware, and methodology-driven AI solutions.

Thirdly, security teams should prioritize the development or procurement of AI AppSec tools that leverage advanced RAG techniques, specifically tool use and function calling. This means looking for solutions that can integrate deeply with enterprise-specific context, such as internal Jira instances for development tickets, organizational policies, and relevant regulatory frameworks (e.g., GDPR or EU Radio Equipment Directive). Contextual richness is paramount for accurate threat modeling, vulnerability analysis, and compliance checks.

Fourthly, reliability and consistency are non-negotiable for AI in security. Defenders must insist on "multiple nines of reliability" for any AI-powered system that impacts their security posture. This requires asking vendors or internal development teams about their strategies for mitigating LLM hallucinations, ensuring structured output, and guaranteeing consistency. Techniques like Chain of Thought reasoning, Source Citation, and sophisticated Prompt Engineering are critical indicators of a mature AI AppSec solution.

Finally, defenders need to actively define clear goals and outcomes for AI-driven security processes. They should understand when AI is an appropriate tool (e.g., for repetitive, context-heavy analysis) and when it is not (e.g., for major business judgment or tasks requiring absolute determinism). By understanding these boundaries and advocating for robust engineering practices, security professionals can harness the power of AI to build a more resilient and secure software ecosystem.

Key Takeaways

• The "Vibe Coding" Paradox: AI-driven software creation makes development more accessible but simultaneously escalates the volume of potential security vulnerabilities, creating an urgent need for scalable AppSec solutions.
• AI's Fit for AppSec: Large Language Models (LLMs) are well-suited for many security tasks that require understanding diverse contexts, comparing design to implementation, transforming information into normalized outputs, and executing consistent, repeatable workflows at scale.
• Beyond Naive LLM Use: Simple LLM prompting (e.g., feeding an architecture diagram to ChatGPT) is insufficient for complex, structured security processes like threat modeling, which demand enterprise-specific context and methodology.
• Advanced RAG for Context: Tool use and function calling (termed RAG 2.0) are crucial for building rich, dynamic context by allowing LLMs to programmatically interact with internal systems (e.g., Jira, policy databases) and external APIs, moving beyond basic semantic search.
• Engineered Reliability is Paramount: Production-grade AI AppSec tools require an engineered system to ensure "multiple nines" of reliability, accuracy, and consistency, mitigating the probabilistic nature of LLMs and their tendency to hallucinate.
• Strategies to Combat Hallucinations: Key techniques for building reliable AI systems include Chain of Thought reasoning (showing work), Source Citation (linking to original sources), Prompt Engineering (explicitly allowing the LLM to state "I don't know"), and LLM as Judge (using one LLM to evaluate another).

About the Speaker(s)

Emily Choi-Greene is a speaker who is deeply engaged in the practical application of AI to solve complex security challenges. Based on her presentation, she is involved in developing and implementing advanced AI systems, specifically focusing on enhancing the reliability and effectiveness of LLMs for tasks such as threat modeling. Her reference to demonstrating "how we do LM as Judge at Clearly" suggests her affiliation with a company named Clearly, where she applies these cutting-edge techniques to build production-grade AI AppSec tools.