Guardrails beyond Vibes: Shipping Security Agents in Production

Guardrails Beyond Vibes: Shipping Security Agents in Production

Speakers: Jeffrey Zhang, Security Engineer, Stripe; Siddh Shah, Software Engineer, Stripe

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=KrKk8BGPeQA

Reading time: ~9 minutes

TL;DR

Stripe's security engineering team replaced ad hoc "vibe checks" with a rigorous engineering discipline for deploying AI security agents in production. By combining modular multi-agent architectures with a golden-standard evaluation pipeline using LLM-as-judge, they shipped a threat modeling agent and a security routing agent with measurable accuracy and the confidence to iterate rapidly. Their core lesson: every meaningful quality improvement came from systematic evaluation, not intuition.

Introduction

Security teams everywhere face the same arithmetic problem: too many security reviews, not enough human time. For Stripe, which operates multiple specialized security teams and fields requests from developers across the company, that problem is compounded by complexity. A developer who needs security guidance has to find the right team, submit the right information, and wait for a human to review it. At scale, this creates a bottleneck that slows product development and leaves security guidance arriving too late to be useful.

Jeffrey Zhang (security engineer) and Siddh Shah (software engineer) tackled this with two AI agents: a threat modeling agent that automates the generation of security risk assessments, and a security routing agent that directs developer questions to the right security team. Their talk at [un]prompted 2026 went beyond describing the agents to give a detailed account of how they actually shipped them — the architectural decisions, the evaluation methodology, and the lessons they would not repeat.

▶ Watch: Introduction and Problem Statement (00:00)

The Two Problems and the Shared Challenge

Both agents address structurally different problems, which drove different architectural choices. But both share a fundamental challenge: hallucinations.

Security routing. Stripe has many specialized security teams. Getting a developer to the right one requires understanding the nature of their question, mapping it to the right team's scope, and providing guidance that is actionable and complete. A routing error doesn't just waste time — it requires getting security engineers and developers re-engaged to undo the misdirection.

Threat modeling. As AI enters the software development lifecycle and development velocity increases, demand for security reviews is growing. A threat modeling agent that can generate risk assessments and mitigations from a security review intake form — without requiring a human to do the full review — could dramatically reduce this backlog. But an agent that halluccinates risks that don't exist, or misses risks that do, creates noise that erodes developer trust in the security function.

▶ Watch: Problem Definition and Agent Overview (00:00)

Threat Modeling Agent Architecture: Sequential and Specialized

The first design decision for the threat modeling agent was whether it should be a fast, conversational agent or a slower, more thorough asynchronous one. For threat modeling, accuracy matters more than response time. A security review ticket can wait. The team chose the async approach.

This led to a modular, multi-agent sequential architecture:

• An orchestrator agent takes inputs from the security review intake form — including the security review category — and manages the pipeline.
• Input agents gather additional context associated with the ticket: Google Docs links, Slack threads, any supplementary material that should inform the review.
• Security agents are specialized by category. A third-party security review agent handles that specific domain; other agents handle other categories. Each security agent addresses a core baseline of questions every security review must answer — data sensitivity, data transport protocols, the authentication story — plus any additional risks relevant to its domain.
• Output agents handle formatting for different audiences, since different consumers of the threat model wanted different representations: a summarized format for humans, MITRE-format structured output for internal tooling.

▶ Watch: Threat Model Agent Architecture (02:00)

The sequential structure was a deliberate choice. When the team gave the orchestrator agent too much agency, it would sometimes skip specialized sub-agents that were supposed to run. Predictable behavior required prescribing the execution order: input agents first, security agents in parallel, output agents last. The team is now considering a hybrid approach — a guaranteed baseline of security agents that always run, layered with orchestrator-level discretion for ambiguous or vague categories.

Company-specific guidance as a core tool. LLMs have strong generic security knowledge, but the most valuable thing in a threat model at Stripe is company-specific guidance: risks and mitigations that are actually actionable in Stripe's specific environment, referencing Stripe's actual infrastructure and policies. The team invested in internal guidance tools to feed that context into the security agents.

Security Routing Agent: From 10 Minutes to 30 Seconds

The security routing agent required a completely different approach. Because any question might come through routing, the domain couldn't be constrained the way threat modeling could. The problem space was open-ended.

The team's first attempt was a single-step LLM call with no tools — just a prompt containing contextual information about Stripe's various security teams. Fast, but it failed for an obvious reason: the model had no knowledge of Stripe's internal terminology, documentation, or team structures. Whenever a developer referenced an internal tool or system, the model hallucinated.

The fix was to give the model agency: strip the hardcoded context from the prompt, give the model tools to research on its own, and let it find the right answer. This improved accuracy significantly, but the agent took about ten minutes to run — entirely impractical for a routing use case that needs to feel responsive.

▶ Watch: Security Routing Agent Design (04:01)

The path to 30 seconds was iterative tool reduction. The team started with a known set of baseline questions they could answer themselves, gave the agent as many tools as possible, then began removing tools one at a time — retesting accuracy after each removal to confirm it wasn't degrading. Eventually they got down to two tools and a runtime of approximately 30 seconds.

The lesson: more tools is not always better. A minimal, well-chosen tool set produces both better performance and more predictable behavior.

The Evaluation Pipeline: Beyond Vibes

The most technically substantive section of the talk addressed how Stripe measures whether these agents are actually working — and how they know when changes are improving them rather than degrading them.

The threat modeling evaluation problem. Threat modeling is more art than science. There's no single right threat model for a given system. This immediately rules out deterministic matching approaches. The team tried comparing MITRE ATT&CK categories between expected and actual outputs — but in two different runs, the agent might correctly identify the same risk and label it under two different MITRE categories. Keyword matching was similarly unreliable for the same reason: what mattered was the meaning behind the risk, not the exact wording.

▶ Watch: The Evaluation Problem and Golden Standard Approach (06:00)

LLM-as-judge with human-defined golden standards. The solution was a hybrid approach that combined human judgment (for creating ground truth) with LLM capability (for scaling evaluation):

1. Human security engineers created "golden standard" test cases from past security reviews — inputs with known correct outputs.
2. The LLM judge's job was narrowly scoped: given the expected output from the golden standard, identify the semantic equivalences between that and the actual agent output — looking at risks and mitigations for meaning, not exact phrasing.

This addressed the obvious circularity concern (using an LLM to evaluate LLM output) by limiting the LLM's role to semantic comparison rather than absolute judgment. The golden standards were human-created; the LLM just scaled the comparison.

What the eval pipeline revealed. The results from this approach were striking:

• Improving the prompt to include a list of specific security domains to always consider (authorization, single sign-on, etc.) produced approximately a 10% accuracy gain.
• Using the eval pipeline to select the best flagship LLM model — building a mega dataset by duplicating golden standard test cases to account for non-determinism, then testing each model against it — produced another approximately 10% accuracy gain.
• A prompt that included instructions for correctly formatting JSON output looked fine on individual test runs — correct JSON, sensible security advice — but the eval pipeline revealed that overall accuracy had dropped 10%, because the model was devoting attention to format compliance rather than accurate security analysis. Without the eval pipeline, this regression would have been invisible.

▶ Watch: Eval Pipeline Results and Model Selection (10:01)

Meeting Users Where They Are

The team's rollout strategy for each agent was calibrated to the accuracy they could credibly demonstrate:

Threat modeling. The team started with a constrained domain: a specific subcategory of security reviews where risks and mitigations were consistent and well-scoped. A constrained domain reduces the variance that makes automation difficult. The agent ran in "shadow mode" — generating threat models in parallel with human reviews, without enforcing them — until the eval pipeline showed confidence in its accuracy.

On the accuracy threshold question: how accurate does an agent need to be before you deploy it? It depends on how it's deployed. An agent whose output goes directly to engineering teams without human review needs very high accuracy — those teams will revolt if it's noisy. An agent with a human in the loop, where a security engineer reviews and confirms the output, can ship at lower accuracy (the team targeted around 80% initially) because the human catches the errors. Stripe chose the latter path.

▶ Watch: Shadow Mode and Deployment Strategy (12:01)

Security routing. The team released to a small internal group via a web page, gathered feedback, then made it invocable in Slack to test how it handled real questions in context. Finally, it was released to all developers in the internal chat UI. Each expansion was accompanied by structured outreach to collect feedback and feed it back into the eval pipeline.

Key Lessons and Broader Learnings

The team closed with several hard-won lessons:

AlphaEvolve didn't work for prompt optimization. Google DeepMind's AlphaEvolve tool, which evolves prompts through a process analogous to natural selection, was tried and ultimately abandoned for this use case. In theory: generate prompt variations, score them through the eval pipeline, keep the best. In practice: for open-ended language tasks within a reasonable cost budget, the variations were too superficial — adding two words, slightly rephrasing — not the kind of meaningful semantic changes that would drive better security reasoning. For mathematical or computational optimization tasks, AlphaEvolve works well. For security guidance prompting, it did not.

▶ Watch: AlphaEvolve and Architecture Learnings (14:02)

Architecture depends on the task. A modular, multi-step sequential architecture worked well for threat modeling, which is structured, has clear phases, and benefits from specialization. The security routing agent's open-ended nature made a simpler, more flexible structure the right choice. There is no one-size-fits-all agent architecture.

Garbage in, garbage out — and teach the agent to say "I don't know." When intake tickets are vague, the threat modeling agent initially hallucinated details to fill the gaps. The team had to explicitly teach it to behave like a security engineer: acknowledge when information is missing, flag what's unknown, and set up the security engineer to ask the right follow-up questions. A threat model that calls out gaps is more useful than one that fabricates specifics.

Notable Quotes

"When we gave the orchestrator agent too much agency, it wouldn't always run the relevant specialized agent that we wanted it to. Predictable behavior required prescribing the execution order." — Jeffrey Zhang (04:01)

"On individual runs without the evaluation pipeline, things might look great. But the eval pipeline is actually showing us that overall accuracy has gone down ten percent because the agent is focusing more on how to properly output a JSON and less on actually providing accurate security guidance." — Jeffrey Zhang (10:01)

"We had to teach it to act more like a security engineer — be okay with saying 'I don't know' and that there's not enough information, rather than hallucinating to produce good output." — Siddh Shah (16:02)

"There's not really a one-size-fits-all here. A multi-step child agent architecture worked great for threat modeling. For security routing, a more simplistic, undetermined structure really worked." — Siddh Shah (16:02)

Key Takeaways

• Evaluation pipelines are not optional. Every meaningful quality improvement Stripe made — prompt tuning, model selection, prompt regression detection — came from the systematic evaluation pipeline, not from spot-checking individual outputs.
• LLM-as-judge works when its scope is narrow. Having the LLM identify semantic equivalences between a human-created golden standard and agent output is a tractable, reliable task. Having the LLM judge whether an output is "good" in an absolute sense is not.
• Sequential agent architectures provide predictable behavior for structured tasks. Giving an orchestrator full autonomy tends to result in skipped steps and inconsistent behavior for multi-phase workflows.
• Human-in-the-loop reduces the accuracy threshold for deployment. An agent that gets 80% right with human review is shippable. An agent that gets 80% right and sends results directly to engineering teams is not.
• Teach agents to acknowledge uncertainty. An agent that flags missing information and asks follow-up questions is more valuable than one that produces confident, hallucinated assessments.

Slides Reference

Slides from this session are available as 2026-04-04-D1-S1-10-46-Guardrails-beyond-Vibes-Shipping-Secur.pdf. Key topics in the slides include: the threat modeling agent architecture diagram (orchestrator → input agents → security agents → output agents), the golden standard evaluation pipeline diagram, the prompt regression example (JSON formatting causing 10% accuracy drop), the security routing agent timeline from web page → Slack → full rollout, and the AlphaEvolve experiment outcomes.