Breaking BOTS: Cheat Blue Team CTFs by Building AI Agents That Investigate

Breaking BOTS: Cheat Blue Team CTFs by Building AI Agents That Investigate

Speakers: Leo, Graphistry

Conference: BSides Seattle

YouTube: https://www.youtube.com/watch?v=46fK1lt6rhI

Overview

Leo from Graphistry presented a deeply practical talk on using AI agents to solve blue team CTFs — specifically Splunk's Boss of the SOC (BOTS) — and what that tells us about the future of AI-assisted security investigations. The talk progresses from a simple "just throw Claude at it" baseline that scores 56% with zero prompt engineering, through systematic prompt engineering and evaluation-driven development, to a breakthrough approach that achieves 100% on the BOTS competition by having the AI conduct a full incident response investigation before seeing any questions.

The broader vision is ambitious: putting a "Bruce Schneier in your pocket" for every defender, where AI agents can conduct genuine security investigations at tier-2 analyst quality. Leo frames CTFs as AI gyms — community-defined benchmarks created by some of the smartest security people in the world — that provide the perfect training and evaluation environment. The talk is equal parts practical guide for individuals wanting to try this at home and a research roadmap for professional AI security tooling.

Beyond the impressive results, the talk contains a critical insight about AI reliability: at a 70% per-step accuracy rate across a 10-step investigation, the overall success rate drops to just 3%. This mathematical reality explains why the journey from C-minus to production-ready requires fundamentally different engineering approaches, and why human-in-the-loop "vibes investigating" remains essential.

Background

▶ Watch: Viral video: AI solves 75% of Splunk BOTS in 2 hours (1:45)

Graphistry has spent the past decade building open-source tools for security and data analysis, contributing to projects like Apache Arrow and NVIDIA Rapids for GPU-accelerated computing. About two years ago, the team began applying AI agents to Splunk's Boss of the SOC CTF, publishing a viral video that showed their system solving three-quarters of the competition in about two hours — tasks that would take a human team an entire day. Each question, even the hard ones, took only two to three minutes.

The BOTS competition is particularly valuable as an AI benchmark because it is free, open, and community-defined. The scenarios include real incident response tasks like cloud incident investigation (S3 instance analysis, timeline scoping) that require tier-2 analyst skills. Importantly, the same incidents appear across multiple log types — cloud logs and Windows security logs for the same activity — creating multi-source data sets that test correlation abilities.

Leo also addressed the contamination problem: modern AI models are trained on internet data that includes CTF writeups and answers. When they test new models on BOTS without connecting them to the database, none of them score zero. Some questions are answerable from common knowledge, and some are path-dependent where inlined answers from previous questions unlock the next one. This requires careful evaluation methodology to ensure results are valid.

Key Findings

▶ Watch: Reliability math: 70% per-step accuracy = 3% over 10 steps (6:00)

Raw Model Performance: Claude Opus 4.5, with zero prompt engineering and no custom tools — just Claude Code connected to Splunk with CLI credentials — scored 56% on BOTS. Sonnet 4.5, a much cheaper model, scored 30%. These baselines demonstrate that even without engineering effort, modern models can meaningfully engage with security investigation tasks.

Open Source Model Parity: Top open-source models perform roughly at the Sonnet level, meaning that with prompt engineering, organizations running their own models can achieve C-minus performance — a meaningful starting point for local, private AI investigation capabilities.

Prompt Engineering Wins: Two key prompt additions produced 10-20% improvement: investigative methodology (telling the model to use OODA or OSCAR frameworks, which the LLM already understands) and cross-validation (forcing the model to corroborate findings across multiple log sources or events, directly combating hallucination).

MCP vs. Raw CLI: Switching from raw curl/CLI to MCP (Model Context Protocol) wrappers produced measurable improvement not through added intelligence but by eliminating the token burn of figuring out REST API ceremony. In cost-and-time-gated investigations, this overhead reduction matters significantly.

Thin vs. Rich MCP: Most vendor MCPs, including Splunk's, are "thin" — they tell the AI how to call the API but do not provide a data dictionary. They do not map which indexes contain which log sources, which columns exist in each source, or how to pivot across sources. Adding this data map layer provides further improvement.

The 100% Breakthrough: The team achieved 100% on BOTS by fundamentally changing the approach. Instead of answering questions cold, they had the AI conduct a full incident response investigation first — without seeing any questions. The AI explored the database, identified incidents, created timelines, generated IR reports, and built visualizations. This produced a "crib sheet" that, combined with question-specific investigation, cut per-question solve time in half and achieved perfect accuracy.

Reliability Math: At 70% per-step accuracy across 10 investigation steps, overall reliability is only 3%. Even at 99% per-step accuracy, 10 steps yields only 90% reliability. This mathematical constraint explains why production AI investigation tools require fundamentally different engineering from C-minus prototypes, and why human-in-the-loop approaches remain essential.

Technical Deep Dive

▶ Watch: Prompt engineering: methodology (OODA/OSCAR) and cross-validation techniques (14:00)

The technical architecture evolved through three distinct phases:

Phase 1 — Caveman Mode: Claude Code connected directly to Splunk with CLI credentials. No MCP, no prompt engineering, no custom tools. This establishes the raw model baseline (56% for Opus 4.5, 30% for Sonnet 4.5).

Phase 2 — Prompt Engineering: Two additions produced the largest gains. First, investigative methodology: simply telling the model to use OODA or OSCAR frameworks activates built-in knowledge about structured investigation without requiring the prompt to explain the methodology. Second, cross-validation: requiring the model to corroborate every finding in an alternate log source or overlapping event. This fights hallucination by forcing the model to verify claims against actual data rather than confabulate answers.

Phase 3 — Eval-Driven Development: The team moved beyond "vibe coding" to what Leo calls "eval-driven coding loops." Instead of writing unit tests and looping until they pass, they run the full CTF benchmark, conduct error analysis on failures (wrong answers, token waste, dead-end investigations), and optimize prompts, skills, and custom tools against the eval set. OpenTelemetry is used to capture all prompts, responses, and tool calls as structured logs, giving the AI feedback on its own performance. Both OpenAI and Claude now expose this telemetry natively.

Phase 4 — Pre-Investigation: The breakthrough approach has the AI perform a complete incident response flow before any questions are asked. The AI connects to the database, explores the data, identifies incidents without being told what they are, creates timelines, generates IR reports, and builds graph visualizations. This pre-investigation creates context that makes subsequent question-answering dramatically faster and more accurate.

The team runs all CTF questions in parallel rather than sequentially (as human teams would), treating each as an independent investigation task. Some questions are path-dependent, where one answer feeds the next, and these are handled by inlining answers.

Demo / Proof of Concept

▶ Watch: Thin MCP vs rich MCP — why data dictionaries matter (17:30)

Leo referenced their viral video from the previous year showing the system solving three-quarters of BOTS in about two hours with all questions running in parallel. The current talk showed the prompt configurations (which fit on a single slide), the Claude Code invocation on the bottom of the screen, and the 100% result achieved in the most recent run from "this week." A live demonstration was originally planned but skipped due to time and setup constraints. Leo committed to sharing all prompts publicly on Twitter and LinkedIn.

Defensive Implications

▶ Watch: Call to action: try it yourself, CTFs should add AI tracks (22:00)

The implications for defenders are dual-edged. On the positive side, AI investigation tools that can achieve 100% on structured CTF benchmarks represent a transformative capability for SOC operations. The "Bruce Schneier in your pocket" vision means tier-2 investigation quality could be accessible to every organization, not just those that can afford elite analysts.

The practical path for defenders starts with using existing AI coding tools (Claude Code, similar tools) connected to their SIEM via CLI or MCP, trying CTF challenges as practice, and iterating through prompt engineering. The cross-validation prompting technique — forcing AI to corroborate findings across multiple log sources — directly addresses the hallucination problem that makes AI investigation unreliable.

For the CTF community specifically, Leo recommended creating AI tracks rather than banning AI, arguing that these competitions are the community's opportunity to shape AI investigation capabilities and define what matters in security analysis.

Graphistry is preparing to release free tooling for AI-assisted security investigation (bring-your-own-keys, fully private AI), with early access available through their website for users who specify their database platform.

Key Takeaways

• Claude Opus 4.5 scores 56% on Splunk BOTS with zero prompt engineering; adding investigative methodology and cross-validation prompts produces 10-20% improvement per technique
• The mathematical reality of compound accuracy (70% per step across 10 steps equals 3% overall reliability) explains why the gap from prototype to production is enormous, and why human-in-the-loop investigation remains essential
• The breakthrough to 100% came from having the AI conduct full incident response (identify incidents, create timelines, generate reports) before answering any questions, fundamentally changing the approach from cold-start Q&A to contextualized investigation
• MCP wrappers over raw APIs save meaningful time by eliminating REST ceremony, but "thin" MCPs that lack data dictionaries leave significant performance on the table
• Eval-driven development with OpenTelemetry feedback loops is fundamentally different from vibe coding and is the engineering discipline needed to move up from C-minus to production-ready
• CTFs are the ideal AI benchmarks because they are community-defined, free, open, and built by the smartest security practitioners — the CTF community should create AI tracks rather than ban AI participants

About the Speaker(s)

Leo is from Graphistry, a company that has spent the past decade building open-source and free tools for security and data analysis, including contributions to Apache Arrow and NVIDIA Rapids for GPU computing. The team behind this research includes Alex Maurice, Alex Warren, Tanoy, Synindra, and Thomas Cook. Graphistry is organizing a free one-day AI security event (virtual and in-person at RSA) with trainings. Leo's background spans science and engineering, and the approach to AI security investigation reflects both perspectives — scientific rigor in evaluation methodology and engineering pragmatism in tool-building.