Tinker Tailor LLM Spy: Investigate & Respond to Attacks on GenAI Chatbots

Tinker Tailor LLM Spy: Investigate & Respond to Attacks on GenAI Chatbots

Speakers: Alan Scott, Senior Staff Engineer, Airbnb

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=QfUdKtkBRjA

Overview

Alan Scott's Black Hat Asia talk, "Tinker Tailor LLM Spy: Investigate & Respond to Attacks on GenAI Chatbots," shifts the focus from demonstrating novel exploits to equipping security professionals with the knowledge and tools to investigate and respond to incidents involving generative AI chatbots. As these AI-powered assistants become ubiquitous, serving as internal IT helpdesks, 24/7 customer support, and even data analysis tools, the potential for brand damage, data leaks, and even remote code execution (RCE) grows exponentially. Scott, a Senior Staff Engineer at Airbnb with a background in enterprise security, threat detection, and incident response, emphasizes that the question is no longer if an incident will occur, but when, and whether organizations are prepared to handle it.

The presentation provides a crucial framework for understanding the architecture, common threats, and defensive strategies for LLM-powered chatbots. Through three practical incident scenarios, Scott illustrates the investigative process, highlighting the critical role of comprehensive logging, the identification of subtle attack vectors, and the implementation of robust "guardrails." His insights are particularly relevant for incident responders, security architects, and developers grappling with the unique challenges posed by the non-deterministic nature and complex interactions of large language models in production environments. The talk culminates in a practical playbook designed to help organizations build preparedness for their first GenAI chatbot security incident.

Background

▶ Watch: Introduction and shocking chatbot failures (0:00)

The landscape of chatbots has dramatically evolved from simple query-response systems to sophisticated generative AI agents capable of complex interactions and actions. This transformation is driven by Large Language Models (LLMs), which, at their core, perform intricate pattern matching on massive datasets to generate human-like text. While incredibly powerful, this capability introduces novel security challenges distinct from traditional application vulnerabilities. Scott highlights early examples of LLM misuse, such as the New York City "My City" chatbot suggesting the legal sale of human meat, or a car dealership bot being coerced into agreeing to sell a Chevy Tahoe for $1, demonstrating the immediate risk of reputational damage.

Beyond public-facing blunders, the integration of LLMs with external tools presents even graver risks. Scott references a discovery by JROG security, who, while investigating the Python package Vanna (which allows natural language queries against data, generating SQL), found a more severe vulnerability. Vanna visualizes query results using Plotly, a popular Python visualization library. Critically, the Plotly code itself was dynamically generated via LLM prompting and then executed. This allowed JROG security to achieve full remote code execution (RCE), underscoring how LLMs interacting with code execution environments can dramatically escalate risks beyond simple data retrieval.

Scott frames the problem from an incident responder's perspective, acknowledging that security incidents often involve a vast array of technologies where no single person can be an expert. The key, he argues, is knowing the right questions to ask and understanding the system's architecture, data flow, and potential actions. To assess risk, he proposes classifying chatbots into three categories: low risk (general information, brand damage potential), medium risk (access to personalized sensitive information like PII or PHI), and high risk (ability to perform actions, or "agency," leading to unauthorized access or RCE). This classification guides the understanding of potential incident types and their severity.

Key Findings

▶ Watch: Talk's focus: GenAI chatbot incident response (3:40)

The talk distills several critical findings for securing and responding to incidents involving GenAI chatbots:

1. Comprehensive Logging is Paramount: Effective incident response hinges on detailed logs. Beyond basic inputs and outputs, critical data points include conversation history (via message thread IDs), correlation with user web sessions (IP, user agent, authentication status), model versions, timestamps, and crucially, guardrail metrics (scores, reasons, decisions) and tool inputs/outputs. This holistic logging approach allows for reconstruction of attack paths and identification of subtle bypass attempts.

1. Beyond the User Prompt: Hidden Inputs: Attackers don't just target the primary user prompt. Other, less obvious inputs, such as user feedback mechanisms that feed into reinforcement learning fine-tuning, can be manipulated to subtly alter LLM behavior over time, leading to undesirable or harmful outputs.

1. Novel Attack Vectors Require New Defenses: Traditional security paradigms don't fully encompass LLM-specific attacks. Scott highlights:

• Jailbreaking: Bypassing guardrails to elicit harmful or inappropriate dialogue, often for "screenshot attacks" causing brand damage.
• Prompt Injection: Analogous to SQL injection, where untrusted user input is concatenated with a trusted system prompt, coercing the LLM into unintended actions, potentially leading to data exposure or RCE.
• Model Inversion: An attacker repeatedly queries the LLM, refining prompts to reconstruct sensitive data from its training set, even if the prompts appear innocuous and are spread out over time to evade simple volume-based alerts.

1. Defense-in-Depth with Chained Guardrails: No single defense is sufficient. A layered approach using guardrails is essential:

• Rule-based metrics: Simple keyword/phrase filters for quick fixes, but easily bypassed.
• LLM as a Judge: A separate LLM evaluates the primary LLM's inputs and outputs against specific criteria, providing an objective assessment.
• System Prompts: Instructions embedded in the model's context, setting behavioral guidelines and operational constraints, often given more weight than user prompts, but not foolproof against clever injection.

1. External Tools are Critical Weak Points: LLMs frequently integrate with external tools (e.g., Python interpreters, APIs) to perform actions. Many of these tools, like Langchain's LLM Math, were not designed with untrusted public input in mind. This creates significant RCE risks if user input is allowed to influence tool execution without rigorous sanitization and validation.

1. Sensitive Data in Training or RAG is a Risk: Any sensitive information present in the LLM's training data or accessible via Retrieval Augmented Generation (RAG) must be considered potentially exposed. Even with anonymization attempts, model inversion attacks demonstrate that sensitive data can be reconstructed and leaked if it's part of the underlying knowledge base. Redaction and strict data hygiene are non-negotiable.

Technical Deep Dive

▶ Watch: Incident commander's role and critical questions (5:30)

The core of effective GenAI chatbot incident response lies in understanding the underlying architecture, the mechanisms of attack, and the technical implementation of defenses.

A basic LLM chatbot architecture involves a user prompt being fed into the LLM, which then generates a response. However, this simplicity belies complex interactions and potential attack surfaces.

Logging is the bedrock of investigation. Scott details crucial logging fields:

• User Prompt: The raw input from the user, essential for identifying attack patterns.
• Message Thread ID: Correlates individual prompts into a full conversation history, allowing reconstruction of multi-turn attack paths.
• Web Session Linkage: Connects chatbot logs with web server logs (containing IP address, user agent, authentication status) to identify repeat attackers or link to other malicious activities.
• Chatbot Outputs: To investigate harmful or manipulated responses.
• Model ID: Identifies which specific LLM version handled the request, crucial for debugging model-related security issues or regressions.
• Timestamp: For correlating events across different systems.
• Chatbot Version: Helps in debugging security regressions or vulnerabilities introduced in new releases.

Beyond the direct user prompt, Scott highlights subtle inputs like user feedback mechanisms. In the "Taylor Swift weather bot" scenario, positive feedback on Taylor Swift-themed responses directly fine-tuned the LLM through reinforcement learning, inadvertently rewarding and propagating the unwanted behavior. This demonstrates that any mechanism allowing users to influence model training or behavior is a potential vector for abuse.

Guardrails are the primary defensive mechanisms, acting as "physical barriers on the road" for AI behavior. Scott outlines three key types:

1. Rule-based Metrics: The simplest control, filtering on keywords or phrases in inputs or outputs. While quick to implement for immediate issues, they are easily bypassed by creative prompt engineering (e.g., "popular music artist famous for her eras tour" instead of "Taylor Swift").

1. LLM as a Judge: A more sophisticated guardrail that uses a separate LLM to evaluate the primary chatbot's inputs and outputs. This "judge" is given a specific prompt to assess quality, relevance, or safety based on predefined criteria (e.g., "evaluate the quality of the following weather report on a scale of 0 to 1"). It comprises two components:

• Scorer: Assigns a numerical value and provides reasoning for the evaluation.
• Threshold Check: Determines if the output meets predefined criteria.

Logging the judge's scores, reasons, and decisions is vital for hunting bypass attempts.

1. System Prompt: A set of instructions given to the model separately from the user prompt, generally carrying more weight. It defines the chatbot's context ("You're a highly accurate and reliable weather assistant"), behavioral guidelines ("Responses should be accurate, complete, and concise"), and operational constraints ("Avoid any topics or comments that aren't relevant to a weather report"). While effective, especially with explicit denials, LLMs can still prioritize strong user instructions if they mimic natural language commands.

Attacks like Prompt Injection exploit the concatenation of untrusted user input with trusted system prompts. Analogous to SQL injection, the attacker manipulates the LLM's instructions. In the event planning chatbot scenario, the system prompt instructed the LLM to convert math expressions into Python code for accurate calculation. An attacker injected malicious Python code disguised as a math expression, leading to a curl command executing a download from a malicious site. The LLM, following its system prompt, processed the "math" (malicious Python) and executed it. To mitigate this, an LLM judge was deployed to evaluate the Python code itself before execution, checking if it was a legitimate math statement or "dangerous."

Jailbreaking is a broader term for bypassing guardrails to make the LLM produce harmful or inappropriate content. Prompt injection is a common technique for jailbreaking, as exemplified by the "DAN (Do Anything Now)" prompt, which attempts to override all previous instructions and restrictions.

Model Inversion attacks represent a significant data privacy threat. In the doctor chatbot scenario, despite attempts at anonymization, sensitive patient data remained in the training set. An attacker, by repeatedly asking questions and refining prompts (e.g., "What medication was prescribed for [Name] at [Age] in [Location]?"), could reconstruct and extract this sensitive information. This attack is insidious because prompts often appear innocuous and can be spread over time, making detection difficult without advanced behavioral analysis.

Finally, Scott highlights Retrieval Augmented Generation (RAG) as a crucial architectural component. RAG allows LLMs to access and incorporate external, up-to-date data sources beyond their initial training. The user prompt is passed to an embedding model, which transforms the data to capture its meaning and relationships. This enriched context is then used by the LLM to generate a more informed response. While powerful, RAG introduces new risks related to the security of these external data sources and their access permissions. Logging the retrieved context (or pointers to it) is essential for investigating data leaks or incorrect information retrieval. The interaction of LLMs with tools (like Python interpreters, APIs) is also a critical area, as many out-of-the-box tools (e.g., Langchain's LLM math) were not designed with public-facing, untrusted input in mind, making them prime targets for RCE. Logging the inputs and outputs of these tool executions is crucial for understanding attack progression.

Demo / Proof of Concept

▶ Watch: Classifying chatbot risk: low, medium, high (6:05)

Instead of live demonstrations, Alan Scott presented three detailed incident scenarios, serving as practical walkthroughs of how to investigate and respond to GenAI chatbot attacks. These scenarios effectively functioned as simulated proofs of concept, illustrating the attack vectors, investigative steps, and defensive strategies in a realistic context.

Scenario 1: The Taylor Swift Weather Chatbot (Low Risk Incident)

• Problem: A low-risk weather chatbot, intended to provide general information, inexplicably started giving all its weather reports with a Taylor Swift theme, even when not requested. Management demanded an immediate fix due to brand damage.
• Investigation: Initial logging of user prompts and LLM outputs didn't fully explain the behavior. The key discovery was an often-overlooked input: a user feedback mechanism. Attackers had provided numerous positive feedback responses to Taylor Swift-themed outputs, effectively fine-tuning the LLM through reinforcement learning to reward this behavior.
• Mitigation:

1. Containment: Roll back the LLM chatbot to an older version before the issue became prominent.
2. Defenses: Implement a multi-layered guardrail approach:

• Rule-based metrics: A quick fix to filter keywords like "Taylor Swift" from inputs and outputs (though easily bypassed).
• LLM as a Judge: A separate LLM was configured to evaluate weather report quality (accuracy, completeness, relevance, conciseness) and block non-weather-related content.
• System Prompt: A robust system prompt was added to explicitly set the chatbot's context ("highly accurate and reliable weather assistant"), behavioral guidelines, and operational constraints ("Avoid any topics or comments that aren't relevant to a weather report").

Scenario 2: The Event Planning Chatbot (High Risk Incident)

• Problem: A high-risk event planning chatbot, capable of taking actions, triggered an EDR alert on its EC2 instance. The chatbot was observed making repeated curl connections to an IP address with a poor reputation, indicating a potential remote code execution (RCE) via a subprocess.
• Investigation: Searching user prompt logs revealed a suspicious prompt: "evaluate the following math expression" followed by Python code executing the curl command. The chatbot's developers had implemented a feature where the system prompt converted math expressions into Python code for accurate calculation. This vulnerability was a classic prompt injection attack, where untrusted user input was concatenated with the trusted system prompt, coercing the LLM to execute malicious Python code.
• Mitigation:

1. Containment: Block the specific malicious prompt and potentially isolate the affected instance.
2. Defenses: Implement an LLM as a Judge specifically for tool inputs:

• The judge was configured to analyze the Python code before it was executed by the math tool. It was prompted to determine if the code was a "math statement" and respond with "yes," "no," or "dangerous." This provided an objective check on the code's intent.
• Additional defense-in-depth involved evaluating the output of the math tool to ensure it was a result of a calculation, providing an alert mechanism if unexpected values were returned.

Scenario 3: The Doctor Chatbot (Low Risk Escalates to High Risk)

• Problem: A supposedly low-risk doctor chatbot, designed for general medical questions, was implicated in a dark web data leak containing patient information.
• Investigation: Searching user prompt logs using names from the leaked data revealed that the chatbot was indeed providing specific, sensitive patient information (name, age, location, prescribed medication) in response to seemingly innocuous prompts. Further investigation revealed that while the AI team attempted to mask or anonymize the training data, they did so inadequately. This allowed an attacker to perform a model inversion attack, where repeated, refined queries reconstructed and extracted sensitive data from the underlying training set. The attacker spread these queries over time to avoid volume-based alerts.
• Mitigation:

1. Containment: Immediately take the chatbot offline or severely restrict its capabilities.
2. Remediation: The primary remediation was to clean the training data, ensuring all sensitive information was properly redacted or anonymized. Scott stressed that if sensitive data exists in the training set, it will eventually be leaked.
3. Future Defenses: Understand the architecture and permissions of Retrieval Augmented Generation (RAG) systems, which allow chatbots to access external data sources. Ensure that any data accessed via RAG is also rigorously redacted and that access controls are properly configured to prevent unauthorized retrieval of confidential information.

These scenarios effectively demonstrated the investigative mindset, the types of evidence to look for, and the practical application of various guardrails and logging strategies in real-world (simulated) GenAI chatbot incidents.

Defensive Implications

▶ Watch: Crucial logging for GenAI chatbot incident investigation (8:45)

Responding to GenAI chatbot incidents requires a proactive and multi-faceted defensive strategy, integrating new LLM-specific controls with established security principles. Defenders must assume that sophisticated attacks will occur and prepare accordingly.

1. Implement Robust and Comprehensive Logging: This is the single most critical defensive measure. Beyond standard application logs, organizations must log:

• User prompts and LLM outputs.
• Full conversation history (using message thread IDs).
• Web session details (IP, user agent, authentication) correlated with chatbot interactions.
• Chatbot model versions and timestamps.
• All guardrail metrics: This includes the scores, reasons, and decisions from LLM as a Judge instances. These logs are invaluable for identifying subtle bypass attempts where an attacker might be "getting just a little bit closer" to an exploit.
• Inputs and outputs of all external tools and APIs the chatbot interacts with.
• Retrieved context from Retrieval Augmented Generation (RAG) systems, or at least pointers to the data sources used.

1. Understand and Classify Chatbot Risk: Categorize chatbots based on their data access (general, personalized/sensitive) and agency (ability to perform actions). This helps prioritize defenses and anticipate potential incident types (brand damage vs. data exfiltration vs. RCE).

1. Layered Guardrail Implementation (Defense-in-Depth):

• Rule-based Metrics: Use for immediate, quick fixes for known patterns, but acknowledge their limitations and ease of bypass.
• LLM as a Judge: Employ a separate, objectively prompted LLM to evaluate inputs and outputs against specific security and content criteria before processing or responding. This is particularly effective for evaluating code snippets or complex content.
• Robust System Prompts: Craft detailed system prompts that clearly define the chatbot's purpose, behavioral guidelines, and explicit denials for prohibited topics or actions. While not foolproof, they provide a strong baseline for guiding LLM behavior.

1. Secure External Tooling and API Interactions:

• Never trust user input that might reach an external tool. Implement rigorous sanitization and validation before passing user-derived data to any tool.
• Audit all tools: Many out-of-the-box LLM tools (e.g., Langchain's LLM Math) were not designed with public-facing, untrusted user input in mind. Understand their capabilities and potential for abuse (e.g., RCE).
• Monitor tool execution: Log what tools were accessed, their inputs, outputs, and any internal/external commands executed.

1. Strict Data Hygiene for Training and RAG Sources:

• Redact or anonymize all sensitive data in LLM training datasets. Assume that if sensitive data is in the training data, it will eventually be leaked through model inversion attacks.
• For RAG systems, understand the permissions and architecture of connected data sources. Ensure that only appropriately sanitized and authorized data is retrieved and used, and that access controls prevent the chatbot from pulling confidential data incorrectly.

1. Integrate LLM Monitoring with Existing Security Operations:

• Apply existing security monitoring tools like EDR to the infrastructure hosting chatbots. Malicious activity (e.g., curl commands, suspicious subprocesses) on the underlying systems should still trigger alerts.
• Actively hunt for guardrail bypass attempts by analyzing guardrail logs for inputs that barely met thresholds or were flagged but not fully blocked. Attackers often iterate, and these near-misses can reveal their progression.

1. Develop a GenAI Chatbot Incident Response Playbook: This should outline specific steps for investigation, containment, and remediation, covering:

• Reviewing user prompts for patterns of jailbreaking or prompt injection.
• Analyzing chatbot outputs for inappropriate content or data exfiltration.
• Investigating guardrail metrics to understand why inputs were blocked or allowed.
• Examining tool inputs/outputs and external commands.
• Scrutinizing data sources (training data, RAG) for sensitive information.

1. Cross-Functional Collaboration and Continuous Learning: Involve legal, PR, and engineering teams in preparedness planning. The field of LLM security is rapidly evolving, so continuous learning about new attack types and defensive techniques is essential.

Key Takeaways

• Comprehensive Logging is Non-Negotiable: Log user prompts, LLM outputs, conversation history, web sessions, guardrail metrics, and tool interactions to effectively investigate incidents.
• Understand All Inputs: Beyond direct user prompts, subtle inputs like user feedback mechanisms can be exploited to manipulate LLM behavior.
• Implement Layered Guardrails: Combine rule-based metrics, LLM as a Judge, and robust system prompts for defense-in-depth against evolving threats.
• Beware of External Tool Vulnerabilities: Many LLM tools (e.g., Langchain's LLM Math) were not designed for public exposure and can lead to RCE if user input is not rigorously sanitized.
• Sensitive Data is a Critical Risk: Any sensitive information in training data or accessible via RAG must be fully redacted, as model inversion attacks can reconstruct and leak it.
• Prepare for Novel Attacks: Attacks like prompt injection and model inversion require new investigative techniques and a continuous learning mindset for incident responders.

About the Speaker(s)

Alan Scott is a Senior Staff Engineer at Airbnb, specializing in enterprise security, threat detection, and incident response. His work focuses on securing complex systems and preparing organizations for emerging threats. Known for his practical approach, he is quoted as saying, "red teamer tears are the proof of your effective work," reflecting his dedication to robust security. Alan lives in Austin, Texas, with his wife and four-year-old son, Liam. While his talk presents personal opinions, his professional background at Airbnb provides valuable insights into real-world security challenges within large-scale technology environments. He also maintains a personal newsletter called "Meoward," which sometimes features security information.