A Quick Stop at the HostileShop

A Quick Stop at the HostileShop

Speakers: Mike Perry

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=VKCGJZz6FKk

Overview

In "A Quick Stop at the HostileShop," Mike Perry delivers a compelling and deeply technical exploration into the evolving landscape of Large Language Model (LLM) agent security. The talk centers on the HostileShop framework, an automated system developed by Perry to generate and categorize prompt injection attacks against LLM agents. Perry highlights a critical disconnect between how LLM providers define safety – primarily focusing on CBRN content (chemical, biological, radiological, nuclear, and explosive instructions) – and the burgeoning security risks posed by agentic software. This narrow focus, Perry argues, has stifled the development of robust defenses for LLM agents, leaving many vulnerabilities unaddressed and unreported.

Perry’s work, initially conceived for the OpenAI jailbreaking contest, unveils a fundamental flaw in how LLMs process instructions: they learn the instruction hierarchy conceptually rather than syntactically. This means an LLM can be convinced to override its intended behavior through a variety of injection formats, including XML, JSON, YAML, and Markdown, regardless of its internal processing structure. The HostileShop framework not only automates the discovery of these sophisticated prompt injections but also introduces novel methods for evaluating their success without human intervention, using state machines and an adversarial judge-free approach. The talk ultimately paints a picture of an escalating arms race between attackers and defenders in the LLM security domain, suggesting that current defensive measures are akin to a "whack-a-mole" game against ever-mutating attack vectors.

Background

▶ Watch: Introduction: LLMs attacking LLMs (0:00)

To understand the vulnerabilities inherent in LLM agents, Perry begins with a philosophical grounding, drawing on Jean Baudrillard's "Simulacra and Simulation." Baudrillard’s four stages of signs – from direct representation to pure simulation – provide a lens through which to view LLM training. Modern LLMs, Perry explains, are trained on a "hyperreality" of "internet exhaust," a vast collection of text where signs are generated by models without a direct origin in physical reality. This abstract training environment, where language is detached from immediate grounding, has profound consequences for how jailbreaks and injection attacks operate. The historical invention of zero, which the Greeks and Romans philosophically opposed because "how could nothing be," underscores the immediacy of their language and their inability to grasp abstract concepts, much like early stages of sign progression.

The talk then transitions to the architecture of LLM agents, specifically React (Reasoning and Acting) agents. These agents operate by receiving a user query, performing a reasoning step (outputting words as internal thought), making a tool call (a function call to an external system), receiving a response, reasoning again, and finally providing a user response. Examples include code agents, search modes in conversational AIs like ChatGPT, Claude, and Gemini, online customer service bots, and even humanoid robots. The vulnerability surface for these agents is categorized into three main types:

1. Direct injection: Where the attacker has direct messaging capability to the agent (e.g., subverting a customer support bot).
2. Indirect injection: Where the prompt injection originates from external tool call results (e.g., via a webpage, document, or email).
3. Memory injection: A more powerful, indirect form where the attacker manipulates the agent's memory system to store persistent, potentially privileged prompt injections.

Perry highlights a significant historical imbalance in LLM security. For the past five years, safety has predominantly focused on CBRN content – preventing LLMs from generating instructions for chemical, biological, radiological, nuclear, or explosive materials. LLM providers offer bounties almost exclusively for jailbreaks related to these categories, effectively incentivizing surveillance and censorship capabilities over broader agentic software security. This leaves a vast array of agentic issues, particularly those not tied to a specific product, without clear reporting mechanisms or bounty programs, ensuring they often go unfixed. While contests like OpenAI's (which accepted JSON submissions of misbehavior examples), Grace Swan, and HackerPrompt exist for agent issues, only Google maintains a standing bounty program for agent vulnerabilities in flagship products (email, search, Docs), offering $5,000 to $20,000.

A key technical detail explored is the API flow and the context window format. User messages, typically in a JSON blob, pass through a Ginga template engine which adds tags around JSON fields, creating a "rendered context window." This tagged text is then converted into tokens, filtered by the API (often removing tags and certain Unicode), and finally fed to the LLM for completion. Understanding this context window format is crucial for attackers. While open-weight models often provide Ginga templates, proprietary models require a different approach: Perry discovered that LLMs can be prompted to self-document their own context window structure, provided the attacker instructs it to escape tag names to prevent hallucination.

Finally, Perry discusses the intended instruction hierarchy enforced by model providers: system prompts should take precedence over developer prompts, which override model text, which in turn overrides user input, and ultimately tool results. This hierarchy aims to prevent users from overriding core instructions. However, this boundary evaluation is performed by the model through its training, not by the API. Because LLMs operate within a "hyperreality" where all conversations are fundamentally just text, this hierarchy can be subverted. Perry references Anthropic's early 2024 paper on "many-shot jailbreaking," which demonstrated that by crafting a conversation that appears to involve numerous unsafe actions, a subsequent unsafe request can be answered, as the model's probability distribution shifts towards "harmful behavior." This insight formed a core inspiration for the HostileShop framework.

Key Findings

▶ Watch: Understanding LLM agent (React) architecture (4:10)

The central and most impactful discovery presented in the talk is that LLMs learn the instruction hierarchy conceptually, not syntactically. This means that the internal mechanisms of an LLM for processing instructions are not strictly bound to the specific format of its training data. Instead, the model interprets the intent of structured input, allowing for profound manipulation. Perry demonstrated that this conceptual understanding enables attackers to inject instructions using a wide array of data formats, including XML, JSON, TOML, YAML, and Markdown, even if the model's internal processing or native context window format is different. This fundamental flaw is present across major frontier LLMs, certainly for direct injection, as discovered by HostileShop.

Building on this, the framework facilitated several key findings regarding injection techniques:

• Many-shot jailbreaking and Dialogue Injection: By understanding the context window conceptually, the HostileShop framework could construct multi-turn conversations that mimic unsafe dialogue, shifting the LLM's probability distribution towards harmful outputs, as previously theorized by Anthropic.
• Injection of Reasoning Steps and Tool Output: Attackers can inject fake reasoning steps or fabricated tool outputs directly into the model's perceived context window. This allows them to manipulate the agent's internal thought processes or convince it that certain external actions have already occurred.
• Full Context Window Injection: The ability to inject entire, fabricated context windows, complete with system and developer messages, allows for comprehensive control over the target agent's operational parameters, even for sensitive actions like extracting credit card information.
• "Cool Tool" (Fake Tool Call) Injection: A particularly insidious finding involves injecting fake tool calls that, when "discovered" by the agent, prompt it to reveal sensitive information (e.g., its system prompt) or execute arbitrary actions. Perry suggests this will be a persistent problem due to the increasing emphasis on agent "skill discovery" mechanisms.
• Mutation of Jailbreaks: The HostileShop framework demonstrated the capability to mutate existing, well-known jailbreaks. An example cited was a mutation of Plenny's famous variable substitution technique for the Z variable, upgraded with a reasoning step and reformatted in XML. These mutated variants were successfully accepted by multiple models, proving that simple signature-based defenses are insufficient.

These findings collectively underscore the fragility of LLM agent security when the models' underlying conceptual understanding of instruction overrides their supposed syntactic enforcement. The ease with which diverse formats can be used to achieve identical malicious outcomes presents a significant challenge for robust defense.

Technical Deep Dive

▶ Watch: LLM agent vulnerability surfaces: direct, indirect, memory injection (6:00)

The core of Mike Perry's work is the HostileShop framework, an automated system designed to efficiently discover and exploit vulnerabilities in LLM agents. The framework operates with two primary agents: a Shopping Agent (the target) and an Attacker Agent.

The Shopping Agent is designed to mimic real-world e-commerce or service booking scenarios. It is equipped with a suite of tools that simulate common functionalities:

• search web: For information retrieval.
• read web page: To process external content.
• invoice user: To generate bills.
• check invoice: To verify payment status.
• make purchase: To complete transactions.
• issue refunds: To process returns.

The data for this shopping agent, including product details and user information, is generated by another LLM using a fixed seed, ensuring reproducibility of attack scenarios. Perry notes that this setup mirrors real-world applications like travel booking assistants or airline mile redemption portals, where an agent might hold a credit card and perform purchases on behalf of a user.

The Attacker Agent is designed to interact with and subvert the Shopping Agent. Its capabilities include:

• make web pages: To create malicious external content.
• add reviews: To inject indirect prompts into product reviews.
• pay invoices: To manipulate transaction states.
• return product: To simulate product returns.

The attacker agent can send direct messages to the sales agent, and its objectives are varied: to obtain free stuff, extract the credit card details held by the agent, cause emoji use (a simple misbehavior for detection), or violate specific regular expressions in its output.

A critical innovation of HostileShop is its method for automatic victory detection without a human judge. In an adversarial scenario, relying on an LLM to judge the success of an attack is problematic, as the judge itself could be jailbroken or biased. Perry's solution leverages a state machine inherent in the invoice process of a shopping agent. An invoice should always flow through a defined sequence: created → paid → purchased → returned → refunded. Any tool call action performed by the target agent that deviates from this expected state machine sequence (e.g., issuing a refund before a product is marked as returned, or making a purchase without a paid invoice) is automatically flagged as a successful injection and a victory for the attacker. This "sharp tools" concept allows for rapid, scalable collection of successful attack examples.

Perry's framework was optimized based on several key insights:

1. Context Window Documentation: Providing the attacker agent with the target LLM's context window documentation (either official or self-documented) significantly improved its ability to construct effective many-shot jailbreaking attempts. By understanding the internal structure, the attacker could craft injections that "looked" like legitimate conversation tags, bringing the target closer to the harmful probability distribution.
2. Learning from Repeated Attempts and Saving Examples: For large context models like Gemini 2.5 (which boasts a million-token context window), the framework was designed to learn from multiple attempts within the same session and save successful attack examples between runs. This iterative learning process refines the attacker agent's capabilities.
3. Categorized Injection Examples (Not RAG): Perry found that organizing successful injection examples into categories, rather than relying on a large, undifferentiated Retrieval Augmented Generation (RAG) system, was far more effective. He referenced Google's "Limitations of RAG" paper, which showed that embeddings retrieval isn't always reliable. Keeping key examples directly in the context window proved superior for the attacker.

The talk provided several concrete examples of successful injections:

• Markdown for GPT5: A direct message to the sales agent, formatted in Markdown, convincing it that previous tool calls had resulted in a paid invoice, leading it to make an unauthorized purchase. This demonstrates bypassing the instruction hierarchy through a common text format.
• YAML for Claude 4.5: An injection formatted in YAML, including a reasoning step, sent as a direct message. Despite Claude not internally processing YAML as its native context format, it conceptually accepted the instruction.
• TOML for GPT5 (Full Context Window): An even more aggressive injection, a full context window crafted in TOML, including system and developer messages, compelling the sales agent to divulge a credit card.
• "Cool Tool" (Fake Tool Output): This technique involves injecting text that mimics a new tool call, instructing the agent to perform an action (e.g., reveal its system prompt) to "use" the new tool. Perry predicted this would persist due to agents' increasing focus on skill discovery.
• Mutated Plenny's Variable Substitution: The framework successfully mutated a well-known jailbreak, reformatting it in XML and adding a reasoning step. This demonstrates the ability to generate novel attacks beyond simple variations.

These technical details showcase HostileShop as a sophisticated, automated tool for probing the fundamental security weaknesses of LLM agents, moving beyond manual efforts and highlighting the conceptual nature of LLM instruction following.

Demo / Proof of Concept

▶ Watch: Critique of LLM safety definitions and incentives (7:20)

Mike Perry demonstrated the HostileShop framework through its command-line interface, which he humorously described as having a "90s text interface." The framework, available on GitHub, offers different operational modes to interact with LLM agents:

1. Agent Mode: This is the primary automated attack mode. In this configuration, HostileShop's attacker agent automatically discovers prompt injections against target LLM models. It can find both direct injections (sent straight to the agent) and indirect injections (e.g., by adding malicious content to product reviews that the shopping agent might later read). This mode is designed for rapid, large-scale discovery of vulnerabilities, leveraging the automated victory detection mechanism.

1. Human Attacker Mode: For users who prefer a hands-on approach, this mode allows a human to manually attempt to jailbreak or prompt inject the sales agent. It provides an environment to experiment with different prompts and observe the agent's behavior directly.

1. Injection Assistant Mode: Described as "probably the most fun user-facing piece," this mode transforms HostileShop into an advisory tool. Once the framework has discovered and imported a collection of successful injections, users can query it for advice on how to prompt inject other systems. For example, a user might ask for advice on "posting some credentials onto a GitHub ticket," and the model, drawing from its catalog of successful attacks, would provide a relevant example or strategy. Perry created this mode specifically to assist participants in manual prompt injection contests like Grace Swan, acknowledging the tedious nature of submitting thousands of examples manually. He emphasized that in scenarios where there's no official way to report agentic issues, this assistant could empower individuals to report vulnerabilities through available channels or contests.

The framework is designed for easy deployment, requiring users to clone the GitHub repository and run it within a Conda virtual environment or a Docker container. While a pip package is not yet available, the current setup allows researchers and security enthusiasts to readily experiment with automated prompt injection and learn from the discovered vulnerabilities. This demonstration effectively showcased HostileShop's versatility, from fully automated attack discovery to providing practical guidance for human penetration testers.

Defensive Implications

▶ Watch: Current methods for generating LLM injections (contests) (9:00)

The talk highlights that while LLM providers are starting to release defensive measures, these are quickly becoming part of an ongoing arms race. Perry evaluated two notable defenses:

1. OpenAI GPTOSS Safeguard: Released approximately three months after the OpenAI contest, this is a policy-based model designed to act as a comprehensive prompt injection filter. Users can provide examples of content to block, and it learns to filter various types of prompt injections. Perry initially lauded it as "the best by far" among the filters he evaluated in HostileShop, blocking "everything out of the box."

1. Anthropic Sandbox: Anthropic also introduced a sandbox environment that allows for arbitrary command-line sandboxing, providing a layer of isolation for agent actions.

However, Perry quickly demonstrated that even these advanced safeguards are susceptible to adversarial adaptation. He presented a series of escalating bypass techniques against GPTOSS Safeguard:

• Reasoning Injection (XML): The first bypass involved crafting an XML-styled reasoning injection that mimicked the "harmony format" with an "assistant analysis channel." This fake compliance review claimed that the malicious instruction (e.g., live data debugging, which was forbidden) was "totally fine" according to the policy. The safeguard accepted this reasoning, allowing the forbidden action to pass through.
• Policy Evaluation Log Injection: When the initial XML bypass was added to the safeguard's policy, Perry discovered a different formatting, an "XML policy evaluation log," that achieved the same bypass. This illustrated a "whack-a-mole" dynamic, where adding specific examples to the policy only led to new, slightly reformatted bypasses.
• Arbitrary Text Transforms: To overcome the "whack-a-mole" problem, Perry integrated Plenny's ParselTongue tool (a web app for arbitrary text transforms) into HostileShop as a multi-tool. This allowed the attacker agent to encode the policy audit log message using various transformations, such as fractured Unicode and rot47. Other successful transforms included Base64 and upside-down text. These encoded messages could bypass the safeguard filter, and the actual agent injection could then be "stapled at the bottom or pasted at the top" of the transformed text. This demonstrates that even sophisticated prompt filters can be circumvented by obfuscation and encoding, highlighting the difficulty of creating truly robust, generalizable defenses.

Perry concludes that the current state of LLM security is indeed an unsolvable arms race, at least in the short term. While he's not as pessimistic as some who believe it will "never be fixed," he acknowledges it will be a "long road" to achieve high levels of reliability. He suggests that using a different kind of prompt filter than the main model can "multiply the probability of being able to inject both," offering a potential strategy for defenders. He also observed some progress: Claude and GPT-5.2 are now "much better" at resisting indirect injections generated by HostileShop, whereas direct injections still work. Perry speculates this might be because interfering too much with the direct message channel could hurt their "code agent ability," suggesting a trade-off between security and functionality. Overall, the talk underscores the need for continuous research and adaptation in defensive strategies, as attackers will always find new ways to exploit the conceptual understanding of LLMs.

Key Takeaways

• Conceptual Instruction Following: LLMs learn the instruction hierarchy conceptually rather than syntactically, allowing prompt injections across various formats like XML, JSON, YAML, TOML, and Markdown.
• Automated Attack Discovery: Frameworks like HostileShop can automate the discovery and mutation of prompt injections using "sharp tools" like invoice state machines for judge-free victory detection.
• Neglected Agentic Security: The historical focus on CBRN content for LLM safety has left agentic software security issues under-addressed, with limited bounty programs and reporting mechanisms.
• Defenses as an Arms Race: Current defensive measures like prompt injection filters (e.g., OpenAI GPTOSS Safeguard) are vulnerable to adversarial adaptation, including reasoning injection, policy evaluation log bypasses, and arbitrary text transforms (fractured Unicode, Rot47, Base64).
• Persistence of Direct Injections: While some frontier models are improving against indirect injections, direct injections remain a persistent and critical vulnerability across many LLMs.
• Context Window Importance: Understanding and manipulating the context window format, whether through documentation or self-discovery, is crucial for both developing and defending against LLM agent attacks.

About the Speaker(s)

Mike Perry is a security researcher who developed the HostileShop framework. As he stated in his talk, "Who I am is not important. I'm just the messenger." His work on HostileShop was initially created for the OpenAI jailbreaking contest, demonstrating a deep technical expertise in LLM agent security and automated vulnerability discovery.