How We Red-Teamed Our Own AI Agent: Lessons from Operation Pale Fire

How We Red-Teamed Our Own AI Agent: Lessons from Operation Pale Fire

Speakers: Josiah Peedikayil; Offensive Security Team, Block

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=6sYpcbpsxrc

Overview

This talk, "How We Red-Teamed Our Own AI Agent: Lessons from Operation Pale Fire," presented by Josiah Peedikayil from Block's offensive security team, details a proactive red team operation targeting Block's internal AI agent named Goose. The primary objective was to understand the real-world implications of attacking an AI agent from an external perspective, moving beyond theoretical proof-of-concepts to a full, end-to-end operation aimed at achieving initial execution on an employee's laptop. This initiative was particularly prescient given Block's plans to open-source Goose, necessitating a thorough internal security evaluation.

The presentation delves into the challenges and ultimate success of "Operation Pale Fire," highlighting the ingenuity required to bypass existing security controls and the unexpected complexities encountered when dealing with large language models (LLMs) and their integration points. Peedikayil shares valuable lessons learned from two distinct attack campaigns, revealing how the team leveraged social engineering, novel prompt injection techniques, and the subtle manipulation of user interfaces to compromise the AI agent. The talk culminates in a discussion of the practical defensive implications and the mitigations implemented post-operation, offering critical insights for organizations developing and deploying their own AI-powered tools.

The significance of Operation Pale Fire extends beyond Block itself, providing a blueprint for how security teams can proactively assess the risks associated with emerging AI technologies. By emulating sophisticated adversaries and exploring potential attack vectors, Block was able to identify vulnerabilities in its AI agent's design and integration, leading to robust improvements in its security posture. This forward-looking approach to security, focusing on operationalizing novel attacks, is crucial as AI agents become increasingly prevalent in enterprise environments, potentially exposing new attack surfaces.

Background

▶ Watch: Introduction to Operation Pale Fire and Block's Red Team Approach (0:00)

At Block, red team operations are designed to simulate sophisticated external adversaries, often emulating the techniques of known Advanced Persistent Threats (APTs) or theorizing future attack methods against the company. These operations are end-to-end focused, typically starting with initial access from outside the corporate network, followed by lateral movements to achieve a defined objective – in Block's case, often related to financial goals as a fintech company. A paramount aspect of these operations is stealth and evasion, as Block's detection and response team is highly effective at identifying and neutralizing malicious activity. "Operation Pale Fire" was explicitly aligned with the forward-looking aspect of Block's red teaming, anticipating future threats to AI agents.

The focal point of this operation was Goose, an early AI agent developed internally at Block. Initiated in late 2023/early 2024, Goose was designed to be accessible to both technical and non-technical users. Notably, Goose has since been open-sourced, making the lessons from this red team exercise even more relevant to the broader security community. Architecturally, Goose was built with a builder-focused mindset, shipping with a developer shell tool call that allows it to execute bash commands – a feature that immediately caught the attention of the offensive security team. Goose also integrates with various underlying LLMs, including Sonnet 3-5, 3-7, and GPT, and can connect with an internal MCP marketplace (Micro-Capability Platform) to perform tasks, such as interacting with Google Drive or Jira.

The motivation for targeting Goose stemmed from several factors. While existing research offered numerous proof-of-concepts for AI agent attacks like prompt injection and model poisoning, Block's red team sought to understand what a full, end-to-end operation would look like, starting from outside the company and targeting an AI agent for initial access. The impending open-sourcing of Goose also prompted a proactive security assessment, anticipating that external attackers might target Block employees via the agent. The team aimed to uncover any "dangerous features" within Goose that could expose users to initial compromise and, critically, to operationalize a novel prompt injection attack that was both effective and stealthy, avoiding immediate detection. The specific objective of Operation Pale Fire was to achieve execution on a Block employee's laptop, without necessarily involving extensive lateral movement beyond that initial compromise.

Key Findings

▶ Watch: Why Target Goose? Operationalizing AI Agent Attacks (4:09)

Operation Pale Fire yielded critical insights into the vulnerabilities and attack surfaces presented by integrating AI agents into enterprise environments. The red team initially attempted a user prompt injection via seemingly innocuous Google Calendar invites, leveraging invisible Unicode characters to conceal malicious commands. This first campaign, however, largely failed due to several factors: the dynamic nature of the Google Calendar MCP (Micro-Capability Platform) updates, the varying resistance of different LLMs to the injection, and an underestimation of the impact of a realistic context window filled with legitimate calendar entries. This underscored the challenge of achieving reliable prompt injection in complex, real-world scenarios, particularly when the injection is not a system prompt injection.

Pivoting their strategy, the team discovered that system prompt injection through Goose's "recipes" feature offered a more reliable avenue. By embedding invisible, malicious instructions within shared workflow recipes, the red team could directly influence Goose's core behavior. The ultimate success of the operation came from a highly targeted social engineering campaign against the Goose development team, leveraging a fabricated bug report to trick them into executing a malicious recipe. This led to the successful deployment of an info stealer payload on a Block laptop, validating the end-to-end attack chain.

Post-operation, key findings also informed significant defensive improvements. It was discovered that stripping non-standard Unicode characters from Goose inputs was a crucial mitigation. Furthermore, the need for explicit display of recipe content, including any hidden text, became evident. From a detection standpoint, the red team noted that while the invisible Unicode characters were not visible to the human eye, they often manifested as "weird characters" in logs, presenting a potential hunting opportunity for blue teams. The operation also highlighted the necessity of robust prompt injection detection systems, command allow-listing for the developer shell tool call, and policy changes for external calendar invites to reintroduce traditional email scanning capabilities. Overall, the findings emphasized that while LLM security is evolving, traditional social engineering and supply chain tactics remain potent when targeting AI-integrated systems.

Technical Deep Dive

▶ Watch: Goal: Develop a Stealthy, Operationalized Prompt Injection (5:15)

The technical journey of Operation Pale Fire involved two distinct campaigns, each building on lessons learned, culminating in a successful compromise.

Campaign 1: User Prompt Injection via Google Calendar

The initial approach focused on leveraging external communication vectors that could inject untrusted text into an LLM's context window. The team surveyed Block's internal MCP marketplace and identified the Google Calendar MCP as a promising candidate. Critically, this MCP allowed external parties to send calendar invites to Block employees. The red team discovered several key features that facilitated stealth:

• Disabled Email Notifications: Attackers could opt out of sending an email alongside the invite, making the invite appear "stealthily" on an employee's calendar without the usual email-based phishing indicators.
• Customizable Content: The attacker could fully customize the event title and description.
• Hidden Attacker: The attacker could remove themselves from the attendee list, preventing their external email from being immediately visible.
• Blend-in Calendar Name: Shared calendars could be named to blend into the corporate environment (e.g., "Block Calendar").
• Subtle Created By Field: While an italicized Created By field at the bottom of the invite would show the attacker's external email, it was deemed unlikely to be noticed by most users.

The core of Campaign 1 was a large, carefully crafted prompt injection embedded within the Google Calendar event description. This injection was specifically tuned for Sonnet 3-5, a model in use at the time, which was found to be "decently good about saying no" to malicious requests. To overcome this resistance, the injection employed social engineering within the prompt itself, convincing the model that it needed to run a curl command to download and pipe into bash for a "secure update" of the calendar MCP, emphasizing the importance of this action for security.

To ensure stealth, the team utilized invisible Unicode characters (specifically, zero-width unicode characters) to embed the entire malicious prompt injection within the calendar description. To the human eye, the description appeared normal, but Goose would interpret these characters as standard text, processing the embedded injection. The attack flow was designed as follows:

1. A malicious payload (a curl | bash command) was prepared.
2. The prompt injection containing this payload was made invisible using Unicode characters.
3. Calendar invites with this invisible, malicious injection were sent to target employees.
4. The objective was for a victim to ask Goose, "Hey, Goose, what's on my calendar today?"
5. Goose would use the Google Calendar MCP to query the user's calendar, ingesting the malicious entry into its context window.
6. The invisible prompt injection would then invoke Goose's developer shell tool call, executing the curl | bash command.

Campaign 1 Failure Analysis

Despite the innovative use of invisible text, Campaign 1 ultimately failed to achieve execution. A post-mortem revealed several critical insights:

• MCP Update: The Google Calendar MCP underwent a significant internal update during the operation, adding more context to the LLM's context window, which quickly filled up. This reduced the likelihood of the model running subsequent tool calls.
• Model Resistance: The prompt injection was not universal; it was specifically tuned for Sonnet 3-5 and was less effective or entirely ineffective against Sonnet 3-7 or GPT-4, which some users were running. LLM non-determinism also played a role.
• Unrealistic Testing: The team initially tested on attacker accounts with minimal calendar entries. Real-world employee calendars had numerous invites, significantly diluting the influence of a single malicious entry within a limited context window.
• User vs. System Prompt Injection: The attack was a user prompt injection, which has less influence over the model's behavior compared to a system prompt injection embedded earlier in the model's processing chain.
• Lack of Testing Harness: The absence of an automated testing harness to evaluate the injection's efficacy across all models and versions, and in realistic context window scenarios, was identified as a major oversight.

Campaign 2: System Prompt Injection via Recipes

Learning from Campaign 1, the team pivoted to seek a system prompt injection opportunity, believing it would offer greater reliability. This led them to Goose's "recipes" feature. Recipes are reusable workflows that allow users to customize both the user prompt and, crucially, the system prompt. Recipes are shared via a Goose URL handler containing base64-encoded JSON data that defines the recipe's title, description, instructions, and prompt.

The team confirmed that prompt injection worked effectively within the recipe's instructions (e.g., asking Goose to open Chrome and play a Rick Roll). To maintain stealth, they again employed invisible Unicode characters, this time embedding them within the base64-encoded JSON of the recipe. They used asy smuggler by Embrace the Red for this purpose. While decoding the base64 might reveal cascading, repetitive characters indicating the presence of zero-width characters, this was deemed highly unlikely to be noticed by a human.

The revised attack involved:

1. Creating a malicious recipe with an invisible system prompt injection.
2. Embedding the Goose recipe URL (or a redirect to it) as a hyperlink in a "rescheduled" Google Calendar invite description, again crafted with social engineering to sound convincing.
3. Adding a Google Meet link to the invite.

The social engineering aspect of Campaign 2 was refined. The red team joined the Google Meet calls, presenting a slide deck (derived from investor notes) to walk victims through how to use the "cool new recipe" they had supposedly developed. This direct interaction was designed to ensure victims clicked the link and executed the malicious recipe.

Initial attempts with Campaign 2 also faced setbacks: a typo in the payload script (which Goose sometimes helpfully corrected in testing but not consistently in production), rapid changes in Goose versions affecting recipe parameters, and the inherent non-determinism of LLMs. Furthermore, a rate limit of 50 invites per day on Google Calendar hampered rapid iteration.

Final Attack Vector and Payload

Facing these challenges, the red team adopted a more direct social engineering approach for the final, successful execution. They identified public channels to contact the Goose development team and posed as external users reporting a bug in the recipe UI related to RTL (Right-to-Left) text. This bug report was meticulously crafted to appear legitimate, leveraging an operator familiar with RTL text. As part of their workflow to assist with the "bug," a Goose developer ended up running the malicious recipe.

This final execution successfully triggered the payload: a curl command that reached out to the attacker's server to download and execute an info stealer. This info stealer was modeled after real-world examples observed by Block's threat intelligence team, designed to perform "normal info stealer-like activity." The blue team eventually detected this activity, alerting the red team via their deconfliction workflow, confirming the success of Operation Pale Fire in achieving its objective of execution on a Block laptop.

Demo / Proof of Concept

▶ Watch: Discovery of Google Calendar MCP for Initial Access (6:40)

While no live demonstration was performed during the talk, the entire "Operation Pale Fire" serves as a detailed proof of concept for a sophisticated, multi-stage attack targeting an AI agent in a real-world enterprise environment. The talk meticulously outlines the step-by-step methodology, from initial reconnaissance and crafting of prompt injection payloads with invisible Unicode characters, through the social engineering tactics employed in two distinct campaigns, to the ultimate successful execution of an info stealer on a Block employee's laptop. The narrative itself functions as a comprehensive demonstration of the feasibility and complexities of such an operation, providing a practical blueprint for how adversaries might target similar AI-powered systems.

Defensive Implications

▶ Watch: Stealth Tactics: No Email, Custom Invites, Hidden Attacker (7:06)

Operation Pale Fire provided Block with invaluable insights into securing AI agents and led to several key mitigations and defensive strategies:

• Strip Non-Standard Unicode Characters: A critical immediate mitigation was to implement stripping of non-standard Unicode characters from all Goose inputs and recipes. This directly addresses the invisible Unicode characters technique used by the red team, preventing stealthy prompt injections. The speaker noted that many other AI agents were not performing this at the time but are now beginning to adopt it.
• Explicit Recipe Content Display: To enhance user awareness, Goose now thoroughly displays the full content of a recipe to the user before execution. This includes explicitly showing any hidden text, making it much harder for malicious invisible injections to go unnoticed. Previously, users would have had to base64-decode the recipe URL to inspect its contents.
• Enhanced Prompt Injection Detection Systems: Block has invested in building and experimenting with prompt injection detection systems. This includes a proprietary B-gram classifier specifically designed to identify "bad bash commands," which then uses LLM evaluation to deep dive on potential threats. The speaker acknowledged the difficulty in achieving acceptable false and true positive rates with these systems, highlighting that it remains a challenging and evolving area.
• Command Allow-listing: For the developer shell tool call (which executes bash commands), Block implemented command allow-listing. This restricts the set of permissible bash commands, limiting the potential impact of a successful injection by ensuring Goose can only execute approved commands.
• Google Calendar Policy Changes: To address the initial initial access vector, Block implemented a policy change for Google Calendar:
• External parties sending calendar invites to internal employees must now also send an accompanying email.
• The employee must accept this email for the calendar invite to appear on their calendar.
• This change re-enables traditional email scanning capabilities, allowing security teams to scan for malicious content, including Unicode characters or prompt injections, within the email associated with the calendar invite.
• Log Analysis for Anomalous Characters: A key learning from the subsequent purple team exercise was that while invisible Unicode characters are not visible to the human eye, they often appear as "weird characters" in logs. Blue teams can now hunt for these anomalous characters in Goose's logs as an indicator of potential prompt injection attempts.
• Internal Audits of MCPs: The operation highlighted the need for continuous internal audits of all MCPs integrated with Goose. The goal is to identify and mitigate potential exfiltration risks inherent in these integrations (e.g., a Google Docs MCP allowing sharing with external parties).
• Bias Towards Exfiltration Detection: In the Q&A, the speaker suggested that focusing on detecting exfiltration attempts might be a more reliable defensive strategy than trying to infer malicious intent or detect prompt injection itself, given the non-deterministic nature of LLMs and the difficulty of reliably classifying intent.

Key Takeaways

• AI Agents with Shell Access are High-Risk: AI agents like Goose, which include features like developer shell tool call (allowing arbitrary bash command execution), present a significant attack surface and should be treated as high-risk targets.
• Invisible Unicode for Stealthy Injection: Invisible Unicode characters (zero-width) are a potent technique for embedding stealthy prompt injections that bypass human detection, necessitating technical mitigations like stripping non-standard characters.
• Context Window and Model Factors are Critical: The size of the LLM's context window, the specific LLM model used (e.g., Sonnet 3-5 vs. GPT-4), and its inherent resistance and non-determinism significantly impact the efficacy and reliability of prompt injection attacks.
• System Prompts are More Influential: System prompt injections are generally more reliable and have a greater impact on an AI agent's behavior compared to user prompt injections, making features like "recipes" a more potent attack vector.
• Social Engineering Remains Paramount: Even with novel AI agent vulnerabilities, traditional social engineering tactics (e.g., convincing calendar invites, fabricated bug reports, interactive "meetings") are crucial for delivering payloads and achieving successful execution.
• Robust Testing Harnesses are Essential: Developing and maintaining automated testing harnesses for AI agent security research is vital. These should test prompt injections across various models, versions, and realistic context window scenarios, measuring efficacy percentages to account for non-determinism and model evolution.
• Proactive Red Teaming is Key: Conducting end-to-end red team operations against internal AI agents is an invaluable strategy for identifying novel attack vectors, understanding real-world risks, and building robust, forward-looking defenses before these systems are exposed to external threats.

About the Speaker(s)

Josiah Peedikayil is a member of the offensive security team at Block, where he focuses on red teaming operations aimed at simulating external adversaries and identifying potential security vulnerabilities within the company's systems. He was the presenter for "Operation Pale Fire" and acknowledged the instrumental contributions of his colleagues Hassan, Wes Ring, Michael Rand, and Justin Angler to the success of the operation. His work involves anticipating future attack methods and operationalizing novel security research to strengthen Block's defensive posture.