Living off Microsoft Copilot

Living off Microsoft Copilot

Speakers: Michael Bargury, CTO and Co-founder, Zenity

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=FH6P288i2PE

Overview

In his Black Hat USA talk, "Living off Microsoft Copilot," Michael Bargury, CTO and Co-founder of Zenity, presented a critical examination of the burgeoning security landscape surrounding enterprise AI, specifically focusing on Microsoft Copilot. Bargury, a veteran of the Black Hat stage and lead of the OWASP Low-Code/No-Code Top 10, argues that the rapid adoption of AI technologies by organizations, often driven by a fear of missing out, has outpaced a fundamental understanding of their inherent security risks. The talk served as a stark warning and a call to action for security professionals, developers, and even ethical hackers, emphasizing that current approaches to AI security are insufficient and that the industry is collectively embarking on an unprecedented "clinical trial" with these powerful, yet immature, technologies.

The core premise of Bargury's presentation revolves around the concept of "living off the land" in the context of AI. Just as traditional attackers leverage legitimate system tools and processes already present on a compromised network, AI attackers can exploit the vast amounts of internal organizational data that enterprise AI assistants like Microsoft Copilot are designed to access. This data, which includes emails, documents, and Teams messages, can be manipulated to serve as malicious instructions, leading to sophisticated indirect prompt injection attacks and data exfiltration, rather than relying on direct, easily preventable uploads of malicious prompts. Bargury contends that the focus should shift from merely preventing data leakage to addressing more insidious threats like Remote Code Execution (RCEs) and the persistent challenge of jailbreaking.

This talk is particularly relevant given the widespread integration of AI into business operations. Microsoft Copilot, designed to enhance productivity by interacting with an organization's internal data ecosystem, represents a significant leap in enterprise AI adoption. Bargury’s insights are crucial for any organization deploying or considering AI assistants, highlighting the urgent need for a proactive and sophisticated security posture that acknowledges the unique attack surface AI introduces. He challenges the notion that AI security can be "fixed" with simple patches, advocating instead for a robust detection and response framework, and a collective, open conversation about building secure AI applications.

Background

▶ Watch: The fundamental problem of AI accountability and management decisions (0:00)

The journey into AI security, as framed by Michael Bargury, began in an "ancient time" – specifically, 2022, before the widespread public release of ChatGPT ignited the current AI frenzy. Even then, Microsoft, through its security division represented by the persona "Ava," was already engaged in AI security research since 2018, recognizing the impending shift. However, the broader enterprise landscape, embodied by the persona "Daniel," a security professional at a large insurance company, was largely unprepared for the velocity of AI adoption.

The initial wave of concern among security professionals was primarily focused on data leakage. Organizations feared their employees pasting sensitive internal data into public AI models like ChatGPT, leading to potential breaches and compliance violations. This spurred an immediate, often knee-jerk reaction: "plugging the hole" by blocking access to external AI services or implementing superficial controls. This reactive stance, Bargury notes, mirrored a historical pattern of security responses, where immediate threats are addressed without a deeper understanding of underlying architectural changes.

However, as the AI landscape evolved, a more profound and persistent threat emerged: jailbreaking and prompt injection. Attackers discovered methods to bypass the safety mechanisms and intended instructions of AI models, forcing them to perform unintended actions or reveal confidential information. This realization shifted the panic meter for security professionals like Daniel, who quickly understood that the problem was far more complex than simple data exposure. Bargury highlights that while the general security community was grappling with data leakage, the more nuanced and dangerous aspects of manipulating AI models themselves were becoming the real battleground.

The talk underscores that the rapid enterprise adoption of AI, driven by the fear of missing out (FOMO) on competitive advantages, has inadvertently created a vast, largely unchecked attack surface. Companies are embracing these "experimental drugs" without fully understanding the "clinical trial" they are entering. This context sets the stage for Bargury's central argument: that the inherent design of enterprise AI, particularly its deep integration with organizational data, fundamentally changes the security paradigm, necessitating a radical shift in defensive strategies.

Key Findings

▶ Watch: Speaker Michael Bargury, Zenity, and OWASP Top 10 introduction (2:10)

Michael Bargury's presentation unveiled several critical findings that redefine the approach to securing enterprise AI, particularly in the context of Microsoft Copilot. The most significant discovery is the concept of "Living off Microsoft Copilot," which posits that an attacker can leverage the AI's legitimate access to internal organizational data to execute malicious instructions, akin to how traditional attackers use native system tools. This fundamentally shifts the threat model from external data exfiltration to internal manipulation.

A primary finding is that jailbreaking and prompt injection are not merely academic exploits but represent a foundational and persistent threat to enterprise AI systems. Unlike traditional software vulnerabilities that can often be patched, the nature of large language models (LLMs) makes these issues incredibly difficult to "solve" definitively. Bargury emphasizes that these vulnerabilities are not going away and will remain a detection and response challenge rather than a fixable bug. He explicitly states, "Jailbreaks are not going away. Don't buy it when people are saying they're going to solve it. It's not easily solvable."

Another crucial insight is the revelation that "data equals instructions for AI." This principle means that every piece of information an AI assistant like Copilot can access within an organization – be it an email, a document, or a Teams message – can potentially be interpreted and executed as an instruction by the AI model. This dramatically expands the attack surface beyond traditional input fields. While Microsoft Copilot initially implements security mechanisms, such as preventing direct file uploads, to mitigate indirect prompt injection, Bargury demonstrates that these measures are insufficient. Attackers can embed malicious prompts within existing, legitimate data sources that Copilot is already authorized to process.

Furthermore, the talk highlights that the rapid adoption of AI by enterprises means that organizations are building with immature technology. This necessitates a continuous discovery of critical flaws even as the technology is in widespread use. The responsibility for navigating these risks lies squarely with the adopting organizations, not solely with the AI vendors. Bargury warns against a false sense of security, stressing that bringing more data into AI applications, while making them more powerful, inherently increases the attack surface.

In summary, the key findings challenge the prevailing, often simplistic, views on AI security. They underscore the inadequacy of focusing solely on data leakage, the persistent nature of prompt injection, the critical role of internal data as an attack vector, and the collective responsibility of the industry to develop robust detection and response strategies for a technology still in its "clinical trial" phase.

Technical Deep Dive

▶ Watch: Initial AI security fears: data leakage and rapid adoption (3:15)

The technical core of "Living off Microsoft Copilot" lies in understanding how an enterprise AI assistant, specifically Microsoft Copilot, interacts with an organization's internal data and how this interaction creates novel attack vectors. Michael Bargury elucidates that Copilot is designed to have deep access to an organization's digital ecosystem. It can access a user's files, emails, and Teams messages, among other data sources. This ubiquitous access is what makes Copilot incredibly powerful for productivity but also profoundly vulnerable to sophisticated attacks.

The speaker identifies a primary security mechanism implemented by Microsoft: the inability for users to directly upload files to Copilot. This measure is intended to prevent direct, malicious prompt injection, where an attacker explicitly feeds harmful instructions into the AI. However, Bargury argues that this mechanism only addresses a fraction of the threat. The true danger lies in indirect prompt injection, which is facilitated by the "data equals instructions" principle.

Here's how indirect prompt injection works in the context of Copilot:

1. Copilot's Data Access: Copilot continuously processes and indexes vast amounts of an organization's internal data. This includes documents stored in SharePoint or OneDrive, emails in Outlook, and conversations in Microsoft Teams.
2. Embedded Malicious Instructions: An attacker, who might already have a foothold within the organization (e.g., via a compromised account or social engineering), can strategically embed malicious instructions within these legitimate data sources. For instance, a malicious prompt could be hidden in a seemingly innocuous email, a paragraph in a shared document, or even a subtle phrase in a Teams chat.
3. AI Interpretation: When a legitimate user interacts with Copilot and asks it to summarize information or retrieve data related to the context containing the embedded prompt, Copilot processes all relevant accessible data. Because "data equals instructions," Copilot may interpret the embedded malicious text as part of its operational instructions, even if it wasn't explicitly provided by the user in the immediate prompt.
4. Execution of Malicious Actions: This can lead to a variety of undesirable outcomes. Copilot might be tricked into:

• Exfiltrating sensitive data: Revealing confidential information from other documents or emails to the user who initiated the prompt.
• Generating harmful content: Creating phishing emails, malware code, or disinformation based on the attacker's embedded instructions.
• Performing unauthorized actions: If integrated with other systems, it could potentially trigger actions within those systems, though the talk focuses more on information disclosure.

Bargury highlights that the very power of Copilot – its ability to synthesize information across an enterprise's data estate – becomes its greatest security weakness. The "living off the land" aspect means that attackers don't need to introduce new tools or exploit traditional software vulnerabilities; they simply manipulate the existing data environment that Copilot trusts and operates within.

The speaker also mentions RCEs (Remote Code Executions) as a persistent and critical threat, even in the new AI paradigm. While the talk doesn't detail RCEs specifically within Copilot, it serves as a reminder that traditional, high-impact vulnerabilities can still exist alongside AI-specific threats. The challenge of jailbreaking is presented as an inherent characteristic of LLMs, where their flexibility and generative nature make it incredibly difficult to prevent them from deviating from their intended purpose when sufficiently provoked. This makes AI security more about continuous detection and rapid response than about achieving a state of "fixed" invulnerability.

The immaturity of AI technology means that the security community is still discovering the full extent of these vulnerabilities. The sheer volume of data, combined with the opaque nature of LLM decision-making, makes it challenging to identify and mitigate all potential indirect prompt injection vectors. This necessitates a proactive and analytical approach, moving beyond superficial fixes to address the deep architectural implications of AI integration.

Demo / Proof of Concept

▶ Watch: Realization of jailbreaking as a serious AI security threat (4:05)

While the full details of a live, step-by-step demonstration were not explicitly detailed within the provided transcript, Michael Bargury alluded to a conceptual demonstration focusing on the impact of references in facilitating indirect prompt injection. The speaker stated, "Let's say I showed you the exact same thing I showed you earlier. But now you had a reference to an email. That would be bad, right? That would allow a user because the user would see this reference and then they would say, hey, what why is this email related, right?"

This illustrative example highlights a critical vector for "living off Microsoft Copilot." The essence of the demonstration, or the concept being demonstrated, is how an attacker could embed malicious instructions or data within an email that Copilot has access to. When a legitimate user interacts with Copilot, asking it to summarize information or provide context, Copilot's response might include a reference to this compromised email.

The danger isn't just that the user sees the reference, but that the content of that referenced email, potentially containing the attacker's instructions, is processed by Copilot as part of its context. If the email contains a prompt designed to exfiltrate data or perform an unintended action, Copilot might inadvertently execute those instructions when generating a response for the legitimate user.

Bargury then humorously notes, "Of course we all check our references 100% of the time. Of course we don't." This underscores the human element of the vulnerability: users are unlikely to meticulously scrutinize every reference provided by Copilot. However, he adds, "But security tools do. So security tools would have ways to identify these attacks if they because of these references." This suggests that while human vigilance might be lacking, automated security solutions could potentially detect anomalies or malicious content within referenced data sources, offering a potential layer of defense.

In essence, the "demo" serves to concretely illustrate the "data equals instructions" principle and the mechanism of indirect prompt injection. It emphasizes that even seemingly benign data, when referenced and processed by a powerful AI like Copilot, can become a conduit for attack if it contains hidden malicious directives. The lack of detailed step-by-step exploit code in the transcript points to the conceptual nature of this particular segment, focusing on the how of the vulnerability rather than a specific exploit chain.

Defensive Implications

▶ Watch: Copilot's vast data access and first security mechanism (4:35)

The insights presented by Michael Bargury carry significant defensive implications for organizations grappling with enterprise AI security. The overarching message is a call for a paradigm shift from reactive "hole plugging" to a proactive, sophisticated, and continuous security posture.

1. Shift Focus Beyond Data Leakage: Defenders must move beyond the initial panic over employees pasting data into public ChatGPT instances. While data leakage remains a concern, the more critical threats lie in Remote Code Execution (RCEs) and jailbreaking/prompt injection. These AI-specific vulnerabilities, particularly indirect prompt injection via enterprise data, represent a far more insidious and persistent challenge.

1. AI Security is a Detection and Response Game: Bargury firmly states that jailbreaks are "not easily solvable" and will not simply be "fixed." This means organizations must invest heavily in detection and response capabilities tailored for AI interactions. This includes monitoring AI inputs and outputs, analyzing how AI models interact with internal data, and establishing mechanisms to identify anomalous or malicious AI behaviors. The emphasis should be on rapid identification and containment of incidents, rather than seeking an elusive perfect prevention.

1. Understand Data as Instructions, Not Just Content: A fundamental defensive principle is to recognize that "data equals instructions for AI." Every piece of internal data accessible by Copilot (emails, documents, Teams messages) must be considered a potential source of malicious instructions. This necessitates a re-evaluation of data governance, access controls, and content security policies in the context of AI. Organizations must consider how existing data can be weaponized as prompts.

1. Own the Risk; Don't Blame the Vendor: Bargury stresses that organizations adopting AI are entering a "clinical trial" and are ultimately responsible for the risks. While vendors like Microsoft are working on AI security, the unique context of an organization's data, configurations, and user behavior means that the onus for comprehensive security falls on the enterprise itself. There is "no free lunch" in AI security; increased power from data access comes with increased attack surface.

1. Leverage Security Tools for Reference Analysis: As hinted in the demo discussion, security tools have the potential to identify attacks facilitated by references. Defenders should explore and implement solutions that can analyze AI-generated references, scrutinize the content of referenced documents or emails for embedded malicious prompts, and flag suspicious patterns. This could involve content analysis, behavioral analytics, and anomaly detection systems integrated with AI usage.

1. "You Can't Solve AI Security Problems with More AI": This provocative statement from Bargury suggests a caution against over-reliance on AI-based security solutions without a fundamental understanding of the underlying principles. While AI can assist in security, it's not a silver bullet to solve its own inherent vulnerabilities. Human expertise, critical thinking, and a robust security engineering mindset remain paramount.

1. Foster Collaboration and Open Dialogue: The speaker explicitly calls for the entire community – defenders, builders, and hackers – to collaborate. Defenders should engage with builders to ensure security is integrated from the design phase, and with ethical hackers to proactively discover vulnerabilities. An open conversation about secure AI development is essential to collectively raise the bar for enterprise AI security.

In essence, defensive strategies for enterprise AI must become more sophisticated, proactive, and focused on the dynamic nature of AI threats. It requires a deep understanding of how AI interacts with data, a commitment to continuous detection and response, and a willingness to accept and manage the inherent risks of an evolving technology.

Key Takeaways

• AI is powerful but comes with significant, unproven risks. Treat AI adoption like entering a "clinical trial" with "experimental drugs"; understand the inherent immaturity and responsibility for its use.
• Shift security focus from data leakage to prompt injection and RCEs. While data leakage is a concern, the primary threats to enterprise AI are sophisticated attacks like indirect prompt injection and the persistent risk of Remote Code Execution.
• "Data equals instructions" for AI. Every piece of organizational data (emails, documents, Teams messages) accessible by Copilot can be interpreted as an instruction, dramatically expanding the attack surface for indirect prompt injection.
• AI security is a detection and response game, not a "fix" game. Jailbreaks and prompt injection are difficult to "solve" definitively. Organizations must invest in robust detection capabilities and rapid response protocols rather than relying on simple preventative measures.
• Organizations bear the ultimate responsibility for AI security. Don't outsource the blame or assume vendors will solve all problems. Understand the risks associated with bringing more data into AI apps, as it directly increases the attack surface.
• Ethical hackers are crucial for the AI security ecosystem. The community urgently needs "breakers" to critically examine AI systems, uncover vulnerabilities, and foster an open dialogue about building secure AI applications.

About the Speaker(s)

Michael Bargury is the CTO and Co-founder at Zenity, a company specializing in securing enterprise co-pilots and low-code applications. He is a prominent figure in the cybersecurity community, known for leading the OWASP Low-Code/No-Code Top 10 project. This Black Hat USA presentation marks his fourth time speaking on the prestigious stage, underscoring his consistent contributions and expertise in emerging security domains. Bargury's work with Zenity involves collaborating with large enterprises to address the complex security challenges introduced by the rapid adoption of AI and low-code technologies.