How AI Assistants Are Rewriting the Rules of End-to-End Encryption

How AI Assistants Are Rewriting the Rules of End-to-End Encryption

Speakers: Lauren Stemler, AI Security Strategist at Cisco Foundation AI

Conference: SAINTCON

YouTube: https://www.youtube.com/watch?v=HznHjAoRoOE

Overview

In an era increasingly defined by the integration of artificial intelligence into everyday applications, Lauren Stemler, an AI Security Strategist at Cisco Foundation AI, delivered a critical talk at SAINTCON addressing a burgeoning security concern: the impact of AI assistants on end-to-end encryption (E2EE). Stemler's presentation, "How AI Assistants Are Rewriting the Rules of End-to-End Encryption," posits that the fundamental assumptions underpinning secure, private communication are being challenged by the necessity for AI to process plaintext data. This shift introduces novel vulnerabilities and expands the attack surface in ways traditional cryptographic models were not designed to accommodate.

The core thesis of Stemler's talk is that while E2EE has long served as the gold standard for digital privacy, the introduction of AI assistants into messaging platforms fundamentally alters its security guarantees. These AI functionalities, such as message summarization, response generation, and writing enhancements, require access to the unencrypted content of conversations. This requirement creates a crucial intermediary decryption point, often within cloud-based AI processing infrastructure, thereby exposing sensitive information to new risks without ever necessitating a breach of the underlying cryptographic protocols.

This talk is particularly timely and significant given the rapid proliferation of AI-powered features across popular communication platforms. Stemler argues that to maintain security and privacy in this evolving landscape, a mere abandonment of E2EE is not the answer. Instead, the industry must urgently rethink and adapt its E2EE models to responsibly integrate AI, demanding greater transparency, user control, and the exploration of advanced privacy-preserving technologies. Her insights serve as a vital call to action for defenders, developers, and users alike to understand and mitigate these emerging threats.

Background

▶ Watch: Introduction and thesis: AI breaking E2EE assumptions (0:00)

The concept of encryption, the process of encoding information to prevent unauthorized access, is far from new. Its lineage stretches back to ancient hieroglyphs, evolving through historical milestones such as the German Enigma machine during World War II, which famously used complex rotor mechanisms to secure military communications. The digital age brought about more sophisticated and accessible forms of encryption. The 1970s and 1980s saw the development of public-key encryption, exemplified by algorithms like RSA, which revolutionized secure communication by allowing parties to exchange encrypted messages without first sharing a secret key.

The 1990s marked a significant turning point with the emergence of Pretty Good Privacy (PGP), making robust encryption available to the general public for email. This democratized privacy, moving it beyond the exclusive domain of military or government use, despite initial concerns from state actors. Further standardization came in the 2000s with the Internet Engineering Task Force (IETF) releasing standards for protocols like Transport Layer Security (TLS), leading to widespread adoption of encrypted communications across the internet. Today, billions of people utilize E2EE applications daily, often without realizing the intricate mechanisms safeguarding their conversations on platforms like Signal and iMessage.

At its heart, end-to-end encryption (E2EE) operates on a simple yet powerful principle: only the sender and the intended recipient can read the message. The process involves three key stages:

1. Encryption: The sender's plaintext message is scrambled into ciphertext using a cryptographic key.
2. Transmission: The ciphertext travels across the network, remaining unreadable to any intermediary, including the messaging service provider, internet service providers, or even government entities.
3. Decryption: Upon reaching the intended recipient, the ciphertext is decrypted back into its original plaintext using the recipient's unique key.
4. Authentication: A digital signature confirms the message's integrity and the sender's identity, ensuring the message hasn't been tampered with and is from the claimed source.

Stemler illustrates this with a compelling analogy: imagine sending a secret message in a locked box. You, the sender (User A), put the message in the box and lock it with your key. The postman (the messaging service platform) transports the box but cannot open it. Only your friend (User B), who possesses the matching key, can unlock and read the message. This model ensures unparalleled data security and privacy, protects against third-party surveillance from adversarial groups or governments, improves compliance management for organizations handling sensitive data, and guarantees resistance to tampering, ensuring message integrity. The core assumption, which has held for decades, is that the message remains private from everyone, including the messaging service itself, until it reaches the intended recipient.

Key Findings

▶ Watch: How end-to-end encryption works (3:00)

The central finding presented by Lauren Stemler is that the integration of AI assistants into E2EE messaging applications fundamentally compromises the core principle of end-to-end encryption. This isn't a flaw in the cryptographic algorithms themselves, but rather a paradigm shift in how messages are processed. The moment an AI assistant is introduced, the traditional E2EE model, where messages remain encrypted from sender to receiver, is broken.

The primary reason for this breakdown is the inherent requirement of AI assistants for context, and context, in the realm of natural language processing, necessitates plaintext. For an AI to perform functions like summarizing conversations, generating intelligent replies, or offering writing enhancements (e.g., adjusting tone, correcting grammar), it must be able to "understand" the content of the message. This understanding can only occur if the message is decrypted.

This leads to several critical consequences:

1. New Exposure Point: The message, which previously remained encrypted throughout its journey until the recipient's device, now undergoes decryption at an intermediate stage. This often happens in an AI processing cloud or via an external API endpoint. This point of decryption becomes a new, vulnerable exposure point where sensitive plaintext data exists outside the traditional encryption boundary.
2. Expanded Attack Surface: Instead of attackers needing to break robust cryptographic protocols—a task often considered computationally infeasible—they can now target the AI infrastructure itself. Each API endpoint or cloud service involved in AI processing represents an additional point of entry. A breach of this AI infrastructure could expose vast amounts of sensitive user data without ever "breaking crypto."
3. Lawful Access and Compliance Risk: As AI integration is a relatively new phenomenon, regulatory frameworks and compliance standards are still catching up. This creates a grey area where law enforcement or other entities might seek access to plaintext data not by compelling messaging services to decrypt messages (which they cannot do with true E2EE), but by compelling AI providers to hand over data processed by their systems. The lack of clear regulations exacerbates this risk.
4. Silent Exfiltration: AI models can be susceptible to carefully crafted malicious prompts or inputs that could inadvertently or intentionally cause them to output sensitive content. This "silent exfiltration" could occur without user awareness, as the AI assistant might be tricked into revealing information that it has processed in plaintext, bypassing traditional security controls.
5. Indirect Leakage (Model Training Data): AI models improve through training on vast datasets. If user messages, even temporarily decrypted for processing, are used to train these models, there's a risk of indirect leakage. Sensitive information from billions of messages could subtly embed itself within the model's parameters, potentially leading to accidental revelations or inferences that compromise privacy.

These findings collectively highlight a fundamental shift in the security landscape of digital communication. The advent of AI assistants forces a reevaluation of what "end-to-end" truly means when an intelligent intermediary is involved, creating a pressing need for innovation in privacy-preserving AI.

Technical Deep Dive

▶ Watch: Key benefits of end-to-end encryption (5:00)

The technical implications of integrating AI assistants into end-to-end encrypted messaging are profound, fundamentally altering the architectural model of secure communication. Traditionally, E2EE operates on a strict client-side encryption paradigm:

• Traditional E2EE Architecture:

1. User A (Sender): Composes a message in plaintext.
2. Encryption: The message is encrypted on User A's device using cryptographic keys (e.g., a public key of User B and A's private key for signing). It becomes ciphertext.
3. Messaging Service Platform: The ciphertext is transmitted through the platform. At no point does the platform have access to the plaintext.
4. Decryption: Upon arrival, the ciphertext is decrypted on User B's device using their private key.
5. User B (Recipient): Reads the original plaintext message.

This model ensures that the encryption boundary encompasses the entire journey between the two end-users, guaranteeing that no intermediary can read the content.

With the introduction of AI assistants, this architecture undergoes a significant transformation:

• AI-Augmented E2EE Architecture (Common Model):

1. User A (Sender): Composes a message in plaintext.
2. Encryption: The message is encrypted on User A's device.
3. Transmission to AI Processing: The encrypted message is sent not directly to the messaging service for mere relay, but often to an AI processing cloud or an external API endpoint.
4. Decryption for AI Processing: Crucially, the message is decrypted within this AI processing environment. This decryption is necessary because the AI assistant requires the plaintext to understand the context and perform its functions.
5. AI Analysis and Action: The AI analyzes the plaintext content to:

• Generate message responses: Crafting smart replies based on conversation context.
• Summarize conversations/documents: Condensing long exchanges into brief summaries.
• Provide writing enhancements: Suggesting grammatical corrections, tone adjustments, or rephrasing.

1. Temporary Storage: To maintain context across an ongoing conversation, the AI model may temporarily store portions of the plaintext information. This stored data itself becomes a potential point of compromise.
2. Re-encryption/Forwarding: After processing, if the AI generates a response or summary, this new content might be re-encrypted and then transmitted via the messaging service to User B, or the original message's processed insights are used to augment the user experience.

The critical distinction here is the intermediate decryption step. The message's plaintext now exists, even if momentarily, outside the direct control of the sender and recipient, within a third-party AI system. Stemler highlights the difference between on-device processing and cloud processing:

• On-Device AI Processing: If the AI computation occurs entirely on the user's device (e.g., a local large language model), the plaintext exposure is significantly reduced. The message never leaves the user's device in an unencrypted state for AI processing by a third party. This is a safer option, as the data stays within the user's personal security perimeter.
• Cloud-Based AI Processing (External API): This is the more common and riskier scenario. When messages are sent to an external API for AI processing, there are two primary points of exposure:

1. During Transmission: The message, once decrypted for the AI, might be in transit to or from the AI service.
2. At the API Endpoint: The plaintext resides on the AI provider's servers, within their data centers, where it is analyzed and processed. This becomes a prime target for attackers, as it consolidates sensitive data from potentially millions or billions of users.

The architectural shift means that the security of a user's communication is no longer solely dependent on the strength of their cryptographic keys or the integrity of the messaging service. It now also hinges on the security posture of the AI provider, their cloud infrastructure, and their API endpoints. This significantly broadens the attack surface and introduces new vectors for data compromise that traditional E2EE was never designed to protect against.

Demo / Proof of Concept

▶ Watch: AI assistant capabilities: generating, summarizing, enhancing messages (7:00)

Lauren Stemler underscored the immediate and tangible nature of these AI-driven security risks by citing two prominent real-world case studies: Echolak and the McDonald's Olivia bot incident. These examples serve as concrete proofs of concept, demonstrating how AI's need for plaintext can lead to significant data breaches without the need to "break" cryptographic encryption.

The first case, Echolak, emerged during the summer prior to the talk and garnered significant attention in cybersecurity circles. A research group named AIM Labs described Echolak as the "first weaponizable zero-click attack chain on an AI agent resulting in the complete compromise of Copilot data integrity." This attack specifically targeted Microsoft Copilot, an AI assistant integrated into Microsoft applications like Outlook, widely used in organizational settings.

The mechanism of Echolak was insidious: an attacker could craft a malicious email. When this email was received, it would act as a malicious prompt that would trigger Microsoft Copilot. Copilot, designed to scan documents and long emails to provide summaries or assist with responses, would then inadvertently exfiltrate the sensitive contents of those documents or emails to an attacker-controlled endpoint. The critical impact was that attackers gained access to plaintext data—documents, emails, and other sensitive information—that the AI had legitimate permission to read, all without ever breaching the underlying encryption of email communication or Microsoft's network. This demonstrated a novel attack vector where the AI itself, acting on a deceptive prompt, became the instrument of data leakage.

The second case study involved the McDonald's Olivia bot. McDonald's Corporation had deployed an AI chatbot named Olivia to screen job applicants, a common application of AI in human resources. Researchers discovered a critical vulnerability in this system: the admin panel for the McDonald's Olivia bot was publicly accessible and secured with astonishingly weak credentials—the username admin and the password 123456.

Once logged into this poorly secured admin panel, the researchers found millions of individuals' sensitive data. This included extensive chat logs from applicant interactions, full resumes, names, email addresses, and other personal information. This incident highlighted a different, yet equally dangerous, facet of AI-related security risks: the often-overlooked backend infrastructure supporting AI services. While not a direct cryptographic bypass like Echolak, it showcased how a weak link in the AI's operational deployment—its administrative interface—could expose vast quantities of plaintext data. Stemler succinctly summarized the implication: "Why would I bother hacking into McDonald's when I could just go to this AI backend?" This underscores the point that AI services, by their nature of handling and processing sensitive data, create attractive, and often less fortified, targets for attackers.

Both Echolak and the McDonald's Olivia bot provide compelling evidence that the security concerns raised by AI integration into E2EE are not hypothetical. They are active, exploitable vulnerabilities that have already led to significant data compromises, necessitating an urgent reevaluation of security strategies in this evolving technological landscape.

Defensive Implications

▶ Watch: How AI fundamentally changes the traditional E2EE model (8:00)

Given the fundamental shift in the security landscape brought about by AI assistants, Lauren Stemler outlined several crucial defensive implications and forward-looking strategies. These recommendations aim to protect user privacy without necessitating a complete abandonment of the convenient and powerful functionalities AI offers.

The immediate steps revolve around transparency and user control:

1. Transparency in Data Processing: Users must be explicitly informed about how their data is processed. Specifically, they need to know whether the AI processing of their messages occurs locally on-device (which is significantly safer as plaintext never leaves their personal device) or in the cloud via external APIs (which introduces the new exposure points discussed earlier). This transparency empowers users to make informed decisions about their privacy.
2. Opt-in Features by Default: Rather than having AI features enabled by default, Stemler advocates for an opt-in model. Given the inherent risks and the general public's limited awareness of these issues, AI assistance should be off by default. Users who understand and accept the trade-offs can then choose to enable these functionalities. This respects user autonomy and protects less tech-savvy individuals.
3. Updated Definitions of End-to-End Encryption: Current definitions of E2EE often imply direct communication between two users, with intermediaries unable to read messages. When an AI processing tool is inserted into this flow, the endpoint effectively changes from "User A to User B" to "User A to AI to User B." Messaging service providers leveraging AI must transparently disclose this altered data flow and update their E2EE claims to accurately reflect where plaintext data is accessible.
4. Group Chat Exposure Considerations: A critical, often overlooked, implication is the risk within group chats. If one member of a group chat has AI assistance features enabled, even if all other members have them disabled, the entire conversation's plaintext could be exposed to the AI processing system. This raises significant ethical and privacy concerns, as one user's choice could inadvertently compromise the privacy of others. Platforms need to address how such scenarios are handled and communicated.

Looking towards more advanced and ideal solutions for protecting privacy in an AI-augmented world, Stemler highlighted emerging cryptographic and architectural approaches:

1. Homomorphic Encryption: The "holy grail" for privacy-preserving AI, homomorphic encryption would allow AI models to perform computations on encrypted data without ever decrypting it. This means the AI could generate summaries or responses while the underlying messages remain in ciphertext. While significant progress has been made, fully homomorphic encryption (FHE) is still computationally intensive and not yet practical for widespread, real-time messaging applications. However, it represents an aspirational target for future development.
2. Trusted Execution Environments (TEEs): Trusted Execution Environments are secure, isolated areas within a processor that can run code and process data with strong integrity and confidentiality guarantees, even if the rest of the system is compromised. For sensitive AI computations, TEEs could be used to create a secure enclave where messages are decrypted, processed by the AI, and then re-encrypted, all within a hardware-backed, isolated environment. This would significantly reduce the risk of plaintext exposure during the AI processing phase.
3. Hybrid or Split AI Models: This approach involves intelligently dividing AI computation tasks. Sensitive AI computations (those requiring access to highly private data) would be performed on-device, leveraging local models. Non-sensitive computations or those requiring vast computational resources could still be handled in the cloud via API endpoints, but only with data that has been carefully anonymized or determined not to be privacy-critical. This strategy seeks to balance the benefits of cloud AI with the imperative of privacy.

Stemler's overarching message for defenders is to recognize that privacy models must evolve in tandem with technology. The industry cannot simply bolt AI onto existing E2EE frameworks without rethinking fundamental assumptions. Responsible AI development demands building security and privacy in from the ground up, ensuring that as technology advances, so too does the protection of user data.

Key Takeaways

• AI Assistants Break E2EE Assumptions: The integration of AI assistants into messaging apps fundamentally alters the core premise of end-to-end encryption by requiring access to plaintext for functions like summarization and response generation.
• New Attack Surface Emerges: AI processing clouds and external API endpoints create novel exposure points where sensitive plaintext data resides, significantly expanding the attack surface beyond traditional cryptographic vulnerabilities.
• Real-World Breaches Confirm Risks: Incidents like Echolak (Microsoft Copilot exfiltrating data via malicious prompts) and the McDonald's Olivia bot (weakly secured AI backend exposing applicant data) demonstrate that these are not theoretical risks but active, exploitable vulnerabilities.
• Privacy Models Must Evolve Responsibly: As AI technology advances, privacy frameworks must adapt. Merely adding AI features without re-evaluating security implications leads to critical gaps in data protection.
• Transparency and User Control are Immediate Needs: Messaging platforms must provide clear transparency regarding where and how AI processes user data (on-device vs. cloud) and implement opt-in mechanisms for AI features, rather than enabling them by default.
• Future Solutions Involve Advanced Cryptography and Secure Architectures: Long-term solutions include research into homomorphic encryption for processing data in an encrypted state, utilizing Trusted Execution Environments (TEEs) for secure AI computation, and developing hybrid AI models that balance on-device and cloud processing for sensitive data.

About the Speaker(s)

Lauren Stemler is an AI Security Strategist at Cisco Foundation AI, a role she transitioned into following Cisco's acquisition of Robust Intelligence, where she previously worked. Her professional background also includes experience at Splunk and the University of Utah. Stemler is an alumna of Utah State University, where she completed both her undergraduate and master's degrees. Based in Utah, this presentation at SAINTCON marked her inaugural speaking engagement at a conference, a significant milestone in her career. Her work focuses on the critical intersection of artificial intelligence and cybersecurity, specifically addressing emerging threats and vulnerabilities introduced by AI technologies.