Attacks on GenAI data & using vector encryption to stop them

Attacks on GenAI data & using vector encryption to stop them

Speakers: Patrick Walsh, Co-founder, Ironcore Labs; Bob Wall, Co-founder, Ironcore Labs

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=Lxg9YyFJ8s0

Overview

In this DEF CON 32 talk, Patrick Walsh and Bob Wall, co-founders of Ironcore Labs, shed critical light on the often-overlooked security vulnerabilities inherent in the data ecosystems surrounding modern Large Language Models (LLMs). Titled "Attacks on GenAI data & using vector encryption to stop them," their presentation dissects how Retrieval Augmented Generation (RAG), a ubiquitous technique for enhancing LLM capabilities with private data, introduces new and significant risks. The speakers challenge the prevailing industry misconception that vector embeddings, the numerical representations of data used in RAG systems, are inherently meaningless and therefore secure.

The talk asserts that these embeddings, far from being secure hashes, can be inverted to reconstruct approximations of the original sensitive data. This revelation has profound implications for data privacy and security, especially given the rapid adoption of RAG by a staggering 100% of top SaaS and security companies. Walsh and Wall argue that the widespread use of unencrypted vector data creates a massive attack surface, exposing corporate secrets, personal information, and other sensitive content to potential compromise through malicious inversion attacks.

The core message is a wake-up call for organizations leveraging GenAI: the data flowing into and residing within vector databases is not abstract or anonymized; it carries the semantic essence of its source material and can be reverse-engineered. This necessitates a fundamental shift in how data security is approached in AI systems, moving beyond traditional perimeter defenses to embrace application layer encryption for vector embeddings themselves. The speakers aim to equip defenders with an understanding of these new attack vectors and practical strategies to mitigate them, ensuring that the promise of GenAI is not undermined by critical security oversights.

Background

▶ Watch: Introduction to RAG and its importance for LLMs (0:00)

Large Language Models (LLMs) have revolutionized how we interact with information, but they come with inherent limitations. Firstly, LLMs are prone to hallucinations, generating plausible but factually incorrect information. Secondly, their training data is often stale, meaning they lack current events or proprietary knowledge. Finally, and crucially for enterprises, LLMs are typically not trained on an organization's private, sensitive data, making them unsuitable for internal queries or confidential tasks straight out of the box.

To address these shortcomings, the industry has widely adopted a technique called Retrieval Augmented Generation (RAG). RAG enhances LLMs by providing them with relevant, up-to-date, and often private information at query time, enabling them to answer questions more accurately and contextually. The process typically involves a user asking a question to a chat application. Instead of directly passing this to the LLM, the system first performs a semantic search to find documents or data chunks related to the query. These retrieved documents are then "stuffed" into the LLM's prompt alongside the user's question and chat history, allowing the LLM to generate a response based on specific, relevant information. A practical example cited is querying a "draft FQ 10Q," a private financial document, which would be impossible without RAG.

The backbone of this semantic search is often vector search, a method that differentiates between meanings of words and understands sentences in context. For instance, it can distinguish "arm" as a body part from "arm" as a division of an organization. This process begins by taking an input document, chunking it into smaller pieces (e.g., by sentence or sets of 30 words, often with overlap). Each chunk is then fed into a text embedding model, which is similar to an LLM but produces a vector embedding—a long list of very small numbers, typically hundreds or thousands in length. These numerical vectors represent the semantic meaning of the original text.

To perform searches, a query is also converted into a vector. The system then searches across a collection of stored vectors to find those that are "closest" in meaning. This proximity is determined using distance comparison operations, such as Euclidean distance. If two vectors (representing a query and a document chunk) are numerically close, their underlying meanings are considered similar. This capability relies heavily on vector databases, specialized systems designed for rapid nearest-neighbor or approximate nearest-neighbor searches across vast datasets. Companies like Pine Cone, Weviate, and Quadrant have seen hundreds of millions in venture capital investment in this space. However, traditional databases such as Mongo, Postgres, and Oracle are rapidly integrating vector capabilities, suggesting that vector search will soon be a ubiquitous feature across almost all data storage solutions.

The prevalence of RAG is staggering. Walsh and Wall conducted an informal survey, revealing that 100% of the top 10 SaaS companies and 100% of security companies are integrating LLM capabilities over private data, almost certainly relying on RAG. This means that vast quantities of sensitive corporate and personal data are being processed, chunked, and stored as vector embeddings in cloud services and vector databases, making the security of these embeddings a paramount concern.

Key Findings

▶ Watch: Understanding vector search for AI and semantic search (2:00)

The central and most critical finding of this talk directly challenges a dangerous misconception prevalent within the burgeoning GenAI industry: the belief that vector embeddings are akin to cryptographic hashes—meaningless on their own and only useful for comparison, thereby rendering them intrinsically secure. Patrick Walsh directly recounts an interaction with the CEO of a vector database company who espoused this very view, stating that "a vector was like a hash and it was basically meaningless except for comparison purposes." Walsh vehemently refutes this, highlighting two key points: first, that even hashes can be abused, and second, that the underlying premise about vectors is fundamentally incorrect.

The speakers conclusively demonstrate that vector embeddings are not meaningless. Instead, they are highly semantic representations of the original data. A key discovery is that these embeddings can be inverted back into approximations of the original text. This is achieved using an inversion model, which is specifically trained to predict sentences from vector embeddings. While the inversion might not be a perfect, byte-for-byte reconstruction, it can produce text that is semantically very close to the original. For example, the talk illustrates how an inversion model might reconstruct "our income is down" from a vector that originally represented "our earnings is down." The subtle difference does not diminish the fact that highly sensitive information, such as financial performance indicators, can be extracted from the vector itself.

This invertibility fundamentally changes the security posture of RAG systems. It means that the vectors stored in vector databases or passed through various components of the AI pipeline are not anonymized or abstract data points. They are, in fact, sensitive data in a different form. Any unauthorized access to these vector embeddings could lead to the reconstruction and leakage of the private information they represent. This finding underscores that the entire RAG data flow, from the original document to its chunked form, its vector embedding, and ultimately its inclusion in an LLM prompt, must be treated as a sensitive data pipeline requiring robust security measures. The security of GenAI data can no longer ignore the semantic content embedded within these numerical representations.

Technical Deep Dive

▶ Watch: The rise and integration of vector databases (4:10)

The architecture underpinning most private data applications leveraging GenAI is Retrieval Augmented Generation (RAG). This process begins when a user submits a query to an application, such as a chat interface. Rather than immediately routing the query to a large language model (LLM), the application first performs a semantic search against a corpus of relevant documents, which could be internal corporate records, proprietary databases, or personal files. The objective of this search is to identify and retrieve document chunks that are most pertinent to the user's question. These retrieved documents, along with the user's current query and any prior chat history, are then assembled into an augmented prompt, which is finally sent to the LLM. The LLM then uses this enriched context to generate a more accurate, relevant, and informed response. An illustrative example given is querying a "draft FQ 10Q," a sensitive, non-public financial document, which the LLM can only answer by being supplied with the document's content via RAG.

The intelligence behind this semantic search is primarily vector search. Unlike traditional keyword-based search, vector search understands the meaning and context of words and sentences. It can discern between polysemous words, such as "arm" referring to a body part, a verb, or a corporate division. The technical workflow for vector search involves several key steps:

1. Document Ingestion and Chunking: Input documents are first broken down into smaller, manageable pieces or "chunks." This chunking strategy can vary, often by sentence, fixed word counts (e.g., 30 words), or other logical divisions, frequently with overlapping sections to preserve context across boundaries.
2. Vector Embedding Generation: Each of these document chunks is then processed by a text embedding model. This model, while conceptually similar to an LLM, has a distinct purpose: it transforms natural language text into a vector embedding. A vector embedding is a high-dimensional numerical representation—a long list of small floating-point numbers, typically hundreds or even thousands in length. Crucially, these numbers encode the semantic meaning of the original text chunk. Text chunks with similar meanings will have vector embeddings that are "close" to each other in this high-dimensional space.
3. Vector Storage in Vector Databases: These generated vector embeddings are stored in a specialized system known as a vector database. The primary function of a vector database is to efficiently perform nearest neighbor or approximate nearest neighbor (ANN) searches across massive collections of vectors. Companies like Pine Cone, Weviate, and Quadrant are prominent in this field, although the functionality is increasingly being integrated into traditional relational and NoSQL databases such as Mongo, Postgres, and Oracle.
4. Query Embedding and Similarity Search: When a user submits a query, it undergoes the same embedding process, being converted into a query vector. This query vector is then used to search the vector database. The database calculates the "distance" or similarity between the query vector and all stored document vectors. Common distance metrics include Euclidean distance, cosine similarity, or dot product. The goal is to retrieve the document vectors that are numerically closest to the query vector, as these represent the most semantically relevant document chunks. For instance, if query vector 'A' is 2.2 units away from document vector 'B' but 5.6 units away from document vector 'C', the system will prioritize 'B' as more relevant.

The critical security revelation of the talk centers on the nature of these vector embeddings. Contrary to the belief that they are irreversible hashes, Walsh and Wall demonstrate their invertibility. A vector inversion model is a machine learning model specifically trained to perform the reverse operation: taking a vector embedding as input and generating a natural language sentence that approximates the original text. While not a perfect reconstruction, the output is semantically rich enough to expose sensitive information. The example provided, where an inversion model translates a vector representing "our earnings is down" back to "our income is down," clearly illustrates that the core meaning, and thus the sensitive data, is preserved and recoverable from the vector itself. This means that unauthorized access to vector databases or intercepting vector embeddings in transit could lead to the leakage of confidential information, demanding robust encryption and access controls throughout the RAG pipeline.

Demo / Proof of Concept

▶ Watch: Widespread adoption of LLMs on private data (5:15)

While the talk did not feature a live, interactive demonstration in the traditional sense, the speakers provided a clear conceptual proof of concept for the invertibility of vector embeddings. They illustrated how a vector inversion model, trained specifically for this purpose, could take a vector embedding and reconstruct an approximation of the original text.

The core example presented involved a vector derived from the phrase "our earnings is down." When passed through an inversion model, the resulting reconstructed text was "our income is down." This demonstration is crucial because it highlights that despite the numerical abstraction, the semantic content, and therefore the sensitivity, of the original data is largely preserved within the vector. The slight difference in wording ("earnings" versus "income") does not obscure the highly sensitive financial information being conveyed. This serves as a powerful testament to the fact that vector embeddings are not opaque hashes but rather semantically rich data representations that require the same, if not more, rigorous protection as the original plaintext.

Defensive Implications

▶ Watch: Debunking vector meaninglessness: Introducing inversion models (6:10)

The findings presented in this talk have profound defensive implications for any organization utilizing GenAI with private data, particularly those employing Retrieval Augmented Generation (RAG). The realization that vector embeddings are not meaningless hashes but reconstructible representations of sensitive data necessitates a fundamental re-evaluation of security postures.

1. Treat Vector Embeddings as Sensitive Data: The primary implication is that vector embeddings must be classified and protected with the same rigor as the original plaintext data. This includes data at rest in vector databases (e.g., Pine Cone, Weviate, Quadrant, or traditional databases like Mongo, Postgres, Oracle with vector capabilities), data in transit between components of the RAG pipeline (e.g., embedding models, vector databases, LLMs), and data in use (e.g., during similarity calculations).

1. Implement Application Layer Encryption: Traditional database encryption (disk encryption, TLS) is insufficient if the data is accessible and invertible once decrypted within the application layer. Defenders must consider application layer encryption for vectors. This involves encrypting the vector embeddings themselves before they are stored or transmitted, and decrypting them only at the point of use for comparison, ideally within a trusted execution environment. This ensures that even if a vector database is compromised, the stolen embeddings remain unintelligible without the decryption keys.

1. Rethink Access Controls for Vector Databases: Given the invertibility of vectors, access controls for vector databases must be as stringent as those for traditional databases holding sensitive information. Implement robust authentication and authorization mechanisms, enforce the principle of least privilege, and ensure that only authorized services and personnel can access or query the vector store.

1. Monitor for Inversion Attempts and Anomalous Access: Organizations should develop capabilities to detect unusual access patterns or attempts to exfiltrate large quantities of vector data. While direct inversion might be hard to detect, the presence of an inversion model or suspicious data egress from vector stores could indicate a compromise. Logging and auditing of vector database interactions are crucial.

1. Understand the Full Data Lifecycle and Attack Surface: The entire RAG pipeline represents an extended attack surface. This includes:

• Original Documents: Where private data initially resides.
• Chunking Process: The service responsible for segmenting documents.
• Text Embedding Models: The models that generate vectors (potentially cloud-based services).
• Vector Databases: Storage for embeddings.
• LLM Prompts: The augmented prompts containing retrieved sensitive data.
• LLM Responses: The output, which might inadvertently leak information.

Defenders need a comprehensive understanding of where sensitive data (and its vector representations) resides at each stage, who has access, and what protections are in place.

1. Vendor Security Assessments: Given that 100% of top SaaS and security companies are using RAG, organizations must rigorously vet the security practices of their GenAI service providers. Inquire specifically about how vector embeddings are protected, whether they are encrypted, and what measures are in place to prevent inversion or unauthorized access.

1. Data Minimization and Retention: Apply data minimization principles. Only embed and store vectors for data that is strictly necessary for RAG functionality. Implement clear data retention policies for vector embeddings, ensuring they are deleted when no longer required, reducing the window of exposure.

By treating vector embeddings as first-class sensitive data and implementing robust, application-layer security measures, organizations can significantly mitigate the risks highlighted by Walsh and Wall, enabling the secure adoption of GenAI technologies.

Key Takeaways

• Vector embeddings are not secure hashes: They are semantic representations of original text and can be inverted to reconstruct sensitive information.
• RAG systems create new attack surfaces: The widespread use of Retrieval Augmented Generation (RAG) means private data is being chunked, embedded, and stored in vector databases, making it vulnerable to inversion attacks.
• Application layer encryption is crucial: Protecting vector embeddings requires encryption at the application layer, not just at rest or in transit, to secure the data's semantic content.
• 100% of top SaaS and security companies use RAG: This indicates a vast and growing ecosystem where sensitive vector data is prevalent, demanding immediate security attention.
• Inversion models pose a direct threat: These models can turn seemingly abstract vectors back into readable approximations of confidential data, such as financial figures or personal details.
• Comprehensive security for the entire GenAI pipeline is essential: Every stage, from original document ingestion to vector storage and LLM prompting, must be secured with robust access controls and data protection mechanisms.

About the Speaker(s)

Patrick Walsh and Bob Wall are the co-founders of Ironcore Labs. Their company is dedicated to helping organizations protect data within their cloud applications by making application layer encryption more accessible, practical, and usable. Their expertise lies in securing data in complex cloud environments, which they extend to the emerging challenges posed by Generative AI and vector databases.