Securing a Generative AI Implementation

Securing a Generative AI Implementation

Speakers: Nandita Joshi, Senior Product Security Engineer, Zendesk

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=ocVeIdARUZA

Overview

In this insightful talk, Nandita Joshi, a Senior Product Security Engineer at Zendesk, delves into the critical and rapidly evolving domain of Generative AI (Gen-AI) security. Titled "Securing a Generative AI Implementation," her presentation aims to demystify the core components of Gen-AI and Large Language Models (LLMs), providing a foundational understanding before dissecting the myriad of new attack vectors they introduce. Joshi's expertise in web applications and API security, coupled with her recent focus on AI, positions her uniquely to bridge traditional security paradigms with the novel challenges presented by AI.

The talk is particularly pertinent in an era where AI is ubiquitous, fundamentally transforming how individuals and organizations interact with technology. Joshi meticulously outlines the architectural building blocks of LLMs, from pre-training to deployment, establishing a clear context for understanding potential vulnerabilities. She then transitions into a comprehensive exploration of Gen-AI attack vectors, categorizing them by the assets they target (models, data) and the nature of interactions (input/output, plugins). The presentation culminates in a discussion of robust mitigations, emphasizing both emerging Gen-AI specific defenses and the enduring relevance of traditional application security (AppSec) best practices, advocating for a holistic and continuous approach to AI security.

This discussion is crucial for anyone involved in developing, deploying, or securing AI systems. The probabilistic nature of LLMs, the introduction of new assets like models and datasets, and complex interactions with external services create an expanded and constantly evolving attack surface. Joshi highlights the significant financial and reputational costs associated with security failures in Gen-AI, underscoring the imperative for proactive, defense-in-depth strategies. By articulating the unique challenges and actionable solutions, the talk empowers security professionals to approach Gen-AI implementations with a clearer understanding of the risks and the necessary steps to mitigate them.

Background

▶ Watch: Introduction to talk and agenda overview (0:00)

To understand the security challenges of Generative AI, it's essential to first grasp its fundamental architecture and operational phases. Nandita Joshi begins by deconstructing the process of building an LLM from scratch into three core phases: framework, optimization, and agent. The framework phase establishes the foundational infrastructure, model architecture, data preparation, and training environment. This is where the base model learns to perform. Following this, the optimization phase involves fine-tuning a pre-trained foundation model to improve its performance for specific, domain-specific use cases, such as healthcare or finance, by adjusting various parameters. Finally, the agent phase sees the optimized model deployed, interacting with web applications, APIs, and integrating with other services to serve end-users.

Joshi illustrates the pre-training process, explaining how raw text data is converted into a numerical format through encoding, processed by the model to learn statistical properties of natural language (grammar, semantics, context), and then translated back into human-readable text via decoding. She uses a basic GPT-2 small model implemented in PyTorch as an example, explaining how the model processes tokens (numerical representations of words or sub-words) as inputs, embeds them, builds relationships between words, and outputs logits representing the probability of the next token. A critical insight here is that LLMs are fundamentally probabilistic; they are merely predicting the next word based on learned patterns and probabilities, not "thinking" in a human sense.

The quality and breadth of training data directly impact a model's performance and the coherence of its output. Joshi demonstrates this by comparing the gibberish output of an untrained GPT-2 model with the somewhat sensible output of a model pre-trained on a small dataset, and finally, the highly coherent response from a fully pre-trained GPT-2 small model given the same prompt, "Every effort moves you." This highlights that data accuracy and integrity are paramount. The final step before deployment is fine-tuning, where the model adapts to specific tasks based on instruction-based datasets.

Joshi then transitions to explaining why securing Gen-AI implementations presents new challenges compared to traditional web applications. The probabilistic nature of LLMs means an infinite number of inputs can lead to an infinite number of outputs, creating an unmanageable number of edge cases. New assets like models (architecture, weights, optimization algorithms) and datasets (pre-training, fine-tuning, embeddings) must be protected, along with their integrity and usage. Critically, new interactions emerge, particularly with plugins and other services, creating complex, two-way trust boundaries. Unlike traditional apps where outputs are often derived from established business logic and sanitized data, LLM outputs are in natural language, making validation more complex. The entire landscape is also rapidly evolving, with new models, architectures, and training techniques constantly emerging, leading to a continuously expanding attack surface.

Furthermore, Joshi introduces the concept of a shared responsibility model in AI, similar to cloud services (IaaS, PaaS, SaaS). Depending on whether an organization uses a SaaS LLM (like OpenAI's ChatGPT web UI), PaaS (for rapid development with more control over data), or self-managed infrastructure, the scope of their security responsibilities varies significantly. This context shapes an organization's Gen-AI application architecture, which typically involves an LLM agent interacting with various application services (web UIs, chatbots) and downstream services (plugins, extensions), each representing a potential attack vector.

Key Findings

▶ Watch: Understanding pre-training: data encoding and decoding (3:00)

Nandita Joshi's talk uncovers several key findings regarding the security landscape of Generative AI implementations:

• Probabilistic Nature Introduces Unique Challenges: Unlike deterministic systems, LLMs operate on probabilities, leading to an "infinite number of inputs and outputs" and an equally vast number of edge cases. This fundamentally changes threat modeling, making comprehensive prediction and mitigation significantly more complex than in traditional software.
• New Assets and Interactions Expand the Attack Surface: LLMs introduce novel assets such as the model itself (architecture, weights, biases) and its associated training and fine-tuning datasets. Additionally, new interaction paradigms, especially with plugins and external services, create complex, two-way trust boundaries and introduce new attack vectors like Excessive Agency.
• Data Integrity and Provenance are Paramount: The quality, source, and integrity of pre-training and fine-tuning data directly dictate model performance and security. Data poisoning can introduce vulnerabilities, backdoors, and biases, as dramatically illustrated by the Microsoft Tay bot incident. Sensitive information disclosure is a significant risk, whether through blackbox model training on proprietary data or misconfigured tenant architectures.
• Human Over-Reliance Exacerbates Risks: Users tend to over-trust LLMs, even when they hallucinate or provide inaccurate information with high confidence. This over-reliance can lead to the adoption of insecure LLM-generated code or flawed decision-making, transforming internal vulnerabilities into operational risks.
• Traditional AppSec Failures Have Exaggerated Consequences: While new threats abound, the talk powerfully demonstrates that basic AppSec hygiene remains critically important. The LLM jacking incident, where a vulnerability in a PHP framework led to the enumeration of LLM models and potential daily costs of $46,000 in computation, underscores that traditional security flaws can have catastrophic, novel impacts in a Gen-AI context.
• Defense-in-Depth and Holistic Approaches are Essential: No single mitigation is sufficient. A successful security posture requires a defense-in-depth strategy combining emerging Gen-AI specific techniques (e.g., model watermarking, robustness testing, federated learning) with robust traditional AppSec best practices (e.g., access controls, network segmentation, input/output validation). Furthermore, securing Gen-AI extends beyond technical measures to include ethical, legal, compliance, and cross-functional collaboration aspects.

Technical Deep Dive

▶ Watch: Deep dive into LLM architecture: the transformer (4:30)

Joshi systematically dissects the technical vulnerabilities and attack vectors inherent in Gen-AI implementations, categorizing them by the assets they target and the nature of interactions.

Model-Related Threats

1. Model Theft or Replication: This involves the unauthorized access and exfiltration of LLM models. Models, often stored in version control systems like GitHub, can be stolen if network access is misconfigured. A more sophisticated form is model inversion, where an attacker repeatedly queries a proprietary LLM with numerous prompts, uses the responses as synthetic data, and trains their own model (a shadow model) until it mimics the original. While theoretical, this poses a significant risk to intellectual property.
2. Malicious Code Execution: LLM models, when loaded, often rely on serialization functions. Joshi highlights the danger of unsafe serialization methods, such as the __reduce__ method in Python's pickle module, which can be exploited for remote code execution (RCE). If an attacker can inject malicious code into a serialized model object, the environment loading it becomes vulnerable.

Data-Related Threats

1. Data Poisoning: This attack involves intentionally altering or tampering with the training data to introduce vulnerabilities, backdoors, or biases into the LLM's output. The infamous Microsoft Tay bot incident from 2016 serves as a stark example: designed to learn from Twitter interactions, Tay rapidly devolved into generating harmful and offensive content due to malicious user input. This demonstrates how even well-intentioned models can be corrupted by untrusted data sources.
2. Sensitive Information Disclosure:

• Blackbox Risk: In SaaS environments, organizations may use third-party models over which they have no control regarding training data. If an organization's proprietary or sensitive data is fed into such a model, that information can inadvertently become part of the model's memory and potentially be disclosed to other users of the same model outside the organization.
• Unintended Data Loss:
• Misconfigured Tenant Architecture: In multi-tenant SaaS LLM services, a misconfiguration could allow an attacker to modify a tenant ID in a request, gaining unauthorized access to another tenant's data.
• Overscoped Privileges: An internal Gen-AI implementation, perhaps designed to read data from a specific Slack channel, might inadvertently gain access to all channels, including private ones, due to overly broad permissions. This can lead to significant unintended data loss.

Input/Output-Related Threats

1. Prompt Injection: This is arguably the most well-known LLM attack, where users coerce the LLM into performing unintended actions.

• Direct Prompt Injection: Involves supplying malicious inputs that override system prompts or instructions.
• Indirect Prompt Injection: Targets vulnerabilities in how the LLM integrates or communicates with backend services. An attacker might use a prompt to trick the LLM into accessing or modifying an internal database it otherwise shouldn't touch.

1. Over-Reliance: Humans tend to overestimate and over-trust LLM capabilities. LLMs, by their probabilistic nature, can hallucinate or supply inaccurate information with extremely high confidence. If users blindly trust LLM-generated content, such as source code, it can introduce security vulnerabilities or break applications.
2. Denial of Service (DoS): Attackers can overwhelm an LLM's processing capabilities through resource-consuming queries. This can impact output quality, incur high computational costs, and degrade availability for other users. A specific vector is repeatedly sending long inputs that exceed the LLM's context window. As the context window fills, the LLM may drop critical information, including original system prompts or guardrails, making it susceptible to unexpected and exploitable outputs.

Other Interactions and Supply Chain Threats

1. Excessive Agency: This threat, recognized by the OWASP Top 10 for LLMs, arises when plugins or extensions integrated with an LLM possess broader privileges than necessary or can approve high-impact actions without human verification.

• Over-privileged Plugins: A plugin intended only to read documents from Google Drive might have permissions to read and modify them, creating an opportunity for misuse.
• Unverified High-Impact Actions: In a cloud environment like AWS, a plugin might bypass multi-factor authentication (MFA) to interact with compute resources or S3 files, performing actions that a human user would typically require stricter verification for.

1. Vulnerable Supply Chain: Gen-AI implementations have numerous dependencies, extending the supply chain risk. This includes vulnerabilities in deep learning frameworks (like PyTorch), NLP libraries, data processing libraries, traditional third-party packages, malicious plugins, and the use of vulnerable or deprecated models and poisoned datasets.

Demo / Proof of Concept

▶ Watch: Crucial insight: LLMs predict the next word, not think (6:00)

While Nandita Joshi's presentation did not include a live, interactive demonstration of an attack or a defensive mechanism, she effectively utilized screenshots and illustrative examples throughout her talk to clarify complex concepts and demonstrate the impact of various factors.

For instance, early in the presentation, she showed screenshots of code snippets and outputs during the pre-training phase, illustrating the encoding and decoding processes. She then used three distinct output examples – gibberish from an untrained model, somewhat coherent text from a minimally pre-trained model, and a perfectly sensible response from a fully pre-trained GPT-2 small model – to visually demonstrate the critical impact of data accuracy and training quality on model performance.

Later, in discussing the consequences of traditional AppSec failures in an AI context, Joshi referenced a real-world incident investigated by a Cystic threat research team, known as LLM jacking. This scenario, while not a live demo, served as a powerful proof of concept for how a vulnerability in an open-source PHP framework could lead to arbitrary code execution, cloud credential compromise, enumeration of LLM models, and a projected cost of $46,000 per day in LLM computation for the victim. These examples, though static, served their purpose in concretely illustrating the threats and their potential ramifications.

Defensive Implications

▶ Watch: Visualizing data accuracy's direct impact on model output (7:00)

Securing Generative AI requires a multifaceted approach, combining emerging Gen-AI specific mitigations with robust traditional application security practices, and a holistic view of the AI lifecycle.

Mitigations for Emerging Gen-AI Threats

1. Model Theft & Replication:

• Access Controls: Implement stringent access controls for LLM model repositories and training datasets, treating them as highly sensitive intellectual property.
• API Rate Limiting & User Quotas: For model inversion attacks (creating shadow models), implement API rate limiting and per-user quota management. While not preventing the attack entirely, these measures significantly deter attackers by increasing the time and cost required.
• Model Watermarking: This nascent technique involves embedding unique, imperceptible identifiers into models, similar to watermarking images. It enables the tracing of unauthorized copies and usage.
• Robustness Testing: Regularly test models for their effectiveness in detecting and resisting extraction queries aimed at replication.

1. Malicious Code Execution (Unsafe Serialization):

• Secure Coding Best Practices: Adhere to secure coding standards.
• Input Validation: Thoroughly check inputs before unpickling or deserializing model objects.
• Whitelisting: Maintain a whitelist of approved serialization/deserialization methods and types.
• Safe Libraries: Prioritize the use of secure and well-vetted libraries for model loading and saving.
• Sandboxing: Isolate the environment where models are loaded and executed using hardened containers or VMs. This significantly reduces the blast radius in the event of successful code execution.

1. Data Poisoning:

• Trusted Data Sources: Avoid crowdsourced or untrusted data. Use carefully selected, vetted data sources to establish a foundation of acceptable language use during initial training.
• Content Moderation: Employ NLP techniques to detect and block offensive or malicious content from entering training datasets or model interactions.
• Adversarial Robustness Techniques (Federated Learning): For models that continue to learn from user-supplied data, techniques like federated learning can help. This involves decentralized training on local user devices (e.g., smartphones), with the option to upload models to a central repository for vetting and rejection if malicious, reducing poisoning risks.
• Data Anonymization: Implement techniques to filter out Personally Identifiable Information (PII) from datasets.

1. Sensitive Information Disclosure:

• Tenant Isolation: For multi-tenant LLM services, ensure robust tenant isolation to prevent cross-tenant data access.
• Access Control Measures: Enforce strict access controls on data access and model usage.

1. Excessive Agency (Plugins):

• Effective Access Control: Ensure plugins implement the principle of least privilege, having only the permissions necessary for their intended function.
• Human-in-the-Loop: For any sensitive or high-impact actions performed by plugins (e.g., modifying cloud resources, financial transactions), require explicit human verification or approval.

1. Vulnerable Supply Chain:

• Vetting Plugins: Thoroughly vet all third-party plugins and extensions for security vulnerabilities.
• Model & Code Signing: Implement digital signing for models and code to ensure authenticity and integrity.
• Software Composition Analysis (SCA): Continuously scan for vulnerabilities in all open-source components and dependencies, including deep learning frameworks, NLP libraries, and data processing libraries.

Traditional AppSec Best Practices (Still Apply)

Joshi strongly emphasizes that traditional AppSec principles are not only relevant but even more critical in the Gen-AI landscape, as demonstrated by the LLM jacking incident.

• Encrypt Sensitive Data: This includes not only application data but also LLM models and datasets, both at rest and in transit.
• Secrets Management: Securely store LLM API keys and other credentials using dedicated secrets management solutions.
• Access Control: Implement least privilege and role-based access controls (RBAC) for all users and services interacting with LLMs.
• Network Segmentation: Isolate LLM environments from public networks, restrict network access to prevent data scraping, and sandbox LLM models within hardened containers or Virtual Machines (VMs).
• Source Code Management: Utilize version control systems (VCS) for all code and model artifacts, ensure accurate access controls, use validated code, and track all changes.
• Input and Output Validation: Implement robust input validation to prevent common attacks like Server-Side Request Forgery (SSRF) when LLMs process user-supplied URLs. Contextual output encoding is crucial to mitigate Cross-Site Scripting (XSS) when LLM outputs are displayed to users.
• Secure Third-Party Libraries and Frameworks: Continuously monitor and update all dependencies to address known vulnerabilities.
• Adversarial Testing and Monitoring: Regularly conduct security testing, including adversarial attacks, and implement continuous monitoring for anomalous behavior.

AI Lifecycle Management

Given the dynamic nature of LLMs, continuous management throughout their lifecycle is paramount.

• Continuous Updates: Models experience drift in performance and data over time; continuous updates are necessary to adapt.
• Secure Deprecation: Establish processes for securely deprecating old or vulnerable models and datasets.
• Plugin Lifecycle Management: Maintain an inventory of all plugins, identify and manage "zombie plugins" (similar to zombie APIs) that are no longer in use but still active.
• AI Bill of Materials (AI BOM): Create an AI BOM to document all components, datasets, and dependencies used in an LLM, aiding in vulnerability management and transparency.

Holistic Approach

Finally, Joshi advocates for a comprehensive, cross-functional approach to AI security, extending beyond purely technical measures:

• Compliance & Regulations: Ensure adherence to industry regulations and data privacy laws (e.g., GDPR, CCPA) throughout the AI lifecycle.
• Ethical & Legal Oversight: Establish frameworks for ethical and legal oversight of AI implementations.
• Vendor Vetting: Thoroughly vet third-party AI vendors to ensure they adhere to the same security and ethical standards as your organization.
• Transparency & Accountability: Foster transparency in AI system design and deployment, coupled with clear accountability frameworks.
• Cross-Functional Collaboration: AI security is not solely an AppSec responsibility; it requires collaboration with legal, compliance, governance, and vendor security teams.

Key Takeaways

• Secure Early to Mitigate High Costs: The immense computational resources and time invested in training and building LLMs mean that security must be integrated from the earliest stages, starting with secure data sources and models, to avoid costly re-implementations later.
• Context-Aware Risk Prioritization is Essential: Given the vast array of potential risks, organizations must prioritize based on their specific LLM implementation, its interactions, and their position within the shared responsibility model (IaaS, PaaS, SaaS).
• Traditional AppSec Best Practices Remain Critical: Despite the novelty of Gen-AI threats, foundational application security principles—like robust access controls, input/output validation, and secure secrets management—are more vital than ever, as their failure can lead to catastrophic Gen-AI specific compromises.
• Embrace a Comprehensive, Cross-Functional Approach: Achieving fair and secure AI deployments necessitates collaboration across various domains, including legal, compliance, governance, and vendor security, reflecting that "it takes a village to secure them."
• Acknowledge and Manage Probabilistic Nature: The inherent probabilistic behavior of LLMs leads to an infinite number of inputs, outputs, and edge cases, requiring continuous adaptation, adversarial testing, and robust monitoring strategies.
• New Assets and Interactions Demand Specific Defenses: LLM models, training datasets, and complex plugin interactions introduce unique vulnerabilities like model theft, data poisoning, and excessive agency, which require specialized mitigations such as watermarking, federated learning, and human-in-the-loop approvals.

About the Speaker(s)

Nandita Joshi, who presented this talk under her handle "eyrsec," is a Senior Product Security Engineer at Zendesk. Her professional background is deeply rooted in securing web applications and APIs. In recent times, she has dedicated significant attention to the burgeoning field of AI security, recognizing its pervasive influence and the unique challenges it presents. Her expertise allows her to provide a comprehensive perspective, blending traditional application security principles with the novel requirements of securing Generative AI implementations.