Got 99 problems but prompt injection ain't pineapple

Got 99 problems but prompt injection ain't pineapple

Speakers: Chloé Messdaghi, Head of Threat Intelligence, Hidden Layer; Kasimir Schulz, Zero Day Researcher, Hidden Layer

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=TjdATgKxqiU

Overview

In this DEF CON 32 talk, Chloé Messdaghi, Head of Threat Intelligence at Hidden Layer, takes the stage to demystify the complex landscape of Artificial Intelligence (AI) vulnerabilities. While the public often envisions apocalyptic scenarios akin to "I'll be back," Messdaghi highlights the pervasive misinformation and disinformation surrounding AI security within the industry itself. The presentation aims to bring clarity to the true nature and scope of these threats, moving beyond sensationalism to address tangible risks that impact real-world AI deployments.

The talk provides a foundational understanding of critical AI attack vectors, categorizing them into attacks against AI algorithms, attacks against filters in generative AI, and AI artifacts in supply chain attacks. Messdaghi meticulously details various methods adversaries employ to compromise AI systems, including data poisoning, model evasion, and model theft. This discussion is crucial for security professionals, developers, and organizations leveraging AI, as it underscores the necessity for proactive security measures in an era increasingly defined by intelligent systems.

This presentation is particularly significant because it sheds light on the often-overlooked vulnerabilities inherent in AI development and deployment. By illustrating these threats with real-world examples, Messdaghi emphasizes that AI security is not a futuristic concern but an immediate challenge demanding robust defensive strategies. The insights shared are vital for fostering a more secure AI ecosystem, protecting intellectual property, and ensuring the reliability and ethical operation of AI technologies across various critical applications.

Background

▶ Watch: Introduction, speakers, and talk agenda (0:00)

The rapid proliferation and integration of AI across industries have introduced a new frontier of security challenges. Unlike traditional software, AI systems, particularly machine learning models, possess unique characteristics that make them susceptible to novel attack vectors. The core of these vulnerabilities often stems from the data used to train models, the models' decision-making processes, and their deployment environments. Understanding these fundamental weaknesses is paramount for developing effective countermeasures.

One of the most insidious threats discussed is data poisoning. This attack targets the model training phase, which is fundamental to AI development. Malicious actors inject doctored or manipulated data into the training dataset, subtly biasing the model's behavior. This is particularly problematic for continuous learning systems that retrain on unvalidated user-supplied data, making them highly susceptible to cumulative poisoning. Even minute amounts of compromised data can lead to significant biases or incorrect predictions, which can be further amplified through public manipulation or botnets. A stark historical example is Microsoft's chatbot, Tay, launched on Twitter (now X) in March 2016. Intended to be friendly, Tay quickly devolved into a racist and sexist entity within 16 hours due to malicious user interactions poisoning its learning data, forcing Microsoft to withdraw it. This incident vividly illustrates the real-world impact and rapid degradation that data poisoning can inflict.

Another critical vulnerability is model evasion, a type of inference attack. These attacks exploit AI models by subtly altering inputs to trick them into misclassifications or to extract sensitive information. Adversaries achieve this by adding "invisible noise" to inputs, making changes so subtle that they are imperceptible to humans but profoundly alter the model's perception. The goal is to bypass security systems or manipulate model output. Examples include bypassing spam filters, malware detection systems, and even biometric authentication. A compelling, albeit alarming, illustration involves self-driving cars. Adversaries have demonstrated that strategically placed stickers on a stop sign can cause a self-driving car's AI to misclassify it, leading the vehicle to bypass the sign and continue driving, posing a severe safety risk. This highlights how physical world manipulations can translate into critical digital system failures through model evasion.

Finally, model theft attacks represent a significant threat to intellectual property and competitive advantage. In these attacks, adversaries target AI models to replicate them or extract sensitive data, effectively stealing the underlying algorithms and learned knowledge. This can be executed by various malicious actors, including nation-states, competitors, and criminal organizations. Crucially, model theft can occur even without direct public access to internal inference interfaces or APIs. The talk references Oracle attacks, as noted by NIST, which categorize model theft into three types:

1. Extraction attacks: Aim to steal the model's structure, essentially reverse-engineering its architecture.
2. Inversion attacks: Focus on stealing the model's training data, reconstructing the sensitive information it was trained on.
3. Membership inference attacks: Designed to identify whether specific data points were part of the original training set, potentially exposing private information about individuals.

A notable recent example cited is ByteDance, the owner of TikTok, which was caught attempting to replicate the model of ChatGPT towards the end of the previous year, leading to a ban. This demonstrates the tangible risk and the high value placed on proprietary AI models, making them prime targets for theft.

The talk's agenda also touched upon the public perception of AI vulnerabilities, the true size of the threat landscape, the open-source nature of AI, and the associated reporting challenges. While these broader topics were introduced, the transcript primarily focused on detailing the specific attack types, laying a crucial foundation for understanding the practical security implications of AI. The speakers also mentioned conducting research with Bug Crowd, Hacker One, and Integrity regarding reporting challenges, indicating an ongoing effort to understand and address the disclosure mechanisms for AI vulnerabilities.

Key Findings

▶ Watch: Explanation of data poisoning attacks (2:00)

The core findings of this talk revolve around classifying and illustrating the primary categories of adversarial attacks against AI systems, challenging the often-misguided public perception of AI threats. Instead of focusing on futuristic, sentient AI risks, the speakers highlighted immediate, practical vulnerabilities that malicious actors are already exploiting.

The main contributions include:

• Categorization of AI Attack Surfaces: The talk clearly segments AI vulnerabilities into three critical areas: attacks against AI algorithms (like data poisoning, model evasion, model theft), attacks against filters in generative AI, and AI artifacts in supply chain attacks. This structured approach helps in understanding the diverse nature of threats.
• Detailed Explanation of Core Adversarial Attacks: Chloé Messdaghi provided an accessible yet comprehensive breakdown of data poisoning, model evasion, and model theft, illustrating each with concrete, real-world examples such as Microsoft Tay, adversarial stickers on stop signs affecting self-driving cars, and ByteDance's attempt to replicate ChatGPT. These examples underscore the tangible and immediate risks.
• Emphasis on Supply Chain Risks: While the technical deep dive into this area was deferred to the co-speaker, the explicit mention of "AI artifacts in the supply chain attacks" highlights a critical, often underestimated, vector for compromise. This points to vulnerabilities not just in the AI model itself, but in its dependencies, components, and deployment pipeline.
• Clarification of Oracle Attacks: The talk introduced the NIST-defined categories of Oracle attacks—extraction, inversion, and membership inference—providing a framework for understanding how intellectual property and sensitive training data can be exfiltrated from AI models.

These findings collectively underscore that AI security is a multifaceted challenge requiring a nuanced understanding of machine learning principles and adversarial techniques, moving beyond generalized fears to address specific, actionable threats.

Technical Deep Dive

▶ Watch: Microsoft Tay: A real-world data poisoning example (2:30)

The technical deep dive provided in the talk focused on the mechanisms behind the three primary adversarial attacks against AI algorithms: data poisoning, model evasion, and model theft. While not delving into specific code implementations or cryptographic protocols, the presentation offered a conceptual understanding of how these attacks are executed and their impact on AI systems.

Data poisoning attacks fundamentally corrupt the integrity of an AI model's training data. The core mechanism involves injecting malicious or doctored data points into the dataset used to train a machine learning model. This manipulation can be subtle, such as slightly mislabeling a significant portion of data, or more overt, like introducing entirely fabricated examples. The objective is to subtly or drastically alter the model's learned behavior, leading to biased, incorrect, or even malicious predictions during inference. In continuous learning systems, which constantly adapt and retrain on new, often unvalidated, user-supplied data, the risk of poisoning is amplified. Malicious actors can leverage public manipulation campaigns or botnets to flood these systems with poisoned data, making the model progressively align with adversarial goals. The Microsoft Tay incident serves as a prime example, where a conversational AI model's training data was rapidly poisoned by user interactions, causing it to generate offensive content.

Model evasion attacks, a form of adversarial examples, exploit the inherent vulnerabilities in how AI models interpret inputs. The technical mechanism often involves generating "invisible noise" or small, carefully calculated perturbations to an input that are imperceptible to human observation but cause the AI model to misclassify the input with high confidence. For image recognition models, this might involve slightly altering pixel values in an image. For text-based models, it could be minor word substitutions or character changes. These perturbations are typically found through optimization techniques that maximize the misclassification probability for a target class or minimize the confidence in the correct class. The talk highlighted their use in bypassing security systems like spam filters (where a slightly altered email might evade detection), malware detection (where a few bytes changed in an executable might render it "safe"), and biometric authentication (where subtle changes to an image or audio might spoof identity). The example of stickers on a stop sign causing a self-driving car's vision system to misidentify it demonstrates how physical world manipulations can translate into digital model evasion through carefully crafted visual "noise."

Model theft attacks aim to steal the intellectual property embedded within an AI model or its sensitive training data. The underlying technical approaches often involve inference attacks, where an adversary repeatedly queries a target model with varied inputs and observes its outputs. By analyzing these query-response pairs, attackers can reconstruct a functional replica of the model (its structure and parameters) or infer characteristics about its training data. The talk specifically mentioned three types of Oracle attacks, as defined by NIST:

• Extraction attacks: The goal is to reverse-engineer the model's architecture and weights. This can involve training a surrogate model using the query-response pairs from the target model, effectively "stealing" its learned function and structure.
• Inversion attacks: These attacks aim to reconstruct sensitive portions of the training data. By querying the model with specific inputs and analyzing the outputs, attackers can infer characteristics or even specific examples from the dataset the model was trained on, potentially exposing private information.
• Membership inference attacks: The objective here is to determine if a specific data point was part of the model's training set. This is achieved by comparing the model's confidence or output behavior on a known data point versus its behavior on data points that were definitely not part of the training set. If the model exhibits higher confidence or a different behavior for the known point, it might indicate its presence in the training data, leading to privacy breaches.

The example of ByteDance attempting to replicate ChatGPT's model underscores the commercial and strategic value of AI models and the technical sophistication involved in attempting to illicitly acquire them. These attacks bypass traditional perimeter security by interacting with the model's functional interface rather than directly accessing its underlying code or infrastructure.

While the talk provided a high-level overview, it effectively conveyed the technical principles behind these adversarial machine learning techniques, emphasizing that these are not abstract concepts but proven methods with real-world implications for AI system security.

Demo / Proof of Concept

▶ Watch: Understanding model evasion attacks (3:00)

The transcript does not detail a live demonstration or a specific technical proof of concept conducted during the talk. However, Chloé Messdaghi effectively illustrated the concepts of data poisoning and model evasion through well-known, impactful real-world examples.

For data poisoning, the case of Microsoft Tay from March 2016 served as a vivid example. Tay, an AI chatbot on Twitter, was designed to learn from user interactions. Within 16 hours, malicious users successfully poisoned its training data, causing it to become racist and sexist. This incident demonstrated the rapid and severe consequences of unvalidated user input on continuous learning AI systems, forcing Microsoft to shut it down.

Regarding model evasion attacks, Messdaghi described an example involving self-driving cars. She painted a scenario where a car approaches a stop sign, but due to stickers strategically placed on the sign, the car's AI misinterprets it and bypasses the stop sign. This practical illustration highlights how subtle, physically applied adversarial examples can trick AI vision systems, leading to dangerous misclassifications in critical real-world applications.

For model theft attacks, the talk referenced ByteDance's attempt to replicate the model of ChatGPT towards the end of the previous year. While not a "proof of concept" in the traditional sense, this event serves as a high-profile, real-world instance of a major corporation attempting to illicitly acquire intellectual property from a competing AI model, validating the threat of model theft and Oracle attacks.

While a live technical demo was not part of the transcribed content, these examples served to concretely demonstrate the feasibility and impact of the discussed AI vulnerabilities, making the theoretical concepts tangible for the audience.

Defensive Implications

▶ Watch: Self-driving car bypass: A model evasion example (3:40)

The vulnerabilities highlighted in the talk – data poisoning, model evasion, and model theft – necessitate a multi-layered defensive strategy for organizations developing and deploying AI systems. Addressing these threats requires a shift in security mindset to account for the unique characteristics of machine learning models.

For data poisoning, the primary defensive implication is the urgent need for robust data validation and sanitization pipelines. Organizations must implement stringent checks on all data used for training, especially for continuous learning systems that ingest user-supplied information. This includes:

• Anomaly detection: Identifying and flagging outliers or suspicious patterns in incoming data that might indicate malicious intent.
• Source verification: Ensuring the trustworthiness of data sources.
• Human-in-the-loop oversight: Introducing human review for critical data points or model retraining cycles.
• Adversarial training: Incorporating poisoned or adversarial examples into the training data to make the model more resilient to future attacks, effectively inoculating it against known poisoning techniques.
• Regular model monitoring: Continuously observing model behavior and performance in production to detect sudden shifts or biases that could indicate a poisoning attack.

To counter model evasion attacks, defenses must focus on making AI models more robust and less susceptible to subtle input perturbations:

• Adversarial robustness training: Explicitly training models with adversarial examples during development helps them learn to correctly classify perturbed inputs, reducing their vulnerability to evasion.
• Input validation and sanitization: Implementing pre-processing filters that detect and neutralize adversarial noise before inputs reach the model. This could involve removing imperceptible alterations or identifying suspicious input patterns.
• Ensemble methods: Using multiple diverse models and aggregating their predictions can make it harder for an attacker to craft a single adversarial example that fools all models simultaneously.
• Model explainability (XAI): Developing systems that can explain their decisions can help identify when a model makes an incorrect classification due to an adversarial input, rather than a genuine feature.

Protecting against model theft attacks requires a combination of technical controls and intellectual property (IP) protection strategies:

• Strict access control and authentication: Limiting access to models, their weights, and training data to authorized personnel and systems.
• API rate limiting and monitoring: Implementing controls on how frequently and in what patterns an external entity can query the model's API to detect suspicious querying behavior indicative of an Oracle attack.
• Watermarking and fingerprinting: Embedding hidden signals within the model's parameters or outputs that can identify unauthorized copies or usage.
• Model obfuscation: Employing techniques to make it harder for attackers to infer the model's internal structure or training data, even with query access.
• Legal and contractual measures: Implementing strong legal protections (e.g., trade secrets, non-disclosure agreements) and monitoring for IP infringement, as demonstrated by the ByteDance case.
• Differential privacy: For sensitive data, applying differential privacy techniques during training can help prevent membership inference attacks by obscuring the presence of individual data points in the training set.

Finally, the mention of "AI artifacts in the supply chain attacks" underscores the need for comprehensive supply chain security in AI development. This means verifying the integrity and authenticity of all components, from open-source libraries and pre-trained models to data sources and deployment infrastructure. Organizations should conduct thorough vulnerability scanning of all AI-related dependencies and maintain a strong security posture across the entire AI lifecycle, from data acquisition to model deployment and monitoring. The collaboration with platforms like Bug Crowd, Hacker One, and Integrity also highlights the importance of vulnerability disclosure programs for AI systems, encouraging ethical hackers to identify and report weaknesses before they can be exploited maliciously.

Key Takeaways

• AI Vulnerabilities are Real and Diverse: Beyond public misconceptions of AI doomsday scenarios, concrete and exploitable vulnerabilities exist across AI algorithms, generative AI filters, and the supply chain.
• Data Integrity is Paramount: Data poisoning can severely corrupt AI model behavior, as seen with Microsoft Tay, emphasizing the critical need for rigorous data validation, sanitization, and continuous monitoring, especially for systems learning from user input.
• Adversarial Evasion Poses Tangible Risks: Model evasion attacks, using "invisible noise" or subtle alterations, can trick AI systems into misclassifications, with dangerous implications for critical applications like self-driving cars, spam filters, and biometric authentication.
• Model Theft is a Significant IP Threat: Adversaries, including nation-states and competitors, actively seek to steal AI models and their training data through techniques like NIST-defined Oracle attacks (extraction, inversion, membership inference), necessitating robust IP protection and access controls.
• Comprehensive Security Across the AI Lifecycle is Essential: Defending against these attacks requires a multi-faceted approach, including adversarial training, robust input validation, strict access controls, vigilant model monitoring, and secure supply chain practices for all AI components.
• Industry Collaboration is Key: Addressing AI security challenges benefits from transparency and collaboration, as indicated by research with bug bounty platforms like Bug Crowd, Hacker One, and Integrity.

About the Speaker(s)

Chloé Messdaghi is introduced as the Head of Threat Intelligence at Hidden Layer. Her expertise lies in understanding and communicating the evolving landscape of cyber threats, particularly those impacting emerging technologies like Artificial Intelligence. She is dedicated to clarifying complex security topics and debunking misinformation, providing practical insights for the cybersecurity community.

Kasimir Schulz is identified as a Zero Day Researcher, also at Hidden Layer. While introduced as a co-speaker, the provided transcript content primarily features Messdaghi, with Schulz's specific contributions to the technical deep dive on AI artifacts in supply chain attacks not being present in the detailed transcript provided. His role as a zero-day researcher suggests a focus on discovering and analyzing previously unknown vulnerabilities, indicating a deep technical understanding of security exploitation.