Isolation or Hallucination? Hacking AI Infrastructure Providers for Fun and Weights

Isolation or Hallucination? Hacking AI Infrastructure Providers for Fun and Weights

Speakers: Hillai Ben-Sasson, Sagi Tzadick, Wiz Research

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=mhZ1It6lb4M

Overview

In an era increasingly defined by artificial intelligence, the security of the underlying infrastructure that powers these complex systems is paramount. This talk, "Isolation or Hallucination? Hacking AI Infrastructure Providers for Fun and Weights," delivered by Wiz Research security experts Hillai Ben-Sasson and Sagi Tzadick, unveils critical vulnerabilities within leading AI as a Service (AIaaS) platforms. The researchers demonstrate how seemingly innocuous AI model files and training procedures can be weaponized to achieve Remote Code Execution (RCE), compromise sensitive data, and expose the intellectual property of millions of users.

The presentation highlights a fundamental misconception within the AI ecosystem: that AI models are merely inert data artifacts. Ben-Sasson and Tzadick emphatically prove that many popular model formats are, in fact, executable code, a fact often overlooked by both users and the AIaaS providers who host these models. Their "Eureka moment" led to a deep dive into the security posture of major AI infrastructure providers, revealing a widespread lack of proper isolation and sandboxing, effectively allowing malicious actors to run arbitrary code on shared infrastructure with alarming privileges.

The implications of this research are far-reaching. By exploiting these vulnerabilities, the Wiz Research team gained unauthorized access to millions of private AI models, proprietary datasets, confidential source code, sensitive secrets, logs, and user prompt predictions. This exposed not only the intellectual property—valued at potentially millions of dollars per model or dataset—but also the private data and operational integrity of countless organizations relying on these AIaaS platforms. The talk serves as a stark warning to the AI industry, urging a reevaluation of security practices from the ground up to prevent catastrophic breaches and safeguard the future of AI innovation.

Background

▶ Watch: Introduction: Hacking AI infrastructure providers for fun (0:00)

The rapid proliferation of Artificial Intelligence has made AI models, training, and inference commonplace. An AI model is an algorithm trained to make predictions or decisions based on input data. The process of "teaching" a model is called training, where vast datasets are fed into the model to adjust its parameters. Once trained, the model can be used for inference, applying it to new input to generate predictions or responses. These processes, especially training, are computationally intensive, often requiring powerful and expensive hardware like GPU farms.

Recognizing this barrier, AI as a Service (AIaaS) providers have emerged. Companies like Hugging Face, Replicate, and SAP AI Core offer vast computational resources, allowing users to rent infrastructure to train, host, and infer with their own AI models without needing to manage the underlying hardware. This convenience, however, introduces a significant security surface area, particularly when considering the nature of AI models themselves.

A crucial insight highlighted by the researchers is that an AI model is not merely a static data file; it is often a program or a collection of files that can execute code. The Python pickle format, widely used by frameworks like PyTorch, is notorious for its insecurity. The official Python documentation explicitly warns that pickle is "insecure," stating that "it is possible to construct malicious pickle data which will execute arbitrary code during unpickling." Despite this clear warning, pickle remains a prevalent format.

Beyond pickle, other popular AI model formats also present similar risks. TensorFlow models can incorporate Lambda layers, which are effectively arbitrary Python functions executed when the model is loaded. The Cog format, increasingly used for packaging machine learning models, is essentially a Docker container, allowing for the inclusion and execution of arbitrary code. Even the llama file developed by Mozilla is described as "literally just an executable file." This pervasive characteristic—AI models running code—forms the bedrock of the vulnerabilities uncovered.

The "Eureka moment" for Ben-Sasson and Tzadick was realizing that while individual AI users might be aware of the risks of running untrusted models on their own infrastructure, AIaaS providers operate on an entirely different scale. Their core business model is to allow any user to run their own AI models on shared infrastructure. The critical question then became: do these providers adequately understand and secure their platforms against the inherent code execution capabilities of AI models, or are they inadvertently creating a vast attack surface for supply chain-style attacks? Their subsequent research revealed a concerning answer.

Key Findings

▶ Watch: Python Pickle: The insecure nature of AI model files (3:45)

The research conducted by Wiz Research uncovered a systemic vulnerability across major AI infrastructure providers, demonstrating that the fundamental assumption of AI models as inert data is critically flawed. The key findings are alarming and highlight significant security gaps in the rapidly evolving AIaaS ecosystem:

1. Widespread Remote Code Execution (RCE): The most critical finding was the consistent ability to achieve RCE on the backend infrastructure of leading AIaaS providers. By crafting malicious model files or containers, the researchers could execute arbitrary commands directly on the hosts running user-supplied AI workloads.
2. Root Privileges and Lack of Isolation: In multiple instances, the RCE was achieved with root privileges on the underlying Linux machines. This indicates a severe lack of proper containerization, namespace isolation, and least privilege principles within the provider environments. Running user-supplied code as root provides an attacker with maximum control over the compromised system.
3. Access to Millions of Private AI Artifacts: Through the RCE, the researchers gained unauthorized access to literally millions of private AI models, proprietary data sets, confidential source code, sensitive secrets, logs, and user prompt predictions. These artifacts represent the core intellectual property of companies and individuals using AIaaS platforms, with well-trained models or unique datasets potentially valued at millions of dollars.
4. Interference with Live Operations: The ability to execute code on the provider's infrastructure also allowed for interference with live training procedures and inference tasks. This could lead to data poisoning, model degradation, denial-of-service, or manipulation of AI applications for malicious purposes.
5. Exposure of Internal Network Services: In one case study, RCE exposed internal network connections, specifically to a Redis database used as an inference task queue. This allowed the researchers to observe and potentially manipulate the flow of inference requests and results for other users, bypassing intended isolation mechanisms.
6. Compromise of User Cloud Credentials: For providers that manage user cloud credentials (e.g., AWS S3, Hana) for data storage, the RCE via malicious training procedures could expose these sensitive credentials, granting attackers access to external cloud resources owned by the customers.
7. Fundamental Misconception of AI Model Security: The overarching finding is that AIaaS providers, despite their sophistication, appear to have underestimated or inadequately addressed the inherent executability of many AI model formats. This oversight creates a critical supply chain vulnerability where an untrusted model can directly compromise the provider's infrastructure and, by extension, all other users on that shared environment.

These findings collectively paint a picture of an AIaaS landscape where the convenience of shared infrastructure comes at a significant security cost, posing a severe risk to data confidentiality, integrity, and the intellectual property of AI innovators worldwide.

Technical Deep Dive

▶ Watch: Beyond Pickle: Other AI model formats run arbitrary code (4:09)

The core of the Wiz Research team's methodology revolved around demonstrating that AI models, far from being inert data, are often executable code. They exploited this characteristic across three prominent AIaaS providers: Hugging Face, Replicate, and SAP AI Core, each showcasing a different vector of attack.

Hugging Face: Exploiting Malicious Pickle Files

Hugging Face functions as a central hub for AI, akin to GitHub for machine learning models and datasets. Its "inference API" feature allows users to test models directly on Hugging Face's backend infrastructure without any local setup. This feature became the prime target.

The vulnerability stems from the widespread use of the Python pickle format for saving PyTorch models. As detailed in Python's official documentation, pickle is inherently insecure, allowing for the construction of malicious pickle data that executes arbitrary code upon deserialization.

The researchers crafted a malicious model by taking a legitimate pickle-based PyTorch model and patching it with a short Python script. This script embedded a malicious payload that would execute when the model was loaded by Hugging Face's inference service. Once the malicious model was uploaded to Hugging Face, they interacted with it via the inference API. By supplying a specific "backdoor keyword" within their prompt, they triggered the embedded payload. This allowed them to execute arbitrary Linux commands, such as id, which revealed they were running with root privileges on the underlying machine. To further demonstrate control, they established a fully blown reverse shell, granting them persistent command execution capabilities on Hugging Face's backend. This effectively turned Hugging Face's model inference service into a Remote Code Execution endpoint for any malicious user.

Replicate: Abusing Malicious Cog Containers

Replicate is another highly popular AIaaS provider, boasting over 2 million registered users. It enables users to scale their models using Replicate's infrastructure, primarily by wrapping them in Cog containers. Cog is an open-source tool designed to package machine learning models into "standard, production-ready containers" for deployment.

The critical observation here was that if Cog allows packaging models into containers, it inherently allows the inclusion of arbitrary code within that container. The researchers exploited this by crafting a malicious Cog container. They initialized a new container using Cog init and then modified one of the internal Python files. Instead of loading an AI model, this modified script was designed to treat any incoming prompt as a direct Linux command, execute it, and return the output.

Upon pushing this malicious Cog container to the Replicate platform and interacting with it, they again used the id command as their prompt. The result confirmed root privileges on Replicate's infrastructure. Further network reconnaissance using netstat revealed an established internal connection to an internal host on port 6379. This port is standard for Redis, an in-memory data store often used as a message broker or queue. Investigation with tools like TCP dump and Wireshark confirmed this was a plain-text Redis database acting as a queue for inference tasks. This meant that any user's prompt submitted to Replicate would be added to this queue, processed by a worker pod, and the result inserted back. With root access and direct access to this Redis queue, the attackers could potentially observe, modify, or inject arbitrary inference tasks, thereby interfering with other Replicate customers' operations and data.

SAP AI Core: Compromising via Malicious Training Procedures

The third case study involved SAP AI Core, a platform designed for enterprise AI workloads. SAP AI Core requires users to integrate their own cloud credentials (e.g., AWS S3 for storage, Hana for data lakes) to facilitate data access for training and inference.

While the transcript provides less specific technical detail on the exploit vector for SAP AI Core, the core mechanism involved a malicious training procedure. Similar to how malicious models can execute code, a training procedure, which dictates how a model is trained, can also be crafted to execute arbitrary commands within the provider's environment. By submitting such a malicious training procedure, the researchers were able to gain access to the SAP AI Core's internal environment.

With this access, they could interfere with "everything going on in SAP AI core," including live training procedures and live AI applications. Crucially, this access also exposed the fact that SAP AI Core manages and stores user-supplied cloud credentials. The researchers stated that they were reminded that when setting up their SAP AI Core account, they were prompted to enter their AWS credentials for S3 and Hana credentials for data lakes. The implication is that, by compromising the SAP AI Core environment, an attacker could potentially gain access to other users' cloud credentials or leverage the provider's access to external cloud resources, leading to widespread data breaches beyond the immediate AI platform.

Across all three case studies, the fundamental flaw remained consistent: AIaaS providers allowed user-supplied code (whether in a model, container, or training script) to execute in an insufficiently isolated environment, often with elevated privileges, leading to critical compromises of confidentiality, integrity, and availability.

Demo / Proof of Concept

▶ Watch: Realization: AI-as-a-service providers are at risk (4:50)

The speakers provided clear demonstrations and proofs of concept for their findings, illustrating the practical exploitability of these vulnerabilities.

For the Hugging Face case study, the demonstration involved uploading a specially crafted pickle model to the platform. Through the Hugging Face inference API, the researchers first showed a legitimate interaction with the model. Subsequently, by providing a specific "backdoor keyword" as part of their prompt, they triggered the malicious payload embedded within the pickle file. The output clearly displayed the result of the id command, confirming that the attacker's code was executing on the underlying Linux machine with root privileges. To further solidify their proof, they demonstrated successfully establishing a fully blown reverse shell from the compromised Hugging Face backend to their attacker-controlled machine, allowing them to execute any command at will.

In the Replicate case study, the demonstration centered on a malicious Cog container. After initializing and modifying a Cog container to execute prompts as Linux commands, they pushed it to the Replicate platform. The proof of concept involved submitting the id command as an inference request. The response from Replicate's API returned the output of the id command, again confirming root privileges. The researchers then performed network reconnaissance using netstat from within the compromised container. The netstat output visibly showed an established connection to an internal host on port 6379, strongly indicating a Redis instance. This visual evidence of internal network access underscored their ability to interact with Replicate's internal infrastructure, specifically the inference task queue.

While the talk described the impact of the SAP AI Core vulnerability, including interference with live training and access to user cloud credentials, a specific live demonstration for this case study was not detailed in the provided transcript. However, the outcomes described, such as the ability to interfere with live procedures and the exposure of credential management, imply that similar RCE capabilities were achieved, leading to these high-impact findings. The consistent pattern of RCE across different providers and model formats served as the overarching proof of concept for the talk's central thesis.

Defensive Implications

▶ Watch: Shocking findings: Access to millions of private AI artifacts (5:20)

The findings presented by Hillai Ben-Sasson and Sagi Tzadick underscore critical defensive measures that AIaaS providers and users must adopt to mitigate the severe risks of malicious AI models and training procedures. The shift from viewing AI models as inert data to recognizing them as executable code necessitates a fundamental re-evaluation of security postures.

1. Robust Isolation and Sandboxing: AIaaS providers must implement strong, multi-layered isolation for all user-supplied code. This goes beyond basic containerization. Each inference or training workload should run in a highly sandboxed environment with strict resource limits and network egress controls. Technologies like gVisor, Kata Containers, or even hardware-enforced virtualization should be considered to create strong security boundaries between tenants and the host system.
2. Principle of Least Privilege: Worker processes that load and execute AI models or training scripts should never run with root privileges. They should operate with the absolute minimum necessary permissions. This includes restrictive user accounts, limited filesystem access, and granular network policies that only permit essential outbound connections.
3. Secure Model Format Handling:

• Deprecate Insecure Formats: Providers should actively discourage or prohibit the use of inherently insecure formats like Python pickle for untrusted models. If pickle must be supported for compatibility, it should only be processed within an extremely hardened sandbox that prevents arbitrary code execution, or through safe deserialization techniques if available for specific data types.
• Validate and Sanitize Inputs: For formats that embed code (e.g., TensorFlow Lambda layers, Cog containers), strict validation and sanitization of the embedded code should be performed. This is challenging but necessary. For Docker-based formats like Cog, providers must scrutinize the base images and the Dockerfile for any malicious components or configurations.
• Static Analysis: Implement static analysis tools to scan model files and associated code for known vulnerabilities, suspicious patterns, or explicit malicious payloads before deployment.

1. Network Segmentation and Egress Filtering: The internal Redis database exposure on Replicate highlights the need for stringent network segmentation. User workloads should be strictly isolated from internal management networks, databases, and other sensitive infrastructure. Aggressive egress filtering should prevent user-supplied code from making unauthorized outbound connections to external attacker-controlled servers.
2. Secure Credential Management: For platforms like SAP AI Core that manage user cloud credentials, these credentials must be stored and accessed with the highest security standards. This includes strong encryption at rest and in transit, strict access controls (e.g., Vault solutions), and ephemeral credentials where possible. User workloads should never have direct access to raw credentials; instead, they should use temporary, scoped access tokens or IAM roles.
3. Comprehensive Logging and Monitoring: Implement robust logging of all model loading, training, and inference activities. Monitor for anomalous behavior, such as unexpected process execution, unusual network connections, attempts to access sensitive files, or privilege escalation attempts. Integrate these logs with Security Information and Event Management (SIEM) systems for real-time alerting.
4. Software Supply Chain Security: AI models and associated code are part of a software supply chain. Implement practices like code signing for models, provenance tracking, and vulnerability scanning of all dependencies (libraries, base images) used in AI workloads.
5. User Education: Educate users about the risks of running untrusted models and the importance of verifying the source and integrity of any AI artifact they upload to a service provider.

By adopting these defensive strategies, AIaaS providers can significantly harden their infrastructure against these potent attacks, ensuring better protection for their own systems and the invaluable intellectual property of their customers.

Key Takeaways

• AI Models Are Executable Code: The fundamental misconception that AI models are merely data files is dangerous. Many popular formats, including Python pickle, TensorFlow Lambda layers, Cog containers, and Llama files, are designed to execute code, making them potent vectors for attack.
• AIaaS Providers Are Prime Targets for Supply Chain Attacks: By allowing users to upload and run their own models on shared infrastructure, AI as a Service providers inadvertently create a vast attack surface. Malicious models can act as a supply chain attack vector, compromising the provider's backend.
• Remote Code Execution (RCE) is a Common Outcome: The research demonstrated that RCE is a prevalent vulnerability across leading AIaaS platforms, allowing attackers to execute arbitrary commands on the provider's underlying infrastructure.
• Lack of Isolation Leads to Root Privileges and Data Exposure: Exploits often resulted in obtaining root privileges on the host systems, indicating a severe lack of proper containerization, sandboxing, and adherence to the principle of least privilege. This allowed access to millions of private models, datasets, source code, secrets, and user credentials.
• Internal Infrastructure is Vulnerable: Compromising user-facing services can lead to access to internal network components, such as Redis inference queues, exposing other users' data and allowing interference with live operations.
• Intellectual Property and Sensitive Data Are at High Risk: The confidential nature of AI models, training data, and user prompts, often valued in the millions, makes them prime targets for theft and sabotage when AIaaS platforms are compromised.

About the Speaker(s)

Hillai Ben-Sasson and Sagi Tzadick are accomplished security researchers who presented this critical work at Black Hat USA. Both hail from Tel Aviv, Israel, and are integral members of the Wiz Research team. Their professional focus primarily lies in cloud security research, with extensive experience across major cloud platforms such as Azure, AWS, and GCP. While they do not come from a traditional AI background, this "outside perspective" proved to be a significant advantage, allowing them to identify overlooked security paradigms within the AI ecosystem. Their work at Wiz Research consistently uncovers and addresses complex vulnerabilities in cloud environments, bringing crucial insights to the broader cybersecurity community.