MLOps Under Attack: Threat Modeling Modern AI Systems - Sandeep Singh

MLOps Under Attack: Threat Modeling Modern AI Systems

Speakers: Sandeep Singh, Security Strategy & Operations, HackerOne

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=PhnlsIuGdyE

Overview

Sandeep Singh's Nullcon talk, "MLOps Under Attack: Threat Modeling Modern AI Systems," provides a crucial examination of the often-overlooked security landscape surrounding modern Machine Learning Operations (MLOps). The presentation delves beyond the surface-level concerns of Large Language Models (LLMs) to explore the entire lifecycle of AI system development, from data ingestion and model training to deployment and monitoring. Singh emphasizes that while the MLOps domain is rapidly evolving, its security posture remains in a nascent stage, presenting significant attack surfaces and vulnerabilities that traditional cybersecurity approaches may not fully address.

The talk serves as a vital resource for a diverse audience, including bug hunters seeking new research avenues, security engineers tasked with fortifying AI systems, and anyone curious about the unique challenges of securing machine learning pipelines. Singh meticulously breaks down the MLOps ecosystem, identifying key personas, processes, and technologies, and then correlates these components with potential attack vectors. The core message is clear: MLOps is essentially DevOps for machine learning, inheriting many of its predecessor's security challenges while introducing entirely new ones stemming from the complexities of data, models, and specialized platforms.

This deep dive into MLOps threat modeling is particularly timely given the pervasive integration of AI across industries. As organizations increasingly rely on AI models for critical functions, the integrity, confidentiality, and availability of these systems become paramount. Singh's presentation underscores the urgent need for proactive security measures, robust architectural designs, and continuous scrutiny of MLOps platforms and practices to prevent sophisticated attacks that could compromise data, models, and the applications built upon them.

Background

▶ Watch: Introduction to MLOps threat modeling and agenda (0:00)

To understand the security challenges in MLOps, it's essential to first grasp its foundational principles and lifecycle. Sandeep Singh defines MLOps as a unified engineering practice and cultural approach that integrates machine learning systems, mirroring the principles of DevOps but with a distinct focus on data and models rather than just software. It encompasses people (data scientists, ML developers, DevOps engineers), processes (CI/CD, deployment, design, planning), and technology (specialized platforms and tools).

The ML Life Cycle is an iterative process with several distinct phases:

1. Planning and Designing (Pre-training): Data scientists define the problem, frame objectives, collect raw data, clean it, process it, and perform feature engineering. For LLMs, this might involve selecting foundational models.
2. Training: Writing code to train models or fine-tune existing ones (e.g., LLMs).
3. Evaluation: Testing model performance against defined benchmarks and test cases.
4. Deployment (Inference): Packaging the model (e.g., containerizing) and exposing it via client-side apps or APIs. This also involves managing the backend infrastructure.
5. Management and Monitoring: Continuously monitoring model performance, looking for data drift or performance degradation, and triggering retraining processes based on predefined conditions. This closes the loop, making it an iterative cycle.

Singh highlights specific tools and platforms prevalent in each phase:

• Plan/Design: Jupyter notebooks (web IDEs for computation), Hugging Face (GitHub for AI models, datasets), Git, Pandas (ML library).
• Training: Frameworks like PyTorch, TensorFlow, and platforms such as MLflow, Kubeflow, Weights & Biases. MLflow is analogized to Jenkins for MLOps, handling training, evaluation, and deployment pipelines.
• Evaluation: Tools like Alibi, alongside MLflow and Weights & Biases.
• Deployment: Standard infrastructure tools like Docker and Kubernetes for containerization, and specialized platforms like Selden Core and KServe that offer "inference as a service."
• Monitoring: Traditional observability tools such as Splunk, Elastic, Prometheus, and Grafana, used to track model metrics, reliability, and security.

Hyperscalers also offer managed MLOps pipelines, including AWS SageMaker (with SageMaker Studio, Pipeline, Clarify), Azure AI, and Google Cloud's Vertex AI. These managed services abstract away some infrastructure complexities but introduce their own cloud-specific security considerations.

Singh then presents a practical view of the ML model development lifecycle: a Model Source (data) is used to Train the Model (involving datasets, frameworks, third-party libraries). The trained model is then deployed to a Model Hub or Model Registry (e.g., MLflow), which stores different versions (dev, prod, staging). Finally, the model is Served either by embedding it into an application (like GPT-4 in ChatGPT) or exposing it as an API endpoint, often through platforms like Selden Core or KServe.

For LLM Ops, the focus shifts slightly. Since foundational LLMs are often pre-trained by large entities, LLM Ops concentrates more on fine-tuning these models, managing them in production, and designing end-to-end applications. A typical LLM application stack involves a user query, context construction (e.g., Retrieval Augmented Generation or RAG) to fetch data, a model gateway, and crucially, input and output guardrails to redact PII or filter harmful content. Additional layers like agents (for creating actions based on output) and comprehensive logging/monitoring are also integral. The rapid adoption of LLMs has also catalyzed the broader development of custom ML pipelines for other models, expanding the general MLOps footprint.

Key Findings

▶ Watch: Overview of the MLOps life cycle stages (3:00)

Singh's talk illuminates several critical findings regarding the security posture of modern MLOps systems, emphasizing that this domain is currently in a "nascent stage" with "not much being looked at right now." This immaturity creates a fertile ground for attackers.

Firstly, the entire MLOps tooling ecosystem, from Jupyter notebooks to deployment platforms, constitutes a vast and often unscrutinized attack surface. Unlike mature DevOps tools, many MLOps platforms were developed by ML engineers and data scientists whose primary focus was functionality and speed, not inherent security. This often leads to a "super lowish" security maturity level compared to traditional software development tools.

Secondly, Singh outlines the prevalent attack scenarios across the ML lifecycle, categorizing them by phase:

• Plan/Design: Attacks targeting raw data and initial model development, such as data poisoning (injecting malicious data to degrade or bias model performance), model poisoning (injecting malicious components or altering training data to create backdoors or manipulate model behavior), and attacks on feature engineering. These often involve compromising ML personas (data scientists, ML developers) or their credentials.
• Build: Code execution vulnerabilities, often stemming from the dynamic nature of ML development environments. Supply chain attacks are a significant concern, where malicious third-party libraries or poisoned public datasets (e.g., from Hugging Face) are introduced into the training pipeline, leading to backdoors or data leakage.
• Deploy: Again, model poisoning (deploying a compromised model), model extraction (reversing an API to reconstruct the model), and container escapes in environments where models are deployed within containers.

A crucial finding is the prevalence of insecure MLOps platform configurations. Singh highlights that platforms like MLflow, Selden Core, and KServe, while powerful, have historically lacked the rigorous security scrutiny applied to other critical infrastructure. This results in vulnerabilities such as:

• Insecure container configurations: Platforms failing to isolate containers properly, leading to multi-tenancy issues and potential container escapes (e.g., Selden Core CVEs allowing access to other customer models).
• Deserialization vulnerabilities: The widespread use of formats like Pickle and Dill for serializing models is a major security risk. These formats can execute arbitrary Python code upon deserialization, making them prime targets for embedding malicious payloads. Even PyTorch files, which are essentially Pickle files wrapped in a zip archive, carry this risk.
• Insufficient isolation: Lack of network or process isolation between different stages of the ML pipeline or between different users/models on a shared platform.
• Lack of authentication: Some platforms have been found to lack basic authentication mechanisms for their deployed model endpoints, making them vulnerable to unauthorized access.

Finally, the CVE landscape for MLOps platforms is rapidly expanding. Singh notes a significant rise in reported vulnerabilities in 2023 and 2024. For instance, MLflow alone had nine critical issues reported in 2024, demonstrating the accelerating discovery of flaws in these foundational tools. This trend underscores the urgent need for enhanced security research and development in the MLOps space.

Technical Deep Dive

▶ Watch: Mapping pre-training, training, inference to MLOps stages (5:00)

The technical deep dive into MLOps security reveals how traditional attack vectors manifest in this specialized environment, alongside novel threats unique to machine learning.

Credential Compromise remains a primary initial access vector. Attackers leverage classic techniques like phishing, exploiting network file shares, or compromising developer devices to steal credentials belonging to data scientists or ML developers. Once compromised, these credentials grant access to the ML pipeline, allowing an attacker to inject poisoned data, alter model code, or manipulate configuration. The impact is amplified if misconfigurations are present, such as overly permissive IAM policies in cloud environments. Singh points out that quick setup scripts for services like AWS SageMaker often create excessively liberal default permissions, granting ML developers far more access than necessary—potentially allowing them to create or delete critical resources beyond their scope. Similarly, improperly segmented environments, where training and deployment stages are not isolated, mean a compromise in one area can easily pivot to another.

Supply Chain Attacks are particularly potent in MLOps due to the heavy reliance on external resources. Developers frequently download pre-built models, datasets, and libraries from public repositories like Hugging Face. Malicious actors can upload poisoned versions of these components. Singh highlights several real-world examples:

• Poisoned Public Datasets: Attackers embed malicious code or data within seemingly benign datasets. When downloaded and used for training, this can introduce backdoors or cause unintended model behavior.
• Malicious Libraries: Similar to traditional software supply chain attacks, compromised ML libraries can execute arbitrary code during the build process.
• Hugging Face Attacks: Research from Wiz demonstrated an "inference API attack" where malicious models were pushed to Hugging Face's inference-as-a-service offering. Due to insufficient tenancy isolation, these models could escape their containers and access other private models or underlying infrastructure. This prompted Hugging Face to implement controls like a pickle scan module and safeguards against auto-execution of Python files in datasets, though these are still evolving.

A critical vulnerability category stems from Insecure MLOps Platform Configurations, particularly serialization/deserialization issues. ML models are often saved and loaded using formats like Pickle (Python's object serialization format) and Dill. The inherent danger of Pickle is its ability to execute arbitrary Python code during deserialization. If a malicious Pickle file—or a PyTorch model file (which is essentially a Pickle file wrapped in a zip archive)—is loaded, it can lead to Remote Code Execution (RCE). Singh cites research from ReversingLabs and JFrog identifying numerous malicious AI models exploiting this exact vulnerability.

Beyond serialization, specific platforms exhibit distinct vulnerabilities:

• Selden Core: A publicly documented CVE (though not specified by number, its details were discussed) revealed that Selden Core, when deploying models, often failed to isolate containers properly. This meant that an attacker who achieved a container escape could potentially access models belonging to other Selden Core customers. Furthermore, Singh mentions a disputed CVE where Selden Core lacked built-in authentication for deployed models, which Aligo research later confirmed was being exploited in the wild, forcing users to implement their own reverse proxies for security.
• MLflow: Has been plagued by issues including path traversal, remote code execution (RCE), and OS command injection vulnerabilities. The rise in reported critical issues for MLflow (e.g., nine in 2024) underscores its ongoing security challenges.
• Kubeflow: Shows a recurring trend of SSRF (Server-Side Request Forgery) issues, which can allow attackers to perform internal network reconnaissance or access sensitive internal services.

The CVE landscape for MLOps platforms is a stark indicator of the problem. Singh notes a dramatic increase in reported vulnerabilities in 2023 and 2024. This suggests that as more security researchers turn their attention to MLOps, a backlog of critical issues in widely used platforms is being uncovered, making it a prime area for security research and bug hunting.

Demo / Proof of Concept

▶ Watch: Essential frameworks and tools for model training (7:00)

While Sandeep Singh explicitly stated, "there's no magic going to happen, we're not going to attack anything right now," his talk effectively serves as a meta-analysis of various demonstrations and proofs of concept conducted by the broader security community. He references several key research efforts that have concretely validated the MLOps attack scenarios he discusses.

One significant body of work comes from the JFrog research team. Over the past year, they have specifically targeted open-source ML platforms such as MLflow, Selden Core, and KServe, disclosing an estimated 10 to 20 CVEs across these platforms. This research provides tangible evidence of insecure platform configurations, including issues related to container isolation, deserialization, and authentication, directly demonstrating the vulnerabilities within these widely adopted tools.

Another notable contribution is from X-Force, who published an extensive white paper on lateral movement within MLOps environments following a credential compromise. Their research not only detailed how attackers could pivot internally but also open-sourced a tool called MLO Kit. This tool acts as a proof of concept, automating many of the attack steps discussed, allowing security professionals to simulate these scenarios and test the resilience of their own MLOps setups.

Singh also highlights the impactful research by Wiz from the previous year, which brought Hugging Face into the spotlight for security concerns. Wiz demonstrated how attackers could abuse issues within Hugging Face's inference-as-a-service offering by pushing malicious models. This allowed them to achieve a form of container escape, accessing other private models and showcasing the critical need for robust multi-tenancy isolation in shared ML model hubs. This research effectively demonstrated the real-world implications of supply chain attacks and insecure platform configurations.

Furthermore, Singh refers to recent discoveries by ReversingLabs and the JFrog research team (again) regarding malicious AI models being hosted on public platforms. These models exploited the inherent dangers of Pickle and PyTorch file formats, embedding arbitrary code that would execute upon download and use. These instances serve as direct proofs of concept for the serialization/deserialization vulnerabilities and the supply chain risks associated with public model repositories, even after some mitigation efforts by platform providers like Hugging Face.

In essence, while Singh did not conduct a live hack, his presentation masterfully aggregated and contextualized a collection of impactful research and tools that serve as powerful demonstrations of the MLOps security challenges, providing actionable insights for defenders and researchers alike.

Defensive Implications

▶ Watch: Standard infrastructure for MLOps model deployment (8:00)

Securing MLOps environments requires a holistic and multi-layered approach that integrates traditional security principles with ML-specific considerations. Sandeep Singh outlines key defensive strategies across data, code, infrastructure, and architectural design.

Data Security is paramount throughout the ML lifecycle. First, encrypting training data and models at rest and in transit is a baseline requirement. More critically, provenance tracking for training data is essential. This involves ensuring data integrity, preventing unauthorized modifications, and maintaining version control (e.g., using AWS S3 object lock or similar mechanisms) to enable auditing and rollback. Granular access control on training data stores is crucial, following the principle of least privilege, to prevent unauthorized access or tampering. Furthermore, organizations must proactively scan for sensitive data like PII (Personally Identifiable Information) or PHI (Protected Health Information) before data is used for training or fine-tuning. This can involve using established data loss prevention (DLP) services or building custom solutions to redact or anonymize sensitive information. Singh provides an example of an AWS Service Control Policy (SCP) that can deny broad actions like sagemaker:DeleteModel, sagemaker:CreateEndpoint, or sagemaker:UpdateEndpoint at the organizational level, preventing accidental or malicious changes to critical model components.

For Code and Model Security, standard software development practices must be adopted and adapted. This includes enforcing signed commits and rigorous code reviews for all model development code. Scanning container images for known vulnerabilities is critical, as many ML models are deployed in containers. Beyond known vulnerabilities, the fundamental architecture of how containers are deployed and interact needs scrutiny. Organizations should also scan dependencies for malicious or vulnerable ML libraries and datasets. Finally, models themselves must be tested for adversarial examples, which encompasses techniques like prompt injection in LLMs, to ensure their robustness against manipulated inputs designed to elicit unintended or malicious outputs.

Infrastructure Security in MLOps heavily emphasizes segmentation control. This is "super important" to prevent data extraction attacks and lateral movement. Different stages of the ML pipeline (development, training, staging, production) must be logically and physically segregated. If an attacker compromises a development environment, strong segmentation should prevent them from easily pivoting to production models or sensitive data stores. This involves implementing robust firewall rules and tightly controlled access policies (e.g., network ACLs, security groups, IAM policies in cloud environments). Singh illustrates this with another SCP example, demonstrating how an organization can deny sagemaker:DeletePipeline actions, protecting the integrity of the ML pipeline definition itself.

Finally, a Secure Architecture is the bedrock for MLOps security. Singh walks through a comprehensive cloud-based ML environment architecture, highlighting critical security boundaries:

1. Data Scientist/ML Developer interaction with the Dev Environment: Requires strong authentication, endpoint security, and least privilege access.
2. Between Data Sources and the ML Studio/Pipeline: Secure data ingestion pipelines, encryption, and strict network controls.
3. ML Pipeline Service to Storage (for Baseline Models): Secure storage configurations, versioning, and integrity checks.
4. Model Registry to Staging/Production: A robust model approval process, secure deployment pipelines, and separate infrastructure for each environment.
5. User/Tester Layer to Production Endpoints: API gateways with authentication, authorization, input/output guardrails, and rate limiting.

At each of these boundaries, defenders must implement appropriate controls: network segmentation (VPCs, subnets), granular IAM policies, regular security audits, and continuous monitoring. The entire system should feed into a centralized logging and monitoring solution (e.g., Splunk, Prometheus) to detect anomalies, performance drifts, and potential security incidents. By integrating these defensive measures throughout the MLOps lifecycle, organizations can significantly reduce their attack surface and build more resilient AI systems.

Key Takeaways

• MLOps Security is Nascent and Under-researched: The security maturity of MLOps platforms and practices lags significantly behind traditional software development, creating a wide-open field for vulnerabilities and attacks.
• Traditional Security Principles Apply, with ML Nuances: Core cybersecurity concepts like authentication, access control, network segmentation, and supply chain security are critical, but must be adapted to the unique aspects of data, models, and ML-specific tools.
• Serialization Vulnerabilities are a Major Threat: Model serialization formats like Pickle and PyTorch files are highly susceptible to remote code execution (RCE) due to their ability to execute arbitrary code, making them a prime target for malicious payloads on platforms like Hugging Face.
• MLOps Platforms Themselves are Vulnerable: Widely used platforms such as MLflow, Selden Core, and Kubeflow have disclosed numerous critical vulnerabilities (e.g., path traversal, container escapes, SSRF, RCE), indicating a need for rigorous security testing and hardening by vendors and users.
• Holistic Security is Essential Across the ML Lifecycle: Effective defense requires a layered approach, including data provenance tracking, granular access controls, robust infrastructure segmentation, continuous monitoring, and proactive scanning for sensitive data and adversarial examples.
• Bug Hunting in MLOps is a Promising Area: Given the immaturity of MLOps security and the rising number of disclosed CVEs, security researchers and bug hunters have a significant opportunity to identify and report vulnerabilities in these critical systems.

About the Speaker(s)

Sandeep Singh is a security professional currently working with HackerOne, where he focuses on running security strategy and operations for various customers. Prior to his role at HackerOne, Sandeep was associated with a boutique firm called Not So Secure, where he engaged in security consulting. His professional interests and expertise span a broad range of cybersecurity domains, including application security, proactive security measures, and vulnerability management. Sandeep's insights into threat modeling modern AI systems stem from his extensive experience in helping organizations build and maintain secure digital environments.