We R in a Right Pickle With All These Insecure Serialization Formats

We R in a Right Pickle With All These Insecure Serialization Formats

Speakers: Casmir Schultz, Principal Security Researcher, HiddenLayer; Tom Bonner, VPP of Research, HiddenLayer

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=yrM1ryBaIJs

Overview

In this compelling Black Hat USA talk, Casmir Schultz and Tom Bonner from HiddenLayer delve into the persistent and evolving threats posed by insecure deserialization, focusing on two widely used yet often overlooked formats: Python's Pickle and R's native serialization. While Pickle's dangers are well-documented, the speakers reveal novel techniques to bypass modern security scanners and introduce a critical, under-scrutinized vulnerability vector within the R ecosystem. The presentation highlights a concerning lack of awareness and robust defenses against these bytecode-based serialization formats, which are increasingly prevalent in machine learning, data science, and inter-process communication.

The core of the issue lies in the deserialization of untrusted data, which, when processed by virtual machines like those underlying Pickle and R, can lead to arbitrary code execution. The researchers demonstrate how attackers can craft highly evasive payloads, circumventing common detection mechanisms and exploiting the inherent trust placed in serialized objects within data-intensive workflows. This talk is crucial for anyone involved in developing, deploying, or securing applications that handle serialized data, particularly within the Python and R communities, emphasizing the urgent need for a shift from superficial checks to deeper, more comprehensive security analysis.

Background

▶ Watch: Introduction to insecure deserialization in ML, Pickle & R (0:00)

The problem of insecure deserialization is far from new, yet its prevalence and impact continue to grow, especially with the rise of machine learning (ML) and data science. Python's Pickle format, in particular, has been notorious for its security pitfalls for over a decade. Marco Laveiro's "Sour Pickles" talk 13 years prior at Black Hat USA already highlighted these dangers, but as Schultz and Bonner note, despite multiple Python versions and updates to Pickle itself, the official documentation still carries a "big red warning" about deserializing untrusted data. Developers, particularly in the fast-paced ML domain, frequently use Pickle to serialize ML models, configuration data, and even for RPC (Remote Procedure Call) and APC (Asynchronous Procedure Call) over unauthenticated channels, often overlooking the inherent risks.

The scope of Pickle-related vulnerabilities is staggering, with the speakers citing "300 or so" vulnerabilities in the last year alone. Advanced Persistent Threat (APT) groups and red team tools like Mythic, Cobra Strike, and Metasploit have integrated Pickle for payload staging, underscoring its utility for adversaries. The latest development in this ongoing cat-and-mouse game is the emergence of anti-malware scanners—both commercial and open-source—attempting to detect malicious Pickle files. However, as the research demonstrates, these scanners often rely on easily bypassed heuristics, creating a false sense of security.

Shifting focus to R, the speakers identified a critical gap in security scrutiny. R is a programming language heavily favored by the data science and statistics communities, making any vulnerability found within it highly impactful. Despite its widespread use, R's serialization format, which also leverages bytecode and a virtual machine similar to Pickle, had undergone virtually no security analysis. Prior to this research, there was "only one pre-existing R CVE," attributed to Cisco Talos, a strikingly low number for an entire programming language, indicating a fertile ground for novel discoveries. This context sets the stage for the researchers' deep dive into both formats, revealing how fundamental design choices can be leveraged for malicious ends.

Key Findings

▶ Watch: Recap of Pickle's stack-based VM and code execution (2:00)

The research by Schultz and Bonner yielded significant findings across both Python's Pickle and the R serialization format, primarily demonstrating how to achieve code execution and bypass contemporary security measures.

For Pickle, the key findings revolved around evasion techniques against anti-malware scanners and the creation of highly bespoke payloads:

1. Omission of Protocol Version (PROTO) Opcode: Scanners often look for a specific byte sequence at the header of a Pickle file (e.g., 0x80 0x04 for protocol version 4). The researchers found that omitting this PROTO opcode entirely still allows the Pickle to be deserialized and executed, but causes many scanners to fail identification. This simple modification can make a malicious Pickle appear as a generic file to naive signature-based detections.
2. Omission of Frame (FRAME) and Stop (STOP) Opcodes: The FRAME opcode, introduced in protocol version 4, is intended to improve reading efficiency by chunking bytecode blocks but is not strictly required for execution. Similarly, the STOP opcode (0x2E), which marks the end of a Pickle stream, can also be omitted. While omitting STOP will cause an EndOfFileError during loading or a ValueError during disassembling, the critical finding is that any malicious payload embedded within the Pickle will still be executed prior to these exceptions being raised, again enabling evasion.
3. Pickle-in-Pickle Embedding: A novel technique demonstrated was embedding a malicious Pickle payload inside another Pickle. Many scanners are not designed to recursively analyze nested serialization formats, allowing the inner malicious Pickle to bypass detection when loaded by the outer, ostensibly benign, Pickle.
4. Obfuscation of Callable References: Attackers can evade deny-lists (which block known dangerous functions like os.system or eval) by using alternative, less common ways to reference these built-in functions. Python's flexibility allows for multiple paths to the same callable (e.g., builtins.eval vs. __main__.__builtins__.eval), making it challenging for deny-list-based scanners to catch all permutations.
5. Opcode Remapping and Custom Payloads: The researchers developed a custom Pickle disassembler and assembler. This tool allows for the creation of "highly bespoke payloads" by handcrafting opcode sequences, defining custom opcodes, and even remapping existing opcode handlers. For instance, they demonstrated remapping the LOAD_SHORT_BIN_STRING opcode handler to another function, then using this remapped opcode to encode strings, creating payloads that are impossible to generate with standard Python pickle.dumps and thus highly evasive.

For R's serialization format, the key finding was a significant, previously unaddressed Remote Code Execution (RCE) vector related to promise objects:

1. Promise Objects and delay_assign: The delay_assign function in R can create promise objects. These objects are designed to defer the evaluation of an expression until its value is actually needed. The researchers discovered that certain R packages, such as datos, extensively use delay_assign to load and evaluate R code from external text files (eval) as soon as the R package is loaded. This means that simply importing a malicious R package can lead to immediate arbitrary code execution, even before any specific function within the package is called. This represents a potent RCE vulnerability that bypasses user interaction and is akin to the immediate execution potential of malicious Pickle files.

These findings collectively underscore the inadequacy of current security practices for deserialization and highlight new avenues for exploitation in widely used data science ecosystems.

Technical Deep Dive

▶ Watch: Evading scanners by removing Pickle protocol version opcode (3:30)

Both Python's Pickle and R's serialization formats operate on similar principles: they define a bytecode language executed by a stack-based virtual machine (VM). Understanding the mechanics of these VMs is crucial for comprehending how code execution is achieved and how evasions are crafted.

Python Pickle's Virtual Machine and RCE Primitives

The Pickle VM processes a sequence of opcodes and data, managing a stack and a "memo" (akin to registers) for storing objects. The fundamental mechanism for achieving code execution in Pickle relies on four key opcodes:

• GLOBAL and STACK_GLOBAL: These opcodes are used to load a callable object (a function or method) from a module or class onto the stack. For example, (builtins, eval) would resolve to the eval function within Python's builtins module.
• INSTANTIATE: This opcode instantiates a class, calling its __init__ method.
• REDUCE: This is the most common RCE primitive. When an object is pickled, if it defines a __reduce__ method, this method is called. The __reduce__ method typically returns a callable and a tuple of arguments. During deserialization, the Pickle VM takes the callable and its arguments from the stack and invokes the callable with those arguments.

A classic example involves builtins.eval:

1. SHORT_BIN_UNICODE loads builtins onto the stack.
2. SHORT_BIN_UNICODE loads eval onto the stack.
3. STACK_GLOBAL combines these to push the eval function object onto the stack.
4. SHORT_BIN_UNICODE loads the arbitrary code string (e.g., print('pwned')) onto the stack.
5. This code string is wrapped in a tuple.
6. REDUCE is called, taking eval and ('print(\'pwned\')',) from the stack, resulting in eval('print(\'pwned\')').

Pickle Evasion Techniques at a Bytecode Level

The researchers' evasion techniques exploit specific structural elements of the Pickle protocol:

• PROTO Opcode Omission: Pickle protocol versions (e.g., protocol 4) are indicated by the byte 0x80 followed by the version number (e.g., 0x04). Many scanners perform a header check for this sequence. By simply omitting these bytes, the Pickle file no longer matches the expected signature, causing scanners relying on this heuristic to fail. The Python pickle.load function is robust enough to infer the protocol version even without an explicit PROTO opcode, allowing the payload to execute.
• FRAME Opcode Omission: The FRAME opcode (0x5D) was introduced in protocol version 4 to facilitate more efficient loading of large Pickle streams by allowing the unpickler to read data in chunks. However, it is not mandatory for the correct parsing and execution of the bytecode. Removing this opcode further alters the file structure, potentially bypassing scanners that expect its presence as part of a valid Pickle format.
• STOP Opcode Omission: Every valid Pickle stream is supposed to terminate with a STOP opcode (0x2E). Scanners often look for this terminator at the end of the file. The researchers demonstrated that if the STOP opcode is omitted, the malicious payload still executes. While Python's pickle.load will eventually raise an EndOfFileError or a ValueError during disassembly, this exception occurs after the payload has already run, making it an effective post-execution evasion.
• Obfuscating Callables: Scanners often employ deny-lists of dangerous functions like os.system, subprocess.call, or eval. However, Python offers multiple ways to access the same callable. For instance, builtins.eval can also be referenced as __main__.__builtins__.eval or by navigating through sys.modules. Crafting payloads that use these less common references makes it harder for simple string-matching deny-lists to identify the malicious intent.
• Custom Assembler for Opcode Remapping: The most sophisticated evasion involves a custom disassembler and assembler. This tool allows researchers to:
• Handcraft arbitrary opcode sequences: Generating Pickle bytecode that standard pickle.dumps might not produce.
• Remap opcode handlers: Within the unpickler's internal state, it's possible to locate the unpickler object itself and then access its opcode handler dictionary. By finding, for example, the handler for LOAD_SHORT_BIN_STRING and remapping it to point to a different function (e.g., builtins.eval), subsequent LOAD_SHORT_BIN_STRING opcodes in the malicious Pickle would then trigger eval with the provided string, rather than simply loading it. This is an extremely powerful technique for obfuscation and subverting the deserialization process itself.

R Serialization and Promise Objects

R's serialization also uses a bytecode-based VM. The critical vulnerability uncovered revolves around promise objects and the delay_assign function.

• delay_assign: This function in R creates a "promise" to evaluate an expression only when its value is first accessed. This is commonly used for lazy loading of data or computations.
• Malicious Promise Objects: The researchers found that some R packages (e.g., the datos package) utilize delay_assign to load content from external text files and then eval (execute) that content as R code. Critically, this evaluation happens as soon as the package is loaded (e.g., via library(datos)), not when a specific function within the package is called.
• RCE Vector: If an attacker can inject malicious code into these external text files or craft a package where delay_assign points to an attacker-controlled script, simply loading the R package becomes an RCE. This bypasses user interaction and provides an immediate execution primitive, similar in impact to a malicious Pickle file being loaded. The bytecode nature of R's serialized objects means that, like Pickle, an attacker can directly embed and execute arbitrary R code if they can control the serialized input.

These technical details underscore the depth of the deserialization problem and the sophistication required to both craft and detect these advanced payloads.

Demo / Proof of Concept

▶ Watch: Advanced evasion: Nesting Pickle payloads ('pickle in a pickle') (6:20)

The talk included several compelling demonstrations showcasing the efficacy of the developed evasion techniques and the RCE in R's serialization.

For Pickle, the speakers illustrated how their crafted payloads could bypass security scanners:

1. They first showed a standard malicious Pickle that would be detected by a typical scanner.
2. Next, they demonstrated how simply omitting the PROTO opcode (the 0x80 byte followed by the protocol version) from the Pickle file's header caused the same scanner to fail in identifying it as a Pickle, yet the payload still executed when loaded by Python.
3. Further, they showed the omission of the FRAME opcode and the STOP opcode (0x2E). Even with the STOP opcode removed, the malicious code executed successfully before any EndOfFileError or ValueError was raised by the Python pickle module, confirming the "execute first, error later" behavior.
4. A more advanced demonstration involved the Pickle-in-Pickle technique, where a malicious Pickle was embedded within another. This recursive structure successfully evaded scanners that only perform a single layer of analysis.
5. The most sophisticated Pickle demonstration involved the use of their custom disassembler and assembler. This tool allowed them to handcraft Pickle bytecode, including the remapping of opcode handlers. They showed how an opcode like LOAD_SHORT_BIN_STRING could be remapped to a different, malicious function within the unpickler's context. Subsequently, using the "remapped" LOAD_SHORT_BIN_STRING in the payload would then trigger the malicious action, proving the ability to create truly bespoke and undetectable payloads that don't conform to standard pickle.dumps output.

For R, the proof of concept focused on the delay_assign vulnerability:

1. The researchers highlighted the datos R package as a real-world example.
2. They showed snippets of the datos package code that extensively used delay_assign to load and eval R code from text files upon package loading.
3. The demonstration involved modifying one of these external text files to contain arbitrary R code (e.g., system("touch /tmp/pwned")).
4. Upon simply loading the datos package using library(datos), the system command embedded in the text file was immediately executed, demonstrating an RCE without requiring any specific function call within the package. This vividly illustrated the "zero-click" nature of the vulnerability upon package import.

These demonstrations provided concrete evidence for the researchers' claims, moving beyond theoretical discussions to practical, impactful exploits.

Defensive Implications

▶ Watch: Shifting research focus from Pickle to the R language (6:50)

The findings presented by Casmir Schultz and Tom Bonner carry profound defensive implications for organizations, especially those heavily reliant on Python and R in their data science, machine learning, and general software development pipelines. The core message remains: never deserialize untrusted data. However, the nuances of the presented attacks necessitate more sophisticated defensive strategies.

For Python Pickle:

1. Strict Input Validation and Source Trust: The most fundamental defense is to strictly control the source of all serialized data. If data originates from an untrusted or unauthenticated channel, it should never be deserialized using Pickle. This applies even to internal systems where one service might implicitly trust another.
2. Beyond Signature-Based Scanning: Current anti-malware scanners relying on header checks (like PROTO bytes), file extensions, or simple deny-lists of function names are demonstrably inadequate. Defenders must push for or implement security solutions that perform deeper bytecode analysis and emulation of the Pickle VM to identify malicious instruction sequences, regardless of structural obfuscation or opcode remapping.
3. Sandboxing and Isolation: If deserialization of untrusted Pickle data is unavoidable (e.g., in a specific research or testing environment), it must occur within a highly isolated and sandboxed environment. This could involve containers, virtual machines, or dedicated processes with minimal privileges and no network access, preventing any malicious payload from impacting the host system or other services.
4. Consider Alternative Serialization Formats: For data that does not require arbitrary code execution (e.g., pure data structures, configuration), organizations should migrate away from Pickle to safer, data-centric serialization formats like JSON, Protobuf, Avro, or MessagePack. These formats are designed to carry data, not executable code, significantly reducing the attack surface.
5. Developer Education: Continuous education for developers, especially those in ML and data science, is crucial. Many developers are unaware of Pickle's inherent dangers and the ease with which it can be exploited. Best practices around secure deserialization must be integrated into training and code reviews.

For R Serialization:

1. Rigorous Package Vetting: The delay_assign vulnerability in R packages highlights the critical need for rigorous vetting of all third-party R packages. Organizations should:

• Scan for delay_assign and eval: Implement static analysis tools to identify packages that use delay_assign in conjunction with eval on external, potentially untrusted, script files.
• Source Code Review: For critical applications, manual or automated source code review of R packages is necessary to understand their behavior upon loading.
• Trusted Repositories: Rely only on trusted package repositories (e.g., CRAN with strong integrity checks) and consider hosting private, vetted package mirrors.

1. Runtime Monitoring and Anomaly Detection: Monitor R environments for unusual process activity, such as R processes spawning shell commands, making unexpected network connections, or accessing sensitive files. Anomaly detection systems can help flag suspicious behavior indicative of an RCE.
2. Isolation of R Workloads: Similar to Pickle, R environments used for data processing, analysis, or model training should be run in isolated containers or virtual machines with strict resource and network access controls. This limits the blast radius of a successful exploit.
3. Supply Chain Security: The R vulnerability underscores a broader supply chain security issue. Malicious code can be injected into widely used packages, affecting all downstream users. Implement robust supply chain security practices to ensure the integrity of all software dependencies.

In essence, defenders must evolve from reactive, signature-based approaches to proactive, deep-analysis methodologies. This includes embracing secure-by-design principles for serialization, implementing defense-in-depth with sandboxing and runtime monitoring, and fostering a culture of security awareness among developers and data scientists. The "cat and mouse" game demands a fundamental shift in how serialized data is perceived and handled.

Key Takeaways

• Insecure deserialization remains a critical and actively exploited vulnerability, particularly in Python's Pickle and R's serialization formats, with significant impact in data science and machine learning ecosystems.
• Existing security scanners are often ineffective against sophisticated Pickle payloads, which can easily evade detection through structural modifications (omitting PROTO, FRAME, STOP opcodes), obfuscation of callables, and recursive embedding (Pickle-in-Pickle).
• R's serialization format, specifically the use of delay_assign with promise objects, represents a significant and largely unaddressed RCE vector. Malicious R packages can execute arbitrary code immediately upon loading, without requiring specific function calls.
• Defenders must shift from superficial heuristic-based detection to deep bytecode analysis and emulation for serialized objects to effectively identify and mitigate advanced threats.
• The fundamental rule of never deserializing untrusted data is paramount. For situations where it's unavoidable, strict sandboxing, input validation, and robust runtime monitoring are essential.
• Organizations should consider safer, data-centric serialization alternatives (e.g., JSON, Protobuf) for data that does not require arbitrary code execution, and implement rigorous vetting processes for all third-party libraries and packages.

About the Speaker(s)

Casmir Schultz is a Principal Security Researcher at HiddenLayer. His work focuses on uncovering vulnerabilities in machine learning libraries and file formats, with a particular emphasis on the security implications of deserializing untrusted data. He demonstrated expertise in crafting bespoke payloads and analyzing bytecode-level security issues.

Tom Bonner serves as the VPP of Research at HiddenLayer. He leads research efforts into the security of machine learning systems and related technologies. Along with Casmir Schultz, he has spent several years investigating the insecure deserialization problem across various programming languages and file formats, contributing to a deeper understanding of these complex vulnerabilities.