Anatomy of a Supply Chain Worm: Building Detection Systems That Work on Zero Days

Anatomy of a Supply Chain Worm: Building Detection Systems That Work on Zero Days

Speakers: Sudhanshu Dasgupta (SafeDep), Sahil Bansal (SafeDep)

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=ITSjIuGfXkg

Overview

In the rapidly evolving landscape of cybersecurity, supply chain attacks have emerged as one of the most insidious and impactful threats. This talk, "Anatomy of a Supply Chain Worm: Building Detection Systems That Work on Zero Days," delivered by Sudhanshu Dasgupta and Sahil Bansal of SafeDep, delves into the sophisticated mechanics of such attacks, using the infamous "Shylock" worm as a primary case study. The speakers meticulously dissect how this multi-stage attack exploited software ecosystems, bypassed traditional security measures, and propagated like a digital contagion.

The core of the presentation focuses on SafeDep's innovative approach to combating these advanced threats, particularly zero-day vulnerabilities in public software registries. Dasgupta and Bansal highlight the critical limitations of conventional static analysis and signature-based detection tools when confronted with novel, obfuscated, and dynamically evolving malware. They then introduce a robust, multi-layered detection architecture that combines continuous monitoring, advanced static analysis, and a sophisticated dynamic analysis sandbox to identify malicious packages before they can inflict widespread damage.

This talk is crucial for anyone involved in software development, supply chain security, or incident response. It underscores the urgent need for proactive and adaptive detection mechanisms that can keep pace with attacker ingenuity. By dissecting a real-world supply chain worm and presenting a practical, technical solution, SafeDep provides invaluable insights into building resilient defense systems against the next generation of software supply chain threats, ensuring the integrity and security of open-source ecosystems.

Background

▶ Watch: Suspicious 12KB to 2MB package size change (1:00)

The concept of a "worm-like" infection is vividly illustrated by the speakers through the analogy of zombies: an infected entity bites another, spreading the contagion in a continuous chain. This metaphor perfectly encapsulates the propagation mechanism of sophisticated supply chain attacks, where a single compromised package can rapidly infect an entire ecosystem. The talk centers around a specific, high-impact incident: the Shylock worm, which first emerged in September of the previous year.

The Shylock attack began with an seemingly innocuous package, control-delux. Initially, this package had a modest size of around 12 KBs and a low download count (200-300 weekly downloads). However, a version upgrade dramatically increased its size to 2 MBs. This sudden, substantial change was the first red flag, indicating the injection of a new component: a post-install script that executed node bundle.js. This script was the initial vector for the Shylock worm.

The attack unfolded in a series of carefully orchestrated phases:

• Phase 1: Credential Theft. Upon installation, the malicious post-install script immediately went to work, stealing critical developer credentials. This included AWS credentials, GitHub tokens, and npm tokens. These stolen secrets were then exfiltrated to the attacker's webhook.
• Phase 2: GitHub Repository Compromise. Leveraging the stolen GitHub tokens, the attacker gained unauthorized access to all of the victim's repositories, including private ones. The attacker then injected malicious payloads into existing workflows within these repositories. A particularly damaging action was the forced migration of private repositories, making them public under the guise of "shy migration." This not only exposed sensitive code but also served as a public defacement.
• Phase 3: npm Package Infection (Worm-like Behavior). With the npm tokens acquired in Phase 1, the attacker utilized the public npm API to search for other packages maintained by the compromised user. Each of these newly identified packages was then infected using the same method, initiating the entire Phase 1-3 cycle again. This mechanism allowed the Shylock worm to spread autonomously and rapidly across the npm ecosystem, exhibiting its namesake "worm-like" behavior.

Just two months (approximately eight weeks) after the initial Shylock 1.0 attack, a more advanced variant, Shylock 2.0, emerged. This new iteration demonstrated significant evolution in stealth and impact:

• Ecosystem Expansion: Shylock 2.0 extended its reach beyond the NodeJS ecosystem to target the Bun ecosystem, demonstrating adaptability to new environments.
• Stealthier Hook: Instead of a post-install script, Shylock 2.0 utilized a pre-install script, meaning the malicious code would execute even if the npm install process was interrupted or failed, making it harder to stop.
• Increased Obfuscation: While Shylock 1.0 was obfuscated to a moderate degree, Shylock 2.0 employed highly obfuscated code, significantly increasing the difficulty of static analysis.
• Larger Payload: The payload size further ballooned from 2 MBs to a massive 9 MBs, making it even more suspicious but also harder to quickly analyze.
• Environment Variable Access: A critical new capability in Shylock 2.0 was its ability to access environment variables, potentially exposing even more sensitive information.
• File Destruction and Modification: Beyond just injecting payloads, Shylock 2.0 actively deleted resource files in compromised repositories and projects. Crucially, before deletion, it modified the package, making recovery virtually impossible.
• Workflow Machine Compromise: Instead of merely adding malicious payloads to workflows, Shylock 2.0 compromised the machines executing those workflows, granting deeper access and control.
• Wider OS Target: While Shylock 1.0 primarily targeted Linux and Mac operating systems, Shylock 2.0 extended its reach to include Windows, broadening its potential victim base.

The impact of the Shylock worm was staggering. Shylock 1.0 compromised an estimated 650 repositories. Shylock 2.0 escalated this dramatically, affecting 25,000 repositories. A public search for "shy the coming second coming" revealed over 26.3K public repositories that had been forcibly migrated, showcasing the vast scale of the compromise.

The speakers highlight why traditional security tools proved ineffective against such a zero-day attack:

• Signature-Based Scanners: Rely on pre-defined signatures of known vulnerabilities. Shylock was a zero-day, meaning no signatures existed at the time of the attack.
• Software Bill of Materials (SBOM): Primarily lists package versions and dependencies. Shylock operated by modifying existing packages with legitimate-looking patch versions, making SBOMs appear clean.
• npm audit: This tool flags vulnerabilities after they have been officially identified and assigned CVEs. By the time npm audit would flag Shylock, the malicious actions (credential theft, repo compromise) would have already occurred within seconds of installation.
• Manual Code Review: While ideal, reviewing large repositories for subtle changes in newly released versions within the critical one-to-two-second window of initial compromise is practically impossible for human developers.

This comprehensive background sets the stage for SafeDep's solution, emphasizing the urgent need for a detection system capable of identifying novel, dynamic, and rapidly propagating threats that bypass conventional defenses.

Key Findings

▶ Watch: Shylock Phase 2: Private repositories made public (3:00)

The talk reveals several pivotal findings regarding the nature of modern supply chain attacks and the requirements for effective detection:

• Sophisticated, Multi-Stage Worms are a Reality: The Shylock worm serves as a stark example of how attackers can orchestrate complex, multi-phase attacks that move laterally across various developer assets (credentials, repositories, other packages) and propagate autonomously, mimicking real-world biological worms. These attacks are no longer simple one-off compromises but rather intricate campaigns designed for widespread impact.
• Traditional Security Tools are Insufficient for Zero-Days: Signature-based detection, SBOMs, and post-disclosure auditing tools like npm audit are fundamentally reactive. They fail to detect novel, zero-day vulnerabilities and obfuscated payloads that haven't been previously identified or cataloged. Manual code review is also impractical given the speed and scale of these attacks.
• Dynamic Analysis is Critical for Uncovering Malicious Intent: Static analysis, while valuable, has inherent limitations (halting problem, Rice's theorem) in fully determining a program's runtime behavior or true intent. The Shylock case demonstrated that malicious behavior, such as downloading and executing new binaries or attempting to contact cloud instance metadata services, often only becomes apparent during actual execution in a controlled environment.
• Contextual Analysis is Paramount: Identifying a legitimate tool (like truffle-log) being downloaded or an IP address (169.254.169.254) being accessed is not enough. The key is to determine if such actions are expected or anomalous for a specific package. Combining multiple signals and applying human intelligence (human-in-the-loop) is essential to differentiate legitimate behavior from malicious misuse.
• Layered Detection Architectures are Essential: A single detection method is easily bypassed. SafeDep's findings emphasize the necessity of a multi-stage pipeline that integrates continuous monitoring, a suite of static analysis techniques (capability detection, YARA, call graphs, LLM for semantic analysis), and a robust, secure dynamic analysis sandbox. This layered approach allows for early flagging, deeper investigation, and higher accuracy.
• Infrastructure Security is a Foundational Requirement for Dynamic Analysis: Running untrusted code in a sandbox introduces its own set of risks (reverse shell, container/VM escape, kernel exploits, internal network exploitation). Building an effective dynamic analysis system requires a strong focus on isolation, multi-layer sandboxing (e.g., Docker-in-Docker), and strict egress controls to protect the analysis infrastructure itself.

These findings collectively highlight a shift in the cybersecurity paradigm, where defenders must move beyond reactive measures to embrace proactive, context-aware, and multi-faceted detection strategies to secure the software supply chain against increasingly sophisticated adversaries.

Technical Deep Dive

▶ Watch: Shylock 2.0 emerges with new attack vectors (4:40)

SafeDep's approach to detecting zero-day supply chain attacks is built upon a sophisticated, multi-stage architecture designed for continuous monitoring and in-depth analysis of public package registries. The system aims to overcome the limitations of traditional tools by combining various analytical techniques.

The overall architecture begins with a continuous monitoring system. This system actively monitors public registries such as npm, RubyGems, and other ecosystem-specific registries. Whenever a new package or a new version of an existing package is released, it triggers SafeDep's analysis pipeline.

The analysis pipeline consists of several key stages:

1. Static Code Analysis: Initial automated examination of the package without execution.
2. Dynamic Analysis: Execution of the package in a sandboxed environment to observe runtime behavior.
3. Human in the Loop: Expert review to verify suspicious findings.
4. Database Storage: Storing analysis results for future reference and threat intelligence.

Static Analysis

The static analysis stage employs multiple specialized analyzers to understand the package's potential behavior:

• Capability Detection: This analyzer identifies what the code can do. It pinpoints the APIs and libraries being used, as well as any potentially risky capabilities the package might access, such as file system operations, network requests, or credential access. Flagging these capabilities helps in identifying potential malicious intent.
• YARA Rules: These are pattern-based rules used to detect specific byte or string patterns commonly associated with known malicious behaviors or obfuscation techniques. YARA rules provide a flexible way to catch specific signatures even in novel attacks if certain patterns are consistent.
• Call Graphs: This analyzer maps the execution flow of the package, illustrating which functions call other functions. Understanding the call hierarchy helps in tracing potential malicious execution paths.
• LLM (Large Language Model) Integration: If a file is flagged by capability detection or YARA rules as suspicious, an LLM is used to perform a semantic analysis of the entire file's source code. The LLM attempts to understand the actual intent and functionality of the flagged code, aiding in the identification of subtle malicious logic that might escape simpler pattern matching.
• Metadata Analysis: This involves examining various metadata points of the package, such as its popularity (stars), OpenSSF scorecard rating (a measure of software security best practices), existence in the OSV.dev ML database (Open Source Vulnerabilities), and critically, the presence of pre-install or post-install scripts. As demonstrated by Shylock 1.0 and 2.0, these scripts are common vectors for initial compromise.

Despite its comprehensive nature, static analysis has inherent limitations:

• Halting Problem: It's impossible to definitively determine if a program will ever halt or run indefinitely (e.g., infinite loops, recursive calls) without executing it.
• Rice's Theorem (Generalized Halting Problem): This states that no non-trivial property about the language recognized by a Turing machine is decidable. In practical terms, it means no tool can fully and accurately verify a program's output or its complete intent for all possible inputs without execution.
• Context Factor: Static analysis often struggles with context. A piece of code might appear suspicious but be benign in its intended use, leading to a high false positive rate. SafeDep accepts this, preferring false positives over missing actual threats (false negatives).
• Runtime Behavior: The true behavior of a package, especially one designed to be dynamic or reactive, can only be fully understood during execution.

Dynamic Analysis

To overcome static analysis limitations, SafeDep developed a robust dynamic analysis infrastructure. The primary goals were:

1. Running open-source packages in a secure, sandboxed environment.
2. Observing the package's runtime behavior (what it executes, what it accesses).
3. Correlating dynamic results with static analysis to classify packages as malicious.

The challenges in building this system included:

• Scale vs. Accuracy Trade-off: Analyzing millions of diverse packages (CLIs, libraries) requires balancing the depth of analysis (accuracy) with the number of packages processed (scale). Deep analysis takes time, but shallow analysis might miss threats.
• Google Package Analysis Limitations: Existing solutions like Google Package Analysis were too deeply coupled with Google's internal systems and did not allow for the necessary correlation with SafeDep's static analysis results.

SafeDep's dynamic analysis flow involves an executor component responsible for creating and managing sandbox environments. For runtime monitoring, they utilize Falco, an open-source tool that leverages eBPF (extended Berkeley Packet Filter) to monitor system calls and kernel events.

The sandbox implementation is crucial for security:

• Docker-in-Docker: This multi-layer sandboxing approach provides an additional layer of isolation, making it significantly harder for a malicious package to escape the container.
• NAT Service: Continuously checks for new packages to be analyzed and hands them off to the executor.

During runtime monitoring, SafeDep mimics the installation behavior (e.g., npm install) and observes all system calls and events. Falco captures these events, which are then passed via Unix sockets to an event handler (a gRPC client). This handler correlates the dynamic events with the initial static analysis data and stores the findings in a database.

Falco detection rules are central to identifying malicious behavior. Examples include:

• Drop and execute a new binary in container: This rule triggers if a package downloads and executes a binary that was not part of the base image. For Shylock, this detected the download and execution of the truffle-log binary. While truffle-log is a legitimate secret scanning tool, its execution by an unexpected package in an unusual context is a strong indicator of compromise.
• Contact EC2 instance metadata service from container: This rule flags attempts to access the specific IP address 169.254.169.254, which is the well-known endpoint for AWS EC2 instance metadata. Accessing this service can lead to the theft of AWS credentials, role permissions, and other sensitive cloud configuration data.

Infrastructure Threats and Mitigations

Running untrusted code presents significant security risks to the analysis infrastructure itself. SafeDep has implemented several mitigations:

• Reverse Shell Access: Attackers might try to establish an outbound connection to their C2 server. Mitigation: Isolated environments with no outbound connections allowed to external IPs.
• Container Escape: Malicious packages might attempt to break out of the container. Mitigation: Multi-layer sandbox using Docker-in-Docker.
• Kernel Exploit: Vulnerabilities in the kernel could be exploited. Mitigation: Dedicated nodes for segregating untrusted code, ensuring internal services are not affected even if a kernel exploit occurs in the sandbox.
• Internal Network Exploitation: Packages might attempt to access SafeDep's internal network (databases, other services). Mitigation: Strict egress restrictions that explicitly define allowed IP addresses and ports, blocking unauthorized internal connections.
• VM Escape: The package might escape the virtual machine itself and gain access to the hypervisor. Mitigation: This is accepted as an infrastructure risk but is mitigated by the cumulative effect of previous isolation and multi-layer security measures, making it exceedingly difficult to achieve.

SafeDep's dynamic analysis system is highly performant, analyzing approximately 35,000 packages daily and processing over 160-180 million events to date. This continuous monitoring, often configured for a 5-second polling interval, ensures that new and upgraded packages across various registries are swiftly put through the rigorous static and dynamic analysis pipeline, providing near real-time threat detection.

Demo / Proof of Concept

▶ Watch: Shylock 2.0: File destruction and Windows compromise (5:50)

While the talk did not feature a live, interactive demonstration by the speakers, it effectively showcased the capabilities of SafeDep's detection system by detailing how it successfully identified the Shylock worm, specifically using the control-delux package as a prime example. This served as a powerful "proof of concept" for their architecture.

The speakers explained that the control-delux package, despite its initial obfuscation, ultimately bypassed the static analysis stage. This highlights a critical limitation of static methods when faced with sophisticated, dynamically behaving malware. However, it was the dynamic analysis component, particularly the Falco-based runtime monitoring, that conclusively flagged control-delux as malicious.

Two key events observed during the sandboxed execution of control-delux demonstrated its malicious intent:

1. "Drop and execute a new binary in container": This Falco rule was triggered because the post-install script within control-delux was observed downloading and then executing a new binary. The binary in question was identified as truffle-log. As the speakers clarified, truffle-log is a legitimate tool for secret scanning. However, the critical factor for detection was the context: a simple package like control-delux had no expected reason to download and execute such a binary during its installation, making this behavior highly anomalous and suspicious.
2. "Contact EC2 instance metadata service from container": Another crucial Falco rule was triggered when the package attempted to access the IP address 169.254.169.254. This IP is the well-known, non-routable address for the AWS EC2 instance metadata service. Accessing this service from within a container running on an EC2 instance is a common technique for attackers to steal AWS credentials, instance profiles, and other sensitive cloud configuration data. Again, for a package like control-delux, attempting to contact this service was completely unexpected and indicative of malicious intent.

These two dynamic events, when combined, presented an undeniable case for control-delux being malware. The human researcher, acting as the "human in the loop," reviewed these specific events and confirmed that such behaviors were entirely inconsistent with the legitimate functionality of the package. This verification led to the package being officially flagged as malware and subsequently published to the OSV.dev database, effectively demonstrating SafeDep's ability to detect zero-day supply chain threats that evade traditional static analysis.

Defensive Implications

▶ Watch: Why traditional security tools failed to detect Shylock (7:40)

The insights gleaned from the Shylock worm and SafeDep's detection architecture offer critical defensive implications for organizations and developers:

1. Do Not Solely Rely on Traditional Scanners: The Shylock attack unequivocally demonstrates that signature-based tools, SBOMs, and post-disclosure vulnerability scanners (npm audit) are insufficient for detecting zero-day supply chain attacks. Defenders must recognize their limitations and invest in more proactive, real-time detection capabilities.
2. Implement Continuous Monitoring of Registries: Organizations should adopt continuous monitoring of their direct and transitive dependencies from public package registries (npm, PyPI, RubyGems, etc.). This means actively scanning for new package releases, version upgrades, and any metadata changes that could indicate compromise, just as SafeDep's system does.
3. Adopt a Multi-Layered Detection Strategy: A single point of failure in detection can be catastrophic. Implement a layered approach that combines sophisticated static analysis (capability detection, YARA rules, call graphs, LLM-driven semantic analysis) with robust dynamic analysis in secure sandboxes. Each layer provides a different perspective on potential threats, increasing the overall detection probability.
4. Prioritize Dynamic Analysis for Suspicious Packages: Any package exhibiting suspicious characteristics during static analysis (e.g., unusual script presence, high obfuscation, sudden size changes) should be automatically escalated for dynamic analysis. This ensures that runtime behaviors, which are often the true indicators of malicious intent, are thoroughly investigated.
5. Scrutinize Install Scripts and Runtime Behavior: Pay extreme attention to pre-install and post-install scripts, as these are primary vectors for initial compromise and credential theft. Monitor for anomalous runtime behaviors such as:

• Downloading and executing new binaries (even legitimate ones like truffle-log if out of context).
• Attempts to access cloud instance metadata services (169.254.169.254).
• Unexpected network connections (especially outbound).
• File system modifications outside of expected installation paths.
• Accessing environment variables.

1. Implement Strict Egress Filtering in Build Environments: CI/CD pipelines and build environments are prime targets. Enforce stringent egress restrictions that only allow connections to explicitly whitelisted, known-good domains and IP addresses. This mitigates the risk of reverse shells and credential exfiltration to attacker-controlled webhooks.
2. Isolate Build and Development Environments: Use multi-layered sandboxing (e.g., Docker-in-Docker, dedicated untrusted nodes) for running untrusted code or building projects with external dependencies. This limits the blast radius in case of a compromise, preventing container/VM escapes or internal network exploitation.
3. Context is Key for Threat Intelligence: Understand that the presence of certain tools or actions isn't inherently malicious. It's the context in which they occur that determines their intent. A package scanner downloading truffle-log might be legitimate; a UI library doing so is highly suspicious. Build internal threat intelligence that understands expected behaviors for different package types.
4. Prepare for Ecosystem-Specific Attacks: Attackers are adapting to new ecosystems (Bun, NodeJS, Windows, Mac, Linux). Defenders must ensure their detection strategies are broad enough to cover all relevant development environments and technologies used within their organization.

By integrating these defensive implications into their security posture, organizations can significantly enhance their resilience against sophisticated, zero-day supply chain attacks and better protect their software development lifecycle.

Key Takeaways

• Supply chain attacks are evolving into sophisticated, worm-like behaviors that rapidly propagate across ecosystems by leveraging stolen credentials and exploiting common package mechanisms like install scripts and version updates.
• Traditional signature-based and static analysis tools are insufficient for detecting novel, zero-day supply chain attacks, as they are reactive and cannot anticipate unknown threats or fully analyze dynamic runtime behavior.
• Dynamic analysis in a secure, multi-layered sandbox is critical for uncovering malicious intent, especially when monitoring system calls (e.g., via Falco) for anomalous activities like downloading new binaries or attempting to access cloud instance metadata services (169.254.169.254).
• A multi-stage detection pipeline combining diverse static analysis techniques (capability detection, YARA, call graphs, LLM for semantic analysis) with robust dynamic analysis and human expert review is essential for high-accuracy, proactive threat identification.
• Strong infrastructure security, including strict egress restrictions and multi-layer isolation (Docker-in-Docker), is paramount when building dynamic analysis systems to prevent the malicious packages from compromising the analysis environment itself.
• Defenders must shift to proactive, continuous monitoring of public registries and adopt layered security approaches that scrutinize runtime behavior, as relying solely on post-facto audits or simple dependency scanning is no longer adequate against modern supply chain threats.

About the Speaker(s)

Sudhanshu Dasgupta is associated with SafeDep, a company focused on securing the software supply chain. He presented the initial context and impact of the Shylock worm, highlighting the failures of traditional detection methods.

Sahil Bansal is also associated with SafeDep. He delved into the technical architecture of SafeDep's detection system, detailing the static and dynamic analysis components, the challenges faced, and the security measures implemented to protect the analysis infrastructure.