XZ Backdoor: Navigating the Complexities of Supply Chain Attacks Detected by Accident

XZ Backdoor: Navigating the Complexities of Supply Chain Attacks Detected by Accident

Speakers: DevSecYoad (CEO and co-founder, New York Security)

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=N4Mxu2hJcwA

Overview

In an era increasingly reliant on open-source software, the talk "XZ Backdoor: Navigating the Complexities of Supply Chain Attacks Detected by Accident" delivered by DevSecYoad, CEO and co-founder of New York Security, provided a sobering look at the sophisticated and stealthy nature of modern supply chain attacks. This presentation delved into the critical distinction between traditional software vulnerabilities and deliberate, malicious injections, using the high-profile XZ Utils backdoor as a central case study. DevSecYoad, drawing from his extensive background in DevOps and DevSecOps at Microsoft, highlighted how these advanced threats exploit fundamental trust relationships within the software development lifecycle, often evading conventional security tools.

The core message of the talk resonated with the growing concern that the open-source ecosystem, while a powerful engine of innovation, also presents a massive attack surface due to its decentralized nature and often volunteer-driven security efforts. The XZ backdoor, famously discovered by accident, underscored the inherent difficulty in detecting deeply embedded malicious code when the source code itself appears legitimate. This article will explore the intricate technical details of various supply chain attack vectors discussed, the challenges they pose to defenders, and the proactive measures organizations can adopt to bolster their defenses against such insidious threats.

The talk is particularly relevant for security professionals, developers, and decision-makers grappling with the escalating risks of software supply chain compromise. It emphasizes that relying solely on traditional vulnerability scanning tools is insufficient for identifying deliberate attacks, which are designed to bypass such mechanisms. By dissecting real-world examples like the XZ backdoor, malicious VS Code extensions, and the 3CX incident, DevSecYoad offered invaluable insights into the attacker's mindset and the sophisticated techniques employed to achieve widespread compromise, urging a fundamental shift in how we approach software security.

Background

▶ Watch: Speaker Introduction and Company Genesis (0:00)

The proliferation of supply chain attacks in recent years is largely attributable to two dominant factors: the pervasive adoption of open-source software and the immense incentive for attackers. As DevSecYoad pointed out, the vast majority of modern codebases incorporate open-source components, often nested many layers deep. Unlike commercial enterprise software, open-source projects frequently lack dedicated security teams or robust security budgets, making it unrealistic to expect them to maintain the same level of security rigor. This disparity creates a significant vulnerability, as a compromise in a widely used, yet under-resourced, open-source package can ripple through hundreds or thousands of downstream companies, offering attackers an unparalleled return on investment compared to targeting a single organization.

A crucial distinction highlighted in the talk is between a vulnerability and a supply chain attack. A vulnerability, such as the infamous Log4j zero-day, is typically an unintentional flaw or bug in code that is later discovered and assigned a CVE (Common Vulnerabilities and Exposures) identifier. While critical and requiring rapid remediation, it stems from a mistake. In contrast, a supply chain attack is a deliberate, malicious act where an attacker intentionally injects harmful code or manipulates the software delivery process. Such attacks often go undetected by traditional security tools like SCA (Software Composition Analysis) or SAST (Static Application Security Testing) because the malicious intent is obscured, and the code might not present as a recognized vulnerability. The XZ backdoor, for instance, had no CVE upon its initial deployment because it was a deliberate act of sabotage, not an accidental flaw.

The talk categorized supply chain attack vectors across the software development lifecycle:

• Developer Section: While internal intruders or developers submitting vulnerable code (identifiable by SAST) are risks, a true supply chain attack here involves a developer deliberately submitting malicious code, often in binary form, directly into a repository.
• Compromise of Source Control: Less common for modern systems like Git due to their visibility and security features, but still a potential vector for stealthy attacks that manipulate history or inject payloads.
• Build Platform: A highly impactful target, as demonstrated by SolarWinds and 3CX. Compromising the build environment allows attackers to inject malicious code into legitimate software artifacts after the source code has been verified, bypassing CI/CD checks.
• Compromise of Repository/Package Manager: Gaining access to an open-source maintainer's credentials allows direct upload of malicious packages to public repositories (e.g., npm, PyPI, Maven), instantly distributing malware to countless users.
• Compromise of Distribution Server: Attacking CDNs or command servers that distribute software updates or agents (like the Job Cloud attack) can deliver malicious payloads directly to end-user systems, often before they even enter the development lifecycle of the recipient.
• Using Bad Packages: The cumulative effect of developers choosing popular but potentially insecure open-source packages, creating a chain of dependencies where the weakest link can compromise the entire software stack. Developers, under pressure to deliver, often prioritize functionality and popularity over rigorous security vetting.

These vectors illustrate a sophisticated threat landscape where attackers are no longer just looking for accidental bugs but are actively seeking to subvert the trusted processes and components that underpin modern software development.

Key Findings

▶ Watch: Vulnerability vs. Deliberate Supply Chain Attack (3:00)

The talk presented several critical findings, primarily revolving around the increasing sophistication and stealth of supply chain attacks, and their ability to bypass conventional security measures. The overarching theme was that traditional vulnerability scanning tools are inadequate for detecting deliberate malicious injections, which are designed to look legitimate or operate outside the scope of typical vulnerability definitions.

1. Deliberate Attacks vs. Vulnerabilities: A fundamental finding is the stark difference between accidental software flaws (vulnerabilities) and intentional supply chain attacks. The latter, by design, are not easily found by tools looking for known CVEs or common programming errors. This means a package can be deemed "vulnerability-free" by SCA tools yet contain highly malicious code.
2. Exploiting Trust and Process: Attackers are adept at exploiting trust relationships (e.g., maintainer trust, signed executables) and inherent process blind spots (e.g., difficulty inspecting binaries, automatic updates). The XZ backdoor exemplified the slow, deliberate cultivation of trust before injecting malicious binaries.
3. Post-Source Code Injection: A recurring pattern across multiple attack vectors (XZ, 3CX, GitHub comment malware) is the injection of malicious code after the source code has been committed or validated. This occurs either during the build process (by manipulating build scripts or injecting binaries) or by sideloading malicious components with legitimate, signed executables. This technique ensures the "legitimate" source code remains clean, making detection extremely challenging.
4. Evasion of EDR and Antivirus: Many sophisticated attacks, including the GitHub comment malware and the 3CX incident, actively employ techniques like bytecode injection, runtime compilation, and DLL sideloading to evade Endpoint Detection and Response (EDR) systems and antivirus software. They leverage the trust placed in signed, legitimate executables to execute their malicious payloads.
5. The Human Factor and Open Source Challenges: The talk highlighted the human element as both a weakness and a strength. Developer education is crucial but insufficient. The inherent challenges of open-source maintenance—limited resources, burnout, and the difficulty of vetting new contributors—create fertile ground for attackers to gain control, as seen with the XZ backdoor.
6. Lack of Granular Permissions in IDE Extensions: The VS Code extension attack revealed a significant architectural flaw: extensions operate with the same permission level as the IDE itself, lacking the granular permission models seen in mobile operating systems. This allows a seemingly innocuous theme extension to access sensitive file system data.

These findings collectively paint a picture of a supply chain threat landscape that demands a more proactive, holistic, and skeptical approach to software security, moving beyond reactive vulnerability management.

Technical Deep Dive

▶ Watch: SDLC Attack Vectors: Developer to Distribution (4:30)

The technical deep dive in DevSecYoad's talk covered three distinct, yet thematically linked, supply chain attack vectors: malicious VS Code extensions, GitHub comment malware, and the highly sophisticated XZ backdoor, complemented by the 3CX incident. Each demonstrated advanced techniques for code injection and evasion.

VS Code Extension Attacks

DevSecYoad began with an accessible yet impactful vector: malicious VS Code extensions. VS Code, being a highly popular IDE (70% user base in 2022), is a prime target due to its extensibility and the sensitive information developers store within it (code, environment variables, deployment credentials). The attack highlighted involved researchers creating a malicious extension named Darkula to mimic the popular Dracula theme.

The ease of becoming a "verified" extension on the VS Code marketplace was a critical vulnerability. Attackers only needed to register a domain name, and they could link any source code repository, even a legitimate one, while the actual extension package contained malicious code. Once a developer downloaded Darkula, the extension would immediately exfiltrate their source code to a Command and Control (C2) center. To boost its popularity, the attackers used a script to repeatedly install the extension, artificially inflating its download count and making it appear as a trending, trustworthy option. This allowed them to compromise hundreds of companies, including large IPO firms.

The core technical issues enabling this attack were:

1. Shared Permission Context: VS Code extensions execute with the same permission level as the IDE itself. An EDR system would trust the signed Microsoft IDE, and thus implicitly trust any extension running within its context, regardless of its true intent.
2. Lack of Granular Permissions: Unlike mobile operating systems, VS Code extensions lack a granular permission model. A theme extension, which should only modify UI aesthetics, could access the file system, demonstrating a significant design flaw.
3. Automatic Updates: Extensions can automatically update in the background. An initially benign extension could later become malicious through an update without the user's knowledge or consent.

GitHub Comment Malware

This section detailed a subtle technique for distributing malware through legitimate platforms. On GitHub (and previously GitLab), users could upload files to comments, even without publishing the comment, and receive an official Microsoft/GitHub URL for the uploaded content. This allowed attackers to host malware on a highly trusted domain.

The specific attack described involved uploading malware that appeared as a readme.txt file, but actually contained bytecode. This bytecode was not detected by EDR or antivirus tools as it wasn't a traditional executable. The attackers would then distribute this link on platforms like YouTube and gaming forums, manipulating users into downloading a "setup" file. This setup file, which was legitimately signed and trusted, would then sideload and compile the bytecode at runtime. This technique cleverly bypassed static analysis and EDR systems by executing the malicious payload only after being initiated by a trusted process. Microsoft later patched this vulnerability by changing the URL for uploaded files in comments from a repository URL to a user-specific URL, reducing the perceived trustworthiness.

The XZ Backdoor

The XZ backdoor was presented as the epitome of sophisticated supply chain attacks. It targeted XZ Utils, specifically the liblzma compression library, which is widely used across Linux distributions and other software. The discovery was accidental: a PostgreSQL developer noticed unusually long (8-second) CPU cycles during SSH login, which eventually led to the uncovering of the backdoor.

The attack unfolded over a long period, involving an attacker (known as Jia Tan) slowly gaining the trust of the original maintainer of XZ Utils. Jia Tan contributed fixes and new features, eventually leading the original, overwhelmed maintainer to hand over control of the repository. Once in control, the attacker introduced two binary files, bad-3-check.ll and good-land.ll, into the repository. These binaries, being opaque to standard code review, were designed to be extracted during the build process.

During the compilation of liblzma, these binary files would inject malicious code into the project's Makefile. This manipulation ensured that the final compiled artifact, liblzma, contained the backdoor, while the visible source code remained legitimate. The injected code was designed to intercept SSH authentication, allowing unauthorized remote access. This sophisticated technique affected several major Linux distributions, including Fedora, Ubuntu, and Red Hat, which had merged the malicious package before its detection.

3CX Double Supply Chain Attack

Another complex example was the 3CX supply chain attack from 2023, a VoIP software used by hundreds of thousands of companies. This was a "double supply chain attack" with the ultimate goal of compromising the build environment.

The initial vector was an employee downloading a discontinued trading software onto their machine. This software opened a reverse proxy, providing the attacker with initial access to the organization's internal network. This initial malware was a signed executable that performed DLL sideloading, injecting malicious DLLs that ran in the same context as the trusted executable, again evading EDR.

From there, the attacker traversed the network to compromise the build servers used by 3CX. Critically, these were persistent servers (not ephemeral containers), allowing the attacker to establish persistency by injecting shellcode or malware. During subsequent legitimate builds of the 3CX software, the attacker's persistent malware injected its own malicious code into the final 3CX binaries. As with XZ, the source code remained legitimate, but the distributed software was compromised.

The payload of the 3CX malware was designed for stealth: it waited 1 to 4 weeks before executing. It would then download an "icon file" (which contained the actual malicious code) and execute it to harvest credentials from browsers and the registry, sending them to a C2 server. A significant defensive challenge arose when EDR systems (like SentinelOne or CrowdStrike) began flagging the malicious 3CX software; 3CX support initially dismissed these as false positives, delaying effective response.

These examples collectively demonstrate a clear trend: attackers are moving beyond simple vulnerabilities to target the very processes and trusted components that deliver software, making detection and defense increasingly complex.

Demo / Proof of Concept

▶ Watch: The 'Weakest Link' in Open Source Supply Chains (7:00)

DevSecYoad provided a compelling proof of concept to illustrate the stealth and potency of such supply chain attacks, specifically mimicking the XZ backdoor's binary injection technique in a different ecosystem. The team forked a Java implementation of the XE library (a Java equivalent of XZ Utils for compression/decompression) and injected malicious code into it. This modified package was then uploaded to Maven, the widely used Java package manager, and remained undetected since May (with the last iteration in August 2023). Crucially, the Maven repository showed the package as having "zero vulnerabilities," highlighting the inadequacy of traditional SCA tools against deliberate attacks.

The demo setup involved a vulnerable application called "them vulnerable scar" which included this malicious Maven package in its pom.xml. An API within this application was designed to receive a file, zip it, and then store it. The key was that whenever the malicious XE library's compression function was invoked by this API, the injected malicious code would also execute.

During the live demonstration, DevSecYoad:

1. Started the Dockerized application containing the compromised package.
2. Showed the /tmp directory within the container, initially empty.
3. Invoked the API to zip a file, which triggered the execution of the malicious library.
4. Demonstrated that the malicious code had created a new file, eval, in the container's /tmp directory.
5. Established a reverse tunnel using telnet from his machine to the compromised container.
6. Wrote a command into the eval file via the telnet session.
7. Upon closing the telnet session, the eval file was automatically executed, running the attacker's command. In the demo, a benign command was used to avoid actual harm, but in a real scenario, this could be any arbitrary command.

This demo powerfully illustrated several points:

• Stealth: The malicious package was on Maven for months, undetected by SCA tools.
• Binary Injection: The source code shown in the repository could be legitimate, while the uploaded binary package contained the malicious payload, a direct parallel to the XZ attack.
• Trigger-based Execution: The malicious code lay dormant until a specific function (compression) of the legitimate library was called, making it harder to detect through passive monitoring.
• Remote Code Execution (RCE): The ability to establish a reverse tunnel and execute arbitrary commands on the compromised system demonstrated the severe impact of such an attack.

The demo underscored the talk's central thesis: the need to look beyond traditional vulnerability scanning and understand the intricate ways attackers can subvert the software supply chain.

Defensive Implications

▶ Watch: VS Code as an Attack Vector: Extensibility Risks (8:00)

DevSecYoad provided several critical defensive implications, emphasizing a shift from reactive vulnerability management to proactive supply chain security:

1. Educate Developers (with caveats): While developer education about vetting extensions or packages is important, it's not a complete solution due to human factors and organizational churn. The human element is often the weakest link.
2. Block VS Code Network Access / Use Extension Scanning Tools: For VS Code, restricting network access for the IDE (and thus extensions) can limit C2 communication, though attackers can still find ways to exfiltrate data. More effectively, tools that analyze extension behavior (like the malicious.io tool mentioned) can perform heuristics (e.g., "why does a theme extension need file system access?") to detect malicious intent.
3. Disposable Build Servers: A paramount defense against build platform compromises (like 3CX) is to use disposable build servers (e.g., ephemeral containers or Docker images). After each build, the server instance should be destroyed, preventing attackers from gaining persistence and injecting code into subsequent builds. Persistent Jenkins or other CI/CD servers are significant attack surfaces.
4. "Signed Doesn't Mean Trusted": Organizations must understand that a signed executable, even from a reputable vendor, does not inherently guarantee trustworthiness. Attackers can leverage signed binaries to sideload malicious DLLs or inject code, as seen in the 3CX attack and a Windows vulnerability existing since 2013 that allows manipulation of certificate signatures without being flagged as malicious. EDR systems often implicitly trust signed binaries, creating a blind spot.
5. Attestation for Software Artifacts: This is a promising, albeit complex, solution. Attestation involves cryptographically signing every stage of the software pipeline—from source code to CI/CD processes to the final package. This creates an auditable chain of trust, guaranteeing that the final package is identical to what was produced from the source code and build steps. While it offers strong integrity guarantees, adoption is currently low due to its implementation complexity.
6. Scan for Binaries in Open Source Repositories: A strong heuristic for detecting suspicious activity in open-source projects is to question the presence of binary files in source code repositories. Unless there's a highly justified reason, open-source projects primarily consist of source code. The sudden introduction of opaque binaries, as in the XZ backdoor, should be a major red flag.
7. SCA Tools Are Insufficient for Deliberate Attacks: The demo explicitly showed a malicious Maven package with "zero vulnerabilities" according to SCA. This underscores that traditional SCA tools, designed to find known vulnerabilities, are ill-equipped to detect deliberate malicious code injections that don't manifest as typical bugs or CVEs. Organizations need to move beyond sole reliance on SCA for supply chain security.
8. Threat Modeling Beyond Vulnerabilities: Organizations must broaden their threat modeling to consider risks beyond just accidental vulnerabilities. This means evaluating the entire software supply chain for potential points of deliberate compromise, including developer accounts, build environments, and package distribution.
9. Reproducible Builds (Challenging): The concept of reproducible builds involves building the same application twice on different machines and comparing their checksums. If the checksums match, it suggests integrity. However, this is incredibly difficult to achieve in practice due to minor environmental differences (e.g., server version, build date) that can alter checksums. While a large project is attempting to make it more accessible, it remains a significant technical challenge.
10. Monitor Activity (with caution): Monitoring for unusual network calls to unknown remote servers is a potential defensive measure. However, DevSecYoad cautioned that this approach often leads to a high volume of false positives, potentially overwhelming security operations centers (SOCs).

In essence, defending against these sophisticated supply chain attacks requires a multi-layered approach that combines architectural changes (disposable build servers), process improvements (attestation), deeper scrutiny of open-source components (binary scanning), and a fundamental re-evaluation of trust assumptions within the software delivery pipeline.

Key Takeaways

• Supply chain attacks are deliberate and bypass traditional vulnerability scanning: Unlike accidental vulnerabilities, sophisticated supply chain attacks are intentionally designed to inject malicious code, often in binary form or post-source, making them undetectable by standard SCA or SAST tools.
• Trust relationships are actively exploited: Attackers leverage trust in open-source maintainers, signed executables, and popular platforms (like VS Code marketplace) to achieve stealthy and widespread compromise.
• Ephemeral build environments are critical: Implementing disposable, containerized build servers is a fundamental defense against persistent build system compromises, preventing attackers from maintaining access and injecting code into subsequent software releases.
• "Signed" does not equal "trusted": The presence of a digital signature on an executable does not guarantee its benign nature; attackers can use signed binaries to sideload malicious DLLs or inject code, bypassing EDR and antivirus systems.
• Beyond vulnerabilities: holistic threat modeling is essential: Organizations must expand their security focus beyond just finding CVEs to actively threat model the entire software supply chain, considering deliberate attack vectors and the integrity of every component and process.
• Attestation offers strong integrity but faces adoption challenges: Cryptographically signing each step of the software development and delivery pipeline (attestation) provides a robust mechanism to guarantee artifact integrity, but its complexity currently limits widespread implementation.

About the Speaker(s)

DevSecYoad is the CEO and co-founder of New York Security. Prior to founding his company, he spent several years in the technology sector, focusing extensively on DevOps and DevSecOps roles across both corporate environments and startups. Notably, he led a group of DevOps and DevSecOps engineers within Microsoft's cloud-optimized security group, where he gained valuable experience in incident response. His direct involvement in responding to a significant security incident at Microsoft served as the catalyst for co-founding New York Security, driven by a desire to proactively protect organizations from similar threats. Beyond his professional pursuits, DevSecYoad is an avid musician, identifying himself as a drummer, and also expresses a unique fondness for gloves.