Broken links - Behind the scenes of Supply Chain breaches

Broken links - Behind the scenes of Supply Chain breaches

Speakers: François Proulx, Senior Product Security Engineer, Boost Security

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=gpqLgEqp_jA

Overview

In this comprehensive talk, François Proulx, Senior Product Security Engineer at Boost Security, dissects the rapidly escalating threat landscape of software supply chain security. The presentation provides a chilling timeline of publicly disclosed supply chain breaches, emphasizing their acceleration in recent years and the likelihood of many more undisclosed incidents. Proulx argues that as traditional application security measures improve, threat actors increasingly shift their focus to the inherently complex and often less-secured software supply chain, targeting its "weakest links."

The core of the talk leverages the Supply Chain Levels for Software Artifacts (SLSA) model, affectionately known as Salsa, as a framework to analyze various attack vectors. SLSA, which recently reached version 1.0, serves as a threat model designed to identify vulnerabilities across the typical software development lifecycle, from source control to CI components and final distribution. By examining a diverse array of real-world incidents through the SLSA lens, Proulx illustrates how attackers exploit weaknesses at every stage, from developer workstations to complex build systems and external dependencies, culminating in a discussion of his own discovered zero-day vulnerability in Terraform modules.

This article delves into the critical insights shared by Proulx, offering a detailed technical breakdown of the discussed attacks and practical defensive strategies. For organizations grappling with the complexities of securing their software supply chain, this talk provides a sobering yet actionable perspective, highlighting the pervasive nature of these threats and the necessity of a holistic security approach that extends beyond traditional application security.

Background

▶ Watch: Overview of supply chain breaches and talk's focus (0:00)

The concept of a "software supply chain" refers to the entire ecosystem of components, tools, and processes involved in developing, building, and delivering software. This includes everything from the initial source code, third-party libraries, build tools, CI/CD pipelines, and ultimately, the distribution mechanisms. Historically, security efforts primarily focused on the final application—identifying vulnerabilities within the compiled code or deployed system. However, as software development has become increasingly modular and reliant on a vast network of internal and external dependencies, the attack surface has expanded dramatically.

The problem of supply chain security has gained significant prominence, particularly after high-profile incidents like the SolarWinds attack, which brought the term into mainstream cybersecurity discourse. Proulx notes that while sophisticated threat actors, potentially nation-states, likely exploited supply chain weaknesses 10-15 years ago, these types of attacks are now becoming much more common and widely publicized. This acceleration is driven by several factors: the increasing difficulty of exploiting "low-hanging fruit" vulnerabilities in well-secured applications, the widespread adoption of open-source components, and the inherent complexity and distributed nature of modern software development.

To systematically understand and mitigate these risks, the SLSA (Salsa) framework was developed. SLSA defines a set of security requirements and controls that help ensure the integrity of software artifacts throughout the supply chain. It breaks down the software delivery process into key stages:

1. Source Control: Where code is stored and managed (e.g., Git, GitHub).
2. CI Component: The Continuous Integration system that builds and tests code (e.g., GitHub Actions, CircleCI).
3. Distribution: How the final software artifact is delivered to consumers (e.g., Docker images, package registries).

Proulx uses this model to illustrate how attackers can compromise any of these stages, often targeting the weakest link to achieve their objectives. The inherent trust placed in upstream components and the often-lax security practices around their integration create fertile ground for malicious actors to inject backdoors, exfiltrate data, or disrupt operations.

Key Findings

▶ Watch: Dropbox source code exfiltration via CircleCI phishing (3:00)

The talk systematically categorizes and details various supply chain attack vectors, demonstrating that no single stage of the software development lifecycle is immune. Proulx's analysis, framed by the SLSA model, reveals several key findings:

1. Developer Workstation Compromise: Attacks targeting developers directly, such as phishing campaigns to steal credentials for source control systems (e.g., Dropbox incident), or even malicious insiders (e.g., academic demonstration of submitting malicious Linux kernel patches). The integrity of the code at its genesis is critical.

1. Source Control System Vulnerabilities: Direct compromise of Git servers (e.g., PHP's self-hosted Git server), allowing attackers to inject malicious code under legitimate maintainer identities, or exploiting configuration weaknesses like default tag protection rules in platforms like GitHub.

1. CI/CD Pipeline Exploits: The CI component emerges as a frequent target. This includes sophisticated in-memory modification of source code during the build process (SolarWinds), command injection vulnerabilities in GitHub Actions workflows (Visual Studio Code), or altering build configurations to pull from malicious sources (Webmin). CI environments often have access to sensitive secrets, making them high-value targets.

1. Dependency-Based Attacks: Exploiting the vast ecosystem of open-source dependencies. This encompasses compromising small, unmaintained libraries that are transitive dependencies of larger, critical software (Enterprise Password Manager), or exploiting widely used legitimate libraries with hidden vulnerabilities (Log4j's remote code execution feature).

1. Package Manager and Registry Manipulation: Attacks leveraging the trust placed in public registries and package managers. This includes dependency confusion where internal package names are mirrored on public registries with malicious versions, typo squatting with slightly altered package names (e.g., requests vs. request in Python), and compromising CDNs or file servers hosting libraries (Twilio).

1. Distribution Channel Compromise: Targeting the final delivery of software. This can involve modifying Docker images on public hubs or compromising mirrors for operating system distributions. The lack of robust cryptographic verification at every step allows for the injection of malicious artifacts.

1. Second and Third-Order Attacks: Proulx emphasizes that compromises can cascade, leading to deeper and more pervasive threats. An attack on a development tool's plugin or extension (Visual Studio Code, NetBeans) can lead to developer workstation compromise, which then enables further attacks on source code, creating a worm-like propagation scenario.

1. The "Weakest Link" Principle: Across all these categories, the overarching theme is that threat actors will find and exploit the easiest path. This often involves targeting less-secured parts of the supply chain that are implicitly trusted by more robust downstream systems.

1. Terraform Modules Zero-Day: Proulx's own research highlights a critical vulnerability in Terraform modules, where the lack of cryptographic verification for these modules (unlike providers) and reliance on mutable git tags allows an attacker to alter module behavior after merge, impacting potentially hundreds of millions of infrastructure deployments.

These findings collectively underscore the multifaceted nature of supply chain threats and the imperative for organizations to adopt a comprehensive, layered security approach that addresses vulnerabilities at every stage of the software development lifecycle.

Technical Deep Dive

▶ Watch: PHP's self-hosted Git server compromise and backdoor (5:00)

Proulx provides a rapid-fire tour through numerous real-world supply chain incidents, detailing the technical mechanisms behind each attack.

The journey begins at the developer workstation. In late 2022, Dropbox disclosed a breach where attackers used a phishing campaign impersonating CircleCI to steal GitHub credentials (username, password, and one-time password) from developers. This allowed them to exfiltrate source code from approximately 100 private GitHub repositories. While Dropbox stated no further compromise occurred, the attacker now possessed valuable intellectual property. Another vector at this stage involves malicious insiders or academics demonstrating the feasibility of injecting malicious code, as seen when researchers successfully convinced Linux kernel maintainers to merge deliberately vulnerable patches.

Moving to source control, the PHP project's self-hosted Git server was compromised in 2021. Attackers injected a logic bomb into the PHP codebase, disguised as a commit from a legitimate maintainer. This backdoor would execute arbitrary code if a specially crafted HTTP user agent string was present in the request header. Although quickly detected and rolled back, it highlighted the risks of self-managed infrastructure. Similarly, the Webmin incident involved the malicious injection of remote code execution (RCE) vulnerabilities directly into the source code, which went unnoticed for over a year. Forensic analysis later revealed that the build system for Webmin was configured to pull from a malicious source control repository instead of the legitimate one, leading to the distribution of compromised software on SourceForge.

The CI component is a prime target due to its access to build environments and often sensitive secrets. The infamous SolarWinds breach involved attackers compromising the CI environment to perform in-memory modification of the Orion software's source code right before compilation. This sophisticated technique allowed the malicious code to be injected into the final binaries without leaving obvious traces in the version control system or standard build logs. Another CI-related vulnerability was found by academic researchers in the GitHub Action used to build Visual Studio Code. A command injection flaw existed in how the workflow parsed bug descriptions from GitHub issues, attempting to extract git commit hashes. An attacker could craft a malicious issue description to execute arbitrary commands within the CI environment, potentially exfiltrating secrets accessible during the build.

Dependencies represent a massive attack surface. An enterprise password manager was compromised in 2021 via a little-known, single-maintainer open-source library that was a transitive dependency. Attackers compromised this small project, injected malicious code, and when the password manager updated its dependencies (likely due to a lax version constraint), the malicious library was incorporated. The Log4j vulnerability (CVE-2021-44228), widely known as Log4Shell, exemplifies the risk of legitimate features being exploited. A specific feature in the popular Java logging library allowed for remote code execution when arbitrary user-controlled data was logged, leading to widespread exploitation and immense patching challenges due to its deep embedding in application stacks.

Package managers and public registries are also heavily targeted. Dependency confusion attacks, first publicized in 2021, exploit how package managers (like npm, PyPI) prioritize public registries over internal ones. If an organization uses an internal dependency with a generic name (e.g., my-custom-string-parser) and an attacker publishes a malicious package with the same name to a public registry, the build system might mistakenly pull the malicious public version. This requires careful configuration of package managers to prioritize internal feeds or fail if an internal package is not found. The Codecov incident in 2021 involved a compromise of their distribution mechanism. Their recommended installation method, curl | bash, lacked code signatures. Attackers modified the version of Codecov hosted on a CDN, allowing them to exfiltrate sensitive secrets visible during the build process for months. Proulx also highlights the prevalence of typo squatting, where malicious packages (e.g., request instead of requests for Python's HTTP library) are published to registries, tricking developers into downloading them. This extends to Docker Hub, with thousands of malicious Docker images using slight variations of popular names like Ubuntu.

The final stage, distribution, also faces threats. The Twilio incident involved a compromise of their S3 file server, leading to the modification of a JavaScript library served directly from a CDN. Users consuming this library via their web pages were affected for several hours. Proulx notes that Subresource Integrity (SRI), which allows embedding a cryptographic hash in the HTML <script> tag, can mitigate this by preventing browsers from loading modified scripts.

Proulx then describes second and third-order attacks, where initial compromises lead to deeper systemic vulnerabilities. Malicious plugins or extensions for IDEs like Visual Studio Code or NetBeans can be compromised. In the NetBeans case, a worm-like scenario was demonstrated where merely opening a project in NetBeans could execute malicious code, infecting other projects and propagating virally. GitHub even implemented rules to detect malicious NetBeans project files.

Finally, Proulx unveils his own research into a zero-day vulnerability affecting Terraform modules. Terraform, a popular infrastructure as code (IaC) tool, uses providers (for cloud APIs) and modules (reusable infrastructure configurations). While providers are cryptographically verified, modules are not. Terraform modules rely on git tags to specify versions, but critically, these tags can be force-pushed to point to different git commit hashes by anyone with write access to the repository, even if Branch protection rules are in place. This means a module's behavior can be altered after it's been reviewed and merged into a default branch, without any cryptographic integrity check by the Terraform registry or client. Proulx discovered over a thousand such exploitable modules, some with hundreds of millions of downloads, demonstrating the potential for widespread infrastructure compromise (e.g., silently altering firewall rules). This vulnerability is exacerbated by improper use of pull request target in GitHub Actions, which can expose secrets to untrusted pull requests if not configured carefully. Proulx responsibly disclosed this to HashiCorp, who fixed the identified vulnerable modules and provided further guidance.

Demo / Proof of Concept

▶ Watch: Webmin's unnoticed malicious code backdoor for over a year (6:50)

While a live demonstration was not explicitly performed during the talk, François Proulx presented a detailed proof of concept for a significant zero-day vulnerability he discovered in Terraform modules. This vulnerability, which he publicly disclosed for the first time in this talk, highlights a critical gap in the integrity verification of infrastructure as code (IaC) components.

Proulx explained that Terraform, a widely used tool for provisioning cloud resources, relies on two key components: providers and modules. While providers (which interface with cloud APIs like AWS, Google Cloud, Azure) are cryptographically verified by HashiCorp, modules (reusable configurations of infrastructure, often shared across teams or publicly) are not. The core of the vulnerability lies in Terraform's reliance on git tags for versioning modules. Proulx demonstrated that an attacker with write access to a module's repository could force-push a git tag to point to a different git commit hash than the one originally reviewed and merged. This effectively changes the module's underlying code without altering the tag itself, and crucially, without any cryptographic integrity check being performed by the Terraform registry or the Terraform client when the module is consumed.

To prove the exploitability at scale, Proulx conducted research that identified approximately 1,000 Terraform modules on the public registry that were susceptible to this "Pawn request" scenario (named after a GitHub blog post detailing similar GitHub Action vulnerabilities). Some of these modules, he noted, were downloaded "more than once per second," indicating their widespread adoption and the massive potential impact of such a compromise. An attacker could, for example, modify a module to secretly alter firewall configurations, exfiltrate data, or create backdoors in deployed infrastructure, even after a legitimate Terraform plan review. The plan review would be based on the initially trusted module, but the executed module, after a tag force-push, could be entirely different.

Furthermore, Proulx connected this to the misuse of GitHub Actions' pull_request_target event. This powerful feature, intended for workflows that need access to repository secrets even from forks, can be dangerously misconfigured. If not implemented with extreme care, it allows untrusted code from pull requests to run with elevated privileges, potentially exfiltrating secrets or manipulating the repository, which could then be used to perform the git tag manipulation on Terraform modules.

The speaker's team worked with HashiCorp to responsibly disclose this vulnerability. HashiCorp subsequently fixed the specific modules identified as vulnerable and provided additional guidance to harden their pipeline against future occurrences. However, Proulx emphasized that the underlying issue of unverified module integrity remains, as a GitHub issue requesting hash pinning for modules has existed since at least 2017 (and revisited in 2021) without resolution. This serves as a stark warning about the implicit trust placed in unverified components within critical infrastructure as code workflows.

Defensive Implications

▶ Watch: SolarWinds Orion software compromised through CI environment (8:10)

Securing the software supply chain is a multifaceted challenge, and Proulx's talk underscores the need for a holistic approach that goes beyond traditional application security. Defenders must shift their focus to the "application security's security," addressing vulnerabilities at every stage of the SLSA model.

1. Harden Source Control:

• Implement and enforce Branch protection rules and, critically, tag protection rules on Git platforms like GitHub. Tags, especially those used for semantic versioning, can be force-pushed by default, making them untrustworthy without explicit protection.
• Mandate multi-factor authentication (MFA) for all access to source control repositories.
• Utilize code signing for every commit and verify these signatures throughout the CI/CD pipeline.

1. Secure the CI/CD Environment:

• Treat the CI component as a hardened, trusted environment. Restrict its access to sensitive secrets only when necessary.
• Implement attestation mechanisms (as defined by SLSA) to cryptographically verify that software artifacts were built in a legitimate, expected CI environment with known configurations and dependencies.
• Generate and verify Software Bill of Materials (SBOMs) at every stage to track all components and their provenance.
• Be extremely cautious with powerful GitHub Actions features like pull_request_target to prevent command injection and secret exfiltration from untrusted pull requests.

1. Manage Dependencies Rigorously:

• Perform regular static code analysis and dynamic testing on your own code and its dependencies.
• Actively scan all dependencies for known CVEs.
• Configure package managers (e.g., npm, PyPI) to prevent dependency confusion attacks. Prioritize internal registries and implement hard fails if internal packages are not found there, rather than falling back to public registries.
• Exercise extreme caution with "curl | bash" installation methods for tools and libraries; prefer signed packages or verified sources.
• Beware of typo squatting and other malicious package variants on public registries. Implement internal policies or tools to vet new dependencies.
• For JavaScript libraries consumed from CDNs, use Subresource Integrity (SRI) to ensure the browser only loads the expected, cryptographically hashed version of the script.

1. Embrace Supply Chain Security Tools and Frameworks:

• Utilize OpenSSF Scorecard to assess the security posture of open-source projects you depend on and your own repositories.
• Leverage OpenSSF All-Star for organization-wide security posture assessment and hardening.
• Explore tools like Deciduous to exhaustively find weaknesses in GitHub configurations and identify mitigations.
• Follow guidelines from organizations like NIST, CISA, and the USDOD, which have been increasingly focused on supply chain security post-SolarWinds.
• Keep an eye on advancements like npm's recent integration of attestation for packages.

1. Address IaC and Developer Tooling:

• For Terraform modules, until official hash pinning is implemented, consider forking critical modules and hosting them in your own trusted source control to ensure their integrity. Actively advocate for the GitHub issue requesting hash pinning.
• Secure developer workstations by vetting IDE plugins and extensions and ensuring they are from trusted sources to prevent second-order and third-order attacks like the NetBeans worm.

The goal is to eliminate the "weakest links" that threat actors are actively seeking. While complete elimination of risk is impossible, a layered defense strategy focusing on verification, attestation, and strong configuration management across the entire software supply chain can significantly raise the bar for attackers.

Key Takeaways

• Supply Chain Attacks are Accelerating: As traditional application security improves, threat actors are increasingly targeting the complex and often less-secured software supply chain, making it the "weakest link" for exploitation.
• Holistic Security is Essential: Defending against supply chain threats requires a comprehensive strategy covering every stage of the software development lifecycle, from developer workstations and source control to CI/CD pipelines, dependencies, and final distribution.
• SLSA Provides a Critical Framework: The Supply Chain Levels for Software Artifacts (SLSA) model offers a structured approach to identifying vulnerabilities and implementing security controls across the software supply chain.
• Dependency Management is a Major Risk: Compromised open-source dependencies (e.g., Log4j, small unmaintained libraries), dependency confusion, and typo squatting in package registries are pervasive threats requiring rigorous vetting and configuration.
• Integrity Verification is Paramount: Relying on mutable git tags or unverified components (like Terraform modules) creates significant vulnerabilities. Cryptographic attestation, code signing, and SBOMs are crucial for ensuring the integrity and provenance of software artifacts.
• Proactive Tools and Practices are Available: Leveraging tools like OpenSSF Scorecard, All-Star, and frameworks from NIST and CIS, along with enforcing strong Branch/tag protection rules and MFA, can significantly enhance supply chain security posture.

About the Speaker(s)

François Proulx is a Senior Product Security Engineer at Boost Security. In his role, he is dedicated to helping software development teams build secure software securely. His work at Boost Security involves not only identifying vulnerabilities within code but also ensuring that the entire supply chain of the software development life cycle is robust and protected against emerging threats.