Securing the chains: Building defensive layers for software supply chains

Securing the chains: Building defensive layers for software supply chains

Speakers: Yadu Krishna, Security Engineer, GED

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=Olh37YigGgM

Overview

In an era defined by interconnected software components, the security of the software supply chain has become a paramount concern for organizations worldwide. This Nullcon talk, "Securing the chains: Building defensive layers for software supply chains," delivered by Yadu Krishna, a Security Engineer at GED, delves into the complexities of establishing robust defenses against the escalating threat of supply chain attacks. Krishna shares insights from his team's journey in developing an open-source solution, Supply Shield, designed to provide comprehensive visibility, actionable intelligence, and scalable security measures for modern software development environments.

The presentation addresses the critical need for an adaptive and developer-friendly approach to supply chain security, moving beyond traditional, fragmented tools. It highlights the challenges organizations face when confronted with zero-day vulnerabilities like Log4j and offers a strategic framework for proactive defense. By focusing on practical implementations, the talk outlines how to identify, categorize, and prioritize software vulnerabilities across various layers of the development and deployment pipeline, ultimately empowering engineering teams to build more secure software.

This article will explore the methodologies and technical innovations presented by Krishna, detailing how his team tackled common pain points such as vulnerability ownership, transitive dependencies, and effective prioritization. It will provide a deep dive into the architecture of Supply Shield, its integration with existing CI/CD workflows, and the future directions for enhancing software supply chain resilience.

Background

▶ Watch: Introduction and defining the software supply chain. (0:00)

The concept of a supply chain, traditionally associated with the procurement and manufacturing of physical goods, finds a direct parallel in the realm of software. A software supply chain encompasses every element involved in creating, delivering, and maintaining a software product. This includes the initial ideas, the source code written by developers, third-party libraries and dependencies, build systems, continuous integration/continuous delivery (CI/CD) pipelines, and the deployment environments. Just as a physical product relies on a bill of materials (BOM), software relies on a Software Bill of Materials (SBOM), which enumerates all the components, dependencies, and third-party tools used in an application.

The importance of securing this chain has been underscored by a dramatic increase in supply chain attacks. Krishna cites alarming statistics: 51% of JavaScript components contain a known vulnerability, and even after significant public awareness, 13% of Log4j downloads remain vulnerable. The infamous Log4j vulnerability serves as a stark reminder of how a single flaw in a widely used component can ripple through countless applications, posing an immense challenge for organizations to identify and remediate.

Traditional approaches to vulnerability management, often relying on standalone tools, proved inadequate in the face of such widespread threats. These tools typically suffer from critical limitations: they are not scalable for large organizations, offer poor visibility into the entire software ecosystem, provide delayed results, and lack historical tracking capabilities. This fragmented and reactive posture leaves organizations ill-prepared for zero-day events and struggles to provide clear, actionable guidance to development teams. The core problem statement for Krishna's team was to build a solution that is scalable, ensures end-to-end visibility, provides instant results, tracks vulnerabilities, and offers clear action items—a solution that could effectively prepare an organization for a zero-day.

Key Findings

▶ Watch: Statistics on supply chain attacks and Log4j vulnerability. (2:30)

The central finding of Krishna's work is that effective software supply chain security requires a multi-layered, evolving approach that prioritizes visibility, granular ownership, and intelligent prioritization. The journey began with an open-source project called Supply Shield, designed to overcome the limitations of traditional security tools.

The first key finding was the strategic decision to start vulnerability scanning at the Docker image level rather than directly from Git repositories. While Git provides source code, Docker images offer a more accurate representation of what will actually be deployed. This approach ensures visibility into exact runtime versions of packages and all their dependencies, including those installed during the build process that might not be explicitly listed in source code manifests (e.g., pom.xml files in Java projects). By scanning Docker images, the team could generate a live SBOM, offering a precise inventory of all "materials" used in the final product.

However, this initial Docker-centric approach quickly revealed subsequent challenges, leading to further key findings and refinements:

1. Distributed Ownership of Vulnerabilities: It became apparent that vulnerabilities are introduced by different teams. A significant portion often originates from the base images provided by platform or SRE teams, not the application code written by developers. This necessitated a mechanism to segregate vulnerabilities based on their source.
2. Prevalence of Unapproved and Outdated Base Images: Organizations often struggle with developers using non-standard or vulnerable versions of base images, introducing significant security risks and operational overhead. A method for identifying and managing these "alien" or outdated images was crucial.
3. The "Transitive Dependency" Dilemma: Developers frequently faced vulnerabilities in packages they did not directly import, leading to confusion and pushback. The Docker-generated SBOMs, while comprehensive, lacked the crucial relationship information needed to trace how a transitive dependency was introduced. This highlighted the need for richer SBOMs.
4. Vulnerabilities in Common/Shared Packages: Many organizations utilize internal shared libraries or components developed by a central platform team. When vulnerabilities appear in these common packages, attributing ownership and responsibility for fixing them to individual microservice developers is inefficient and incorrect. A system to differentiate and assign ownership for these shared components was required.
5. Ineffective Prioritization with CVSS Alone: Relying solely on CVSS scores for prioritization proved insufficient. A high CVSS score does not necessarily mean a vulnerability is exploitable or reachable in a given context (e.g., in a non-public-facing service or a deeply nested function). A more nuanced prioritization strategy was essential.

Each of these findings drove the evolution of Supply Shield, leading to the integration of source code analysis, sophisticated relationship mapping, and intelligent ownership and prioritization algorithms, transforming it into a more comprehensive and developer-friendly supply chain security program.

Technical Deep Dive

▶ Watch: Why Docker images are chosen for vulnerability scanning. (4:30)

The technical architecture of Supply Shield evolved through several iterations to address the identified challenges. Initially, the system focused on Docker image-based scanning. The workflow began with a developer pushing code to a Git repository. During the build process, the application's image would be pushed to an ECR (Elastic Container Registry). An automated system would then fetch this image and use a tool called Sift to generate a Software Bill of Materials (SBOM), which was subsequently stored in S3. This provided a live, queryable SBOM, detailing all dependencies present in the deployed image.

The first major technical hurdle was segregating SCA (Software Composition Analysis) issues based on ownership. Docker images are composed of multiple layers, typically starting with a base image (e.g., FROM python:3.9). Vulnerabilities within this base image are the responsibility of the platform or DevOps team, not the application developer. Supply Shield solved this by utilizing layer hashes. Each layer in a Docker image has a unique hash. By comparing the layer hashes of a service image with known, approved base images, the system could accurately identify which vulnerabilities originated from the base image and attribute them accordingly.

This layer-hash mechanism was extended to identify alien and outdated base images. An "alien" image is one not approved by the organization's platform team, posing security, debugging, and compliance risks. An "outdated" image is an approved base image but an older, vulnerable version. By maintaining an inventory of approved base image layer hashes and their latest secure versions, Supply Shield could flag non-matching hashes as alien and common hashes with available upgrades as outdated. While the talk acknowledged the challenges of outright blocking alien images (developer pushback, breaking existing builds), the identification itself provided critical visibility.

The most significant technical advancement came from addressing the problem of transitive vulnerabilities and the lack of relationship information in Docker-only SBOMs. Developers were often presented with a list of vulnerable packages they didn't directly import, leading to frustration. The initial Docker-based SBOMs generated by Sift lacked the package relationship information required to trace how a vulnerable dependency was introduced. To overcome this, Supply Shield adopted a combined approach integrating Git source code analysis with CDXgen.

CDXgen is an open-source tool that, when run against source code, builds the service and generates a comprehensive SBOM, including crucial relationship information between packages (e.g., Package A imports Package B, which imports vulnerable Package C). This provided the exact runtime package versions and the full dependency graph. The updated pipeline involved Jenkins triggering a build, sending a message to a queue, which was then consumed by Supply Shield. This tool, leveraging CDXgen, generated the enriched SBOM and pushed it to S3. Crucially, Krishna mentioned that his team contributed to ScanCode, an open-source tool, to add the capability to parse and store these detailed package relationships in their database, making the transitive vulnerability paths visible.

Another complex problem was handling vulnerabilities in common or shared packages developed internally. If a platform team maintains a shared authentication library used by many microservices, and that library has a vulnerability, the responsibility for fixing it lies with the platform team, not the individual service developer. Supply Shield implemented an algorithm to segregate these vulnerabilities. By traversing the dependency path backward from a vulnerable package, the algorithm identifies where the package namespace changes from an external, public package to an internal, company-specific package (e.g., com.company.package). The first internal package in that chain is then identified as the responsible entity, effectively assigning ownership to the platform team.

Finally, the system tackled vulnerability prioritization. Recognizing that a high CVSS score alone is insufficient (as it doesn't account for reachability or exploitability in context), Supply Shield adopted a strategy to prioritize vulnerabilities in public-facing services first. This was implemented by integrating with the API Gateway to extract a list of public APIs. By correlating this list with the SCA issues identified from repositories, the system could highlight vulnerabilities affecting publicly exposed services as top priority. Future enhancements include leveraging EPSS (Exploit Prediction Scoring System) and conducting reachability analysis to further refine prioritization, potentially using tools like Atom.

The output of these analyses is presented through dashboards that focus on "actionables" rather than raw vulnerability counts. These dashboards provide developers with a clear list of parent packages to upgrade, excluding base image or common package issues, thereby streamlining the remediation process.

Demo / Proof of Concept

▶ Watch: High-level process of Docker image vulnerability scanning. (6:00)

While the talk did not feature a live, interactive demonstration of the Supply Shield tool in action, it extensively detailed the architectural components, workflows, and algorithmic solutions that constitute its proof of concept. Krishna walked the audience through the logical flow of how a developer's code is scanned, how SBOMs are generated and enriched, and how vulnerabilities are analyzed, segregated, and prioritized.

The "proof of concept" was implicitly presented through the explanation of the system's capabilities:

• Docker image scanning: How Sift generates SBOMs from images pushed to ECR.
• Layer hash segregation: The mechanism for distinguishing base image vulnerabilities from application layer vulnerabilities.
• CDXgen integration: The process of generating comprehensive SBOMs with package relationship information from source code.
• ScanCode contribution: The team's work to enable parsing and storing these relationships, demonstrating the ability to trace transitive dependencies.
• Common package algorithm: The logic for identifying and reassigning ownership of vulnerabilities in shared internal libraries.
• Prioritization logic: The method for identifying public-facing services and prioritizing their associated vulnerabilities.
• Dashboard concept: The visualization of "actionable" items for developers and leadership.

Essentially, the entire talk served as a detailed technical blueprint and validation of the Supply Shield system's design and functionality, illustrating how it works and what problems it solves, rather than a step-by-step live walkthrough. Krishna confirmed in the Q&A that Supply Shield has been deployed and is working successfully in multiple companies, demonstrating its real-world efficacy.

Defensive Implications

▶ Watch: Solving ownership with Docker layer hash-based vulnerability segregation. (8:00)

The insights and solutions presented in "Securing the chains" offer several critical defensive implications for organizations aiming to strengthen their software supply chain security:

1. Prioritize End-to-End Visibility: The fundamental defensive posture begins with comprehensive visibility. Organizations must implement robust SBOM generation across their entire software development lifecycle, ensuring that every component, direct or transitive, is accounted for. This means moving beyond basic dependency lists to rich SBOMs that include package relationships, as demonstrated by the use of CDXgen. You cannot secure what you cannot see.

1. Adopt a Layered Scanning Strategy: Relying solely on source code scanning or solely on runtime image scanning is insufficient. A combined approach, starting with Docker image scanning for runtime accuracy and augmenting it with source code analysis (e.g., Git + CDXgen) for dependency relationship mapping, provides the most complete picture of vulnerabilities. This allows for both precise version detection and root cause analysis.

1. Implement Granular Vulnerability Ownership and Segregation: Defenders must recognize that not all vulnerabilities are the responsibility of application developers. Establishing clear mechanisms to segregate vulnerabilities originating from base images (platform/DevOps responsibility) versus application code (developer responsibility) is crucial. The use of layer hashes and algorithms to identify common package ownership (e.g., shared internal libraries) prevents developer pushback and ensures the right teams are tasked with remediation.

1. Enforce Approved and Up-to-Date Base Images: Organizations should establish a policy for approved base images and actively monitor for the use of alien or outdated images. While strict blocking might be challenging initially, continuous detection and alerting, coupled with clear communication, can drive adoption of secure base images maintained by central teams, significantly reducing the attack surface.

1. Develop Context-Aware Prioritization: Moving beyond raw CVSS scores is essential for effective remediation. Defenders should implement prioritization strategies that consider the reachability and exploitability of vulnerabilities, as well as the criticality of the affected service (e.g., public-facing services first). Future capabilities like EPSS scores and reachability analysis (potentially using tools like Atom) will further enhance this.

1. Foster a Developer-Centric Security Culture: Security must be a collaborative effort. The solutions should be designed to work for engineers, not against them. Providing clear, actionable intelligence (e.g., "number of actionables" instead of raw vulnerability counts), suggesting secure upgrade versions, offering open-source alternatives, and integrating security checks directly into IDEs (via AI-based plugins) helps embed security seamlessly into the development workflow and reduces friction.

1. Embrace Open Source for Common Problems: As Supply Shield itself is an open-source project, the talk implicitly advocates for leveraging and contributing to open-source solutions for shared industry challenges. This fosters community collaboration, accelerates innovation, and provides transparent, auditable security tooling.

By implementing these defensive layers, organizations can shift from a reactive, firefighting mode to a proactive, resilient stance against the ever-evolving threats within the software supply chain.

Key Takeaways

• Visibility is Foundational: You cannot secure what you cannot see. Comprehensive Software Bill of Materials (SBOMs), especially those detailing package relationships, are the first step to understanding and securing your software supply chain.
• Evolve Your Security Approach: There is no one-size-fits-all solution. Security programs must continuously evolve, adapting to new threats and technologies. Starting with Docker image scanning and then integrating source code analysis (CDXgen) for richer context is a powerful progression.
• Segregate Vulnerability Ownership: Clearly define who is responsible for fixing vulnerabilities. Differentiate between issues in base images (platform team), common packages (shared services team), and application code (developer), using mechanisms like layer hashes and namespace analysis.
• Prioritize Smartly, Not Just by Severity: CVSS scores alone are insufficient. Prioritize vulnerabilities based on factors like exploitability, reachability, and whether they affect public-facing services.
• Empower Engineers with Actionable Insights: Security tools should complement and assist developers. Provide clear "actionables," suggest secure upgrade versions, and integrate security feedback directly into developer workflows (e.g., IDE plugins) to foster a collaborative security culture.
• Leverage and Contribute to Open Source: Open-source tools like Sift, CDXgen, ScanCode, and OSV are vital for building effective supply chain security. Contributing back to these projects helps solve common industry problems collectively.

About the Speaker(s)

Yadu Krishna is a Security Engineer at GED. His professional background includes active participation in Capture The Flag (CTF) competitions with Team Bios, where he honed his skills in offensive security. He also dedicates time to hunting for Common Vulnerabilities and Exposures (CVEs), demonstrating a commitment to identifying and addressing security flaws in software. His work at GED focuses on building robust security solutions, particularly in the challenging domain of software supply chain security.