Weaving a VEX Feed Through the Kubernetes Project

Weaving a VEX Feed Through the Kubernetes Project

Speakers: Adolfo Garcia, Software Engineer, Carabiner Systems; Technical Lead, OpenVEX Project (OpenSSF); Kubernetes Release Engineering

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=ZxckMgjlsns

Overview

In this insightful talk, Adolfo Garcia, known as "Puerto," from Carabiner Systems and a key contributor to Kubernetes Release Engineering and the OpenVEX project, delves into the complex yet critical endeavor of generating and managing Vulnerability Exploitability eXchange (VEX) feeds for the expansive Kubernetes project. The presentation highlights the unique challenges posed by a project of Kubernetes' scale and distributed nature, offering a practical roadmap and newly developed tooling to address these.

The core of the talk revolves around the OpenVEX standard, emphasizing its role in enabling precise communication regarding the impact of known vulnerabilities on specific software artifacts. Garcia articulates how VEX can transform vulnerability management by allowing projects to declare whether a detected vulnerability truly affects their product, is not affected, is under investigation, or has been fixed. This capability is becoming increasingly vital for compliance with emerging regulations, such as the Cyber Resilience Act (CRA), which mandates transparency around known vulnerabilities in shipped products.

This initiative is not merely about compliance; it's about shifting the paradigm of vulnerability reporting from a flood of potential issues to a focused, actionable stream of confirmed risks. By integrating VEX generation directly into the Kubernetes release process, the project aims to provide downstream consumers with a clearer, more accurate understanding of their security posture, ultimately enhancing the overall supply chain security ecosystem for cloud-native technologies.

Background

▶ Watch: Speaker introduction and talk overview (0:00)

The landscape of software supply chain security has evolved rapidly, with a heightened focus on understanding and communicating the true impact of vulnerabilities. Traditional vulnerability scanners often produce extensive lists of findings, many of which may be false positives or non-exploitable within a specific product's context. This "noise" can overwhelm development teams and security analysts, hindering effective remediation efforts. The Vulnerability Exploitability eXchange (VEX) standard emerged as a solution to this problem.

VEX documents provide a structured, machine-readable way to convey an impact assessment of a vulnerability on a specific product or component. This assessment can be one of four statuses: affected, not affected, under investigation, or fixed. The goal is to augment raw vulnerability data (e.g., from CVEs or scanner reports) with expert knowledge, allowing consumers to filter out irrelevant findings and focus on actual risks.

Several VEX flavors exist today, including those embedded within CSAF (Common Security Advisory Framework), CycloneDX, and SPDX (Software Package Data Exchange). Adolfo Garcia's talk focuses heavily on OpenVEX, a lightweight, JSON-LD based standard championed by the Open Source Security Foundation (OpenSSF). OpenVEX is designed to be "thin, lean," and easily embedded into other formats, such as attestations, making it highly suitable for modern, automated software supply chain processes. Tools like Trivy and Grype already natively support ingesting OpenVEX documents, demonstrating its practical utility in real-world scanning workflows.

For a project as vast and dynamic as Kubernetes, generating accurate VEX information presents unique challenges. Kubernetes is a CVE Numbering Authority (CNA), constantly evolving with multiple active branches, and involves numerous special interest groups (SIGs) and maintainers. Key questions arise: Who has the authority to issue VEX statements? How are these statements validated and signed? How can the process keep pace with continuous development and patching? Furthermore, the project's output includes a multitude of artifacts, from binaries to container images, each requiring precise VEX assessments. Addressing these complexities is crucial for Kubernetes to meet future compliance demands and enhance its security posture for millions of users.

Key Findings

▶ Watch: Overview of VEX flavors and OpenVEX design principles (4:00)

The Kubernetes project is actively developing a comprehensive VEX feed, addressing the challenges of scale and distributed ownership. The core strategy involves synthesizing VEX information from multiple, distinct sources into a unified feed.

1. Multi-Source VEX Feed Architecture: The ideal Kubernetes VEX feed is envisioned as a convergence of three primary sources:

• Vulnerabilities in Kubernetes Components: This category is further split into two sub-feeds:
• Go vulncheck Automatic Non-Impact Detection: Utilizing the Go toolchain's Go vulncheck to automatically identify Go module vulnerabilities that are not reachable or exploitable by the Kubernetes code. This tool can natively output OpenVEX documents.
• Maintainer-Issued Assessments: A mechanism for project maintainers to explicitly declare the impact status of vulnerabilities based on their domain expertise, particularly for those not covered by automated tools.
• VEX from Kubernetes Security Advisories: Leveraging existing security advisories (which detail affected components) to derive VEX statements for both affected and not affected artifacts across the entire project. This involves a planned rewrite of the advisory generation process to start from the Open Source Vulnerabilities (OSV) format.
• Vulnerabilities in Base Images: Addressing the unique challenge of static base images used by Kubernetes, which are not upgraded after release and thus accumulate vulnerabilities over time. This area still has open questions regarding its full implementation.

1. Vex Flow Tool for Maintainer Assessments: A new open-source tool, Vex flow, has been developed to streamline the process of maintainer-issued VEX statements. This tool automates the vulnerability triage workflow by:

• Periodically scanning Kubernetes branches using OSV scanner.
• Opening GitHub issues for newly discovered vulnerabilities, providing a structured platform for discussion.
• Leveraging Kubernetes' existing owners file mechanism to authorize maintainers for issuing assessments via a ChatOps interface.
• Automatically generating, Sigstore-signing, and publishing OpenVEX documents to a GitHub attestation store based on maintainer input.

1. Advisory Generation Rewrite to OSV: The Kubernetes Security Response Committee is rewriting its advisory generation pipeline. By starting with the OSV format, they can derive all necessary vulnerability data, including CVE JSON, release notes, and crucially, VEX documents, ensuring consistency and automation across all security communications.

1. VEX Publishing Strategy: The immediate plan for the aggregated VEX data is to attach these statements directly to container images in the registry. Future plans include building a continuous VEX stream for all released statements and collaborating with initiatives like Aqua's VEX hub to centralize open-source VEX information.

The talk confirms that the Go vulncheck and Vex flow components are currently implemented and functioning, with the advisory generation rewrite actively underway.

Technical Deep Dive

▶ Watch: Major challenges of implementing VEX for Kubernetes (5:45)

The technical implementation of Kubernetes' VEX feed generation relies on a sophisticated interplay of existing tools, new custom development, and established project processes.

At its foundation, OpenVEX serves as the chosen standard for vulnerability impact communication. OpenVEX is distinguished by its lightweight, JSON-LD format, designed for embedding and machine readability. It explicitly defines four status messages: affected, not_affected, under_investigation, and fixed. This specificity allows consumers to programmatically interpret the impact, going beyond generic scanner output. The ability to embed OpenVEX documents directly into attestations, particularly those signed with Sigstore, is a critical architectural decision, ensuring the integrity and discoverability of VEX data alongside the software artifacts themselves.

One of the primary sources for VEX data is the Go toolchain's Go vulncheck. This tool performs a reachability analysis, determining whether a vulnerable function in a Go dependency is actually called or used by the application's code. This is a powerful mechanism for automatically identifying "not affected" vulnerabilities, effectively reducing false positives from generic dependency scanners. The key here is that Go vulncheck can natively output its findings in OpenVEX format, simplifying integration into the overall VEX pipeline.

For vulnerabilities not automatically triaged by Go vulncheck or requiring human expertise, the Vex flow tool provides a structured workflow for maintainer assessments. This tool operates by:

1. Continuous Scanning: Periodically running OSV scanner against the active Kubernetes branches (typically three to four under maintenance).
2. Issue Creation: Upon detecting a new vulnerability, Vex flow automatically opens a GitHub issue, pre-populating it with relevant vulnerability information. This issue serves as the central point for triage and discussion.
3. Authorization via owners files: Kubernetes utilizes owners files to define code ownership and permissions within its vast repository structure. Vex flow integrates with this system, ensuring that only authorized maintainers (those listed in the owners file for the relevant code area) can issue VEX assessments.
4. ChatOps Interface: Maintainers interact with the GitHub issue using a ChatOps interface. A simple comment, such as not affected followed by a machine-readable justification (e.g., code_not_present, vulnerable_code_not_in_execute_path, vulnerable_function_not_called), and an optional human-readable explanation, triggers the bot.
5. Attestation Generation and Signing: The Vex flow bot processes the maintainer's assessment, converts it into an OpenVEX document, and then signs it using Sigstore. The signed OpenVEX attestation is subsequently pushed to the GitHub attestation store, providing an immutable record of the assessment. The talk acknowledges that currently, only one assessment is required, but future considerations include requiring multiple approvals.

The Kubernetes Security Response Committee is also undertaking a significant effort to rewrite its security advisory generation process. The new pipeline will begin with the OSV (Open Source Vulnerabilities) format as its canonical data source. From this OSV data, the system will derive various outputs: Kubernetes security advisories, CVE JSON for the CVE database, release notes, and critically, VEX documents. This ensures that VEX statements for affected components are consistent with official advisories and that not affected statements can be systematically generated for other artifacts from the same advisory data. The example given is that an advisory might state kube-proxy for Windows AMD64 is affected, from which VEX for other unaffected artifacts can be derived.

The final step in this process is the VEX assembly. The vex assemble subcommand is used to collect all relevant VEX data. This command considers the current state of vulnerabilities in a given branch at release time. Importantly, while Vex flow generates VEX for the branch, the final assembly process filters these to only include vulnerabilities applicable to the specific artifacts being released (e.g., container images). It merges VEX from Go vulncheck and maintainer assessments, and then allows for the addition of subjects (like container image identifiers) to the final VEX document, ensuring the VEX applies accurately to the shipped product. This aggregated and signed VEX document can then be used by policy engines to enforce compliance, such as halting a release if unaddressed, exploitable vulnerabilities are present.

Demo / Proof of Concept

▶ Watch: Coordinating VEX efforts within the Kubernetes project (7:00)

Adolfo Garcia presented a "children's storybook" demo, a series of screenshots illustrating the Vex flow tool and the overall VEX generation process in action. This demonstration showcased a specially crafted demo repository with two simulated vulnerabilities: one that a scanner would pick up but the code never exploits, and another that Go vulncheck would identify as not exploitable. The goal of the demo was to achieve compliance with a hypothetical policy, akin to the CRA, which forbids shipping artifacts with known, exploitable vulnerabilities.

The demo began by establishing a policy check in the release pipeline: if vulnerabilities are present, the release fails. An initial scan of the demo project, which had the two crafted vulnerabilities, would cause this policy to fail.

Next, the GitHub Actions workflow for Vex flow was triggered. This bot automatically created a triage issue for each detected vulnerability. This issue contained all relevant information about the vulnerability, providing a central point for maintainer interaction.

A maintainer, authorized via the owners file mechanism, then interacted with the issue using a ChatOps interface. For one of the vulnerabilities, the maintainer would comment with not affected and a specific machine-readable justification (e.g., vulnerable_code_not_in_execute_path), along with a human-readable explanation.

Upon receiving this input, the Vex flow bot would spring into action. It picked up the assessment, transformed it into an OpenVEX document, signed the document using Sigstore, and then pushed this signed attestation to the GitHub attestation store. Concurrently, another job running Go vulncheck also produced its own OpenVEX attestation for the vulnerability it identified as not exploitable, signing and uploading it to the same store.

The final step involved the VEX assembly process. The assemble subcommand was invoked, specifying the repository and the triage repository. This command gathered all the VEX attestations from the GitHub attestation store, filtering them to apply only to the vulnerabilities present in the specific release artifact (e.g., a container image) at that moment. The result was a comprehensive VEX document for the final product.

With the aggregated VEX document now available, it was fed back into the policy engine. Because the maintainer assessments and Go vulncheck had successfully declared the vulnerabilities as not affected, the policy check passed, allowing the project to be released. This "storybook" effectively demonstrated how automated tools and maintainer input, combined with a robust attestation and assembly process, can enable a project to achieve compliance and release artifacts with clear vulnerability impact statements.

Defensive Implications

▶ Watch: The vision for Kubernetes' ideal VEX feed (8:00)

The Kubernetes VEX feed initiative carries significant defensive implications for both the Kubernetes project itself and its vast ecosystem of users.

Firstly, it provides a proactive and precise vulnerability management strategy for a project of immense scale. By automating the generation of VEX documents through tools like Go vulncheck and Vex flow, Kubernetes can move beyond reactive patching to a more informed understanding of its actual attack surface. This clarity reduces the burden of false positives on maintainers, allowing them to focus on truly exploitable vulnerabilities.

For downstream consumers, the availability of signed, authoritative VEX documents alongside Kubernetes artifacts fundamentally changes how they assess risk. Instead of relying solely on raw scanner output that often flags non-issues, users can ingest the Kubernetes VEX feed to suppress irrelevant alerts from their own vulnerability scanners (like Trivy or Grype). This drastically improves the signal-to-noise ratio, enabling security teams to prioritize real threats and allocate resources more effectively. The dream of scanners suppressing findings based on VEX, as mentioned in the talk, is already a reality with OpenVEX.

The integration of VEX into the release pipeline allows for policy-driven compliance enforcement. As demonstrated in the PoC, a project can establish policies that prevent the release of artifacts with unmitigated, exploitable vulnerabilities. This is crucial for meeting regulatory requirements like the Cyber Resilience Act (CRA), which will demand greater transparency and accountability for known vulnerabilities in shipped software. By embedding VEX in attestations and signing them with Sigstore, Kubernetes provides cryptographic assurance of the integrity and origin of these security claims.

Furthermore, the owners file mechanism for authorizing maintainer assessments introduces a decentralized yet auditable system for security decision-making. This scales well for large open-source projects where a single security team cannot realistically assess every vulnerability across all components. It empowers domain experts to contribute their knowledge directly to the security posture of their respective modules.

Finally, by building a comprehensive VEX feed and planning for a VEX stream, Kubernetes is fostering greater transparency and trust in its software supply chain. This consistent, machine-readable communication about vulnerability impact builds confidence among users and helps them make informed decisions about deploying and operating Kubernetes clusters, ultimately strengthening the security posture of the entire cloud-native ecosystem.

Key Takeaways

• OpenVEX is a Practical Solution: The OpenVEX standard, with its lean JSON-LD format and focus on vulnerability impact, offers a pragmatic approach to addressing the "noise" of traditional vulnerability scanning and communicating precise exploitability assessments.
• Kubernetes is Building a Comprehensive VEX Feed: The project is actively integrating VEX generation into its release process, drawing from automated Go vulnerability analysis, maintainer expertise, and existing security advisory data to create a robust, multi-source VEX feed.
• Automation is Key for Scale: Tools like the Go toolchain's Go vulncheck (for automated non-impact detection) and the newly developed Vex flow (for structured maintainer assessments) are critical for automating VEX generation in a project as large and dynamic as Kubernetes.
• Decentralized, Authorized Triage: Vex flow leverages Kubernetes' owners file system and a ChatOps interface to enable authorized maintainers to issue VEX assessments directly, scaling the triage process while maintaining accountability and provenance through Sigstore signing.
• VEX Enables Policy Enforcement and Compliance: By generating and assembling VEX attestations, Kubernetes can implement security policies that prevent the release of non-compliant artifacts, addressing emerging regulatory requirements like the CRA.
• Improved Downstream Security Posture: The availability of signed VEX documents allows downstream consumers to accurately suppress false positives from their vulnerability scanners, leading to a clearer understanding of actual risks and more efficient security operations.

About the Speaker(s)

Adolfo Garcia, who often goes by "Puerto," is a software engineer currently working at Carabiner Systems, a small startup. His involvement in the open-source community is significant, particularly within the Kubernetes project, where he is a part of the Release Engineering team. In this role, he contributes to building the supply chain security features for the Kubernetes release process. Additionally, Adolfo is a Technical Lead for the OpenVEX project within the Open Source Security Foundation (OpenSSF), demonstrating his deep expertise and commitment to advancing vulnerability exploitability exchange standards. His dual role in both Kubernetes and OpenVEX makes him uniquely qualified to discuss the challenges and solutions for integrating VEX into large-scale open-source projects.