Identity-based Trust - Till Death Do We Part? - John Kjell & Kairo De Araujo

Identity-based Trust - Till Death Do We Part?

Speakers: John Kjell, Co-chair CNCF Security TAG & Maintainer at Control Plane; Kairo De Araujo, Open Source Software Engineer & Maintainer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=26p_qvuCy-s

Overview

In "Identity-based Trust - Till Death Do We Part?", John Kjell and Kairo De Araujo delivered a critical examination of modern software supply chain security, highlighting the evolution from traditional signing methods to identity-based approaches and beyond. The talk, presented at KubeCon EU, delves into the limitations of simply signing artifacts and champions a multi-layered strategy involving project in-toto, The Update Framework (TUF), and Sigstore components like Fulcio and Rekor. Kjell and De Araujo demonstrate that while identity-based signing is a significant improvement, it alone cannot guarantee the integrity or safety of software.

The core message of the presentation is that true trust in software artifacts requires not only knowing who signed something but also understanding how it was built, what processes it underwent, and how it's distributed. They advocate for a comprehensive approach that integrates attestations, policy enforcement, and robust distribution mechanisms to detect subtle compromises that might otherwise slip through basic signature validation. This talk is essential for anyone involved in developing, securing, or consuming software, particularly in cloud-native environments, as it dissects the complexities of modern software provenance and offers practical, open-source solutions to enhance trust.

Background

▶ Watch: Speakers introduce themselves and talk overview (0:00)

Historically, software integrity has relied heavily on cryptographic signatures, where a developer uses a private key to sign an artifact, and users verify this signature with a corresponding public key. While fundamental, this model faces significant challenges: the secure distribution of public keys, the difficulty of revoking a compromised private key, and the need to re-sign and redistribute all affected artifacts if a key is lost or leaked. This process is cumbersome and prone to error, especially in dynamic development environments with many releases and contributors.

The advent of identity-based signing, particularly through projects like Sigstore, aimed to address these key management complexities. Sigstore leverages OpenID Connect (OIDC) to authenticate developers, allowing a Certificate Authority (CA) like Fulcio to issue short-lived, ephemeral keys and certificates. These keys sign artifacts, and the certificates are then published to a transparency log like Rekor. This approach eliminates the long-term management of private keys, as they are destroyed after use. However, Kjell and De Araujo highlight a critical limitation: while Rekor provides an immutable, auditable record of signatures, it's an append-only log, meaning entries cannot be removed. This ensures transparency but offers no mechanism for revocation if a signed artifact is later found to be malicious or compromised. A signature merely states "this is a thing I signed," not "this is a safe thing I signed."

Further efforts to enhance supply chain security introduced concepts like Salsa build attestations, which focus on documenting the build process itself. While valuable for proving an artifact originated from a specific source via a defined build process, Salsa primarily covers the "middle" part of the supply chain. It doesn't inherently account for crucial pre-build steps like vulnerability scans, static analysis, code reviews, or post-build steps like testing and license compliance. This gap leaves a significant portion of the software lifecycle vulnerable to undetected compromises, even if the final artifact is cryptographically signed and its build provenance is attested. The problem exists because the scope of "trust" needs to extend beyond mere identity and build steps to encompass the entire journey from source code to consumer.

Key Findings

▶ Watch: OIDC identity-based signing with ephemeral keys (3:00)

The talk presents several key findings and contributions that collectively form a robust framework for enhancing software supply chain security:

1. Identity-Based Signing is a Foundation, Not a Solution: While Sigstore (with Fulcio and Rekor) significantly improves key management by using ephemeral keys and providing a transparency log, it primarily addresses who signed an artifact. It does not inherently guarantee the artifact's quality, security, or that it was produced according to expected processes. A valid signature merely means the certificate was valid at the time of signing and is recorded, but it doesn't prevent the signing of malicious or buggy software.

1. Attestations Provide Crucial Context and Granularity: Beyond build provenance (e.g., Salsa), the talk emphasizes the need for comprehensive attestations that cover every step of the software supply chain. Projects like Witness allow for the generation of attestations for actions such as vulnerability scans, static analysis, code reviews, and testing. These attestations, when stored and indexed by tools like Archavista, create a verifiable, auditable trail of the artifact's journey.

1. Policy Enforcement is Paramount for Real Trust: The most significant finding illustrated by the "buggy release" demo is that even with valid signatures and attestations, software can be compromised. A Witness policy, often expressed in Rego, is essential to evaluate these attestations against an organization's security and quality expectations. This policy layer can detect subtle manipulations (e.g., ignoring linter rules, running as root) that bypass simple signature checks and green CI/CD pipelines, forcing a deeper inspection of the artifact's actual state.

1. The Update Framework (TUF) Enables Secure Distribution and Revocation: The talk powerfully demonstrates that secure distribution and, critically, revocation are missing links in many supply chain strategies. TUF provides a resilient framework for clients to securely download and verify software. It protects against rollback attacks, inconsistent repositories, and allows for the explicit "un-signing" or revocation of compromised or unwanted versions from the trusted metadata. This ensures that even if a buggy or malicious artifact remains in a registry or CDN, clients will refuse to download it because it is no longer listed in the trusted TUF metadata.

1. A Multi-Layered, Integrated Approach is Necessary: The overall finding is that no single tool or concept is sufficient. Robust software supply chain security requires the integration of identity-based signing (Sigstore), comprehensive attestation generation (Witness), intelligent attestation storage and indexing (Archavista), policy enforcement (Witness/Rego), and a secure distribution mechanism with revocation capabilities (TUF). This layered defense provides a much higher degree of trust and resilience against various attack vectors.

Technical Deep Dive

▶ Watch: Detailed Sigstore ecosystem workflow (Fulsio, Rekor, TUF) (4:00)

The talk meticulously dissects the technical mechanisms underpinning modern software supply chain security, starting with the evolution of signing practices.

Traditional vs. Identity-Based Signing:

In the traditional signing model, a developer generates a private key and a corresponding public key. The private key is used to create a digital signature for an artifact. This private key must be kept secret and secure indefinitely. The public key is then distributed to users, who use it to verify the artifact's signature. The challenges here include the secure distribution of public keys, the cumbersome process of key rotation if a private key is compromised, and the need to re-sign and re-publish all artifacts with a new key.

Identity-based signing, as implemented by Sigstore, offers a significant leap forward. When a developer wants to sign an artifact using a tool like Cosign, they first authenticate their identity using an OIDC provider (e.g., GitHub, Google). This authentication request goes to Fulcio, Sigstore's Root Certificate Authority. Fulcio, upon successful OIDC verification, issues a short-lived, ephemeral private key and an X.509 certificate containing the developer's OIDC identity. This ephemeral private key is used only once to sign the artifact, and then it is immediately destroyed. The resulting certificate and signature are then published to Rekor, Sigstore's transparency log. Rekor is an append-only log, meaning once an entry (a certificate and signature) is added, it cannot be modified or removed, providing an immutable audit trail. End-users can then verify the artifact's signature by checking the certificate against Rekor and ensuring it's part of the trusted root. This process eliminates the long-term management of private keys, shifting the burden to identity providers and the transparency log.

Attestations and Policy Enforcement with in-toto Projects:

While identity-based signing verifies who signed an artifact, it doesn't verify what happened during its creation. This is where in-toto projects like Witness and Archavista come into play, extending the concept of Salsa build attestations. Salsa (Supply-chain Levels for Software Artifacts) focuses primarily on documenting the build process itself, proving that an artifact came from a specific source via a defined build environment.

Witness goes further by enabling the generation of granular attestations for every step in the software supply chain. This includes not only build steps but also pre-build activities like vulnerability scans, static analysis, code reviews, and post-build activities like testing and license compliance. For each step, Witness wraps the execution, capturing inputs, outputs, parameters, and environmental details, and then signs these attestations. These attestations are then stored and indexed by Archavista, an in-toto project that captures the relationships between attestations and artifacts, creating a comprehensive graph of the software's provenance.

The critical next step is policy enforcement. Merely generating attestations is insufficient; they must be evaluated against defined security and quality standards. Witness integrates with Rego, the policy language for Open Policy Agent (OPA). A Witness policy written in Rego can inspect the contents of attestations, checking if specific commands were run with expected parameters (e.g., no --ignore flags for linters), if certain environment variables were set, or if specific security tools reported failures. This allows organizations to define their expectations for the entire supply chain and automatically detect deviations, even if individual steps "pass" and signatures are valid. The example given in the talk of a Dockerfile linter being bypassed by an --ignore flag, which was caught by a Witness policy, perfectly illustrates this capability.

Secure Distribution and Revocation with The Update Framework (TUF):

Even with comprehensive attestations and policy enforcement, the final challenge is secure distribution and, crucially, revocation. If a vulnerability is found post-release, how do you prevent users from downloading a compromised version? The Update Framework (TUF) addresses this.

TUF operates on the principle of client-side verification using a hierarchical system of signed metadata. A TUF client is initialized with a trusted root metadata file, which contains the public keys of the top-level roles responsible for signing subsequent metadata. This root metadata is usually embedded within the client. When a client requests an artifact (e.g., Docker image V1.2), it first fetches the latest TUF metadata. This metadata is structured to protect against various attacks, including rollback attacks (preventing clients from being tricked into downloading older, vulnerable versions) and mix-and-match attacks (ensuring all components come from a consistent repository state).

The TUF metadata lists the hashes and sizes of all approved artifacts. If a version, say V1.1, is found to be buggy or malicious, it can be removed from the TUF metadata. This updated metadata is then signed by authorized parties (e.g., using Sigstore for the metadata signing itself) and published. When a client attempts to download V1.1, the TUF client will check the metadata, find that V1.1 is no longer listed as approved, and refuse to download it, even if the artifact physically remains in the registry or CDN caches. This provides a powerful revocation mechanism that complements the auditing capabilities of Rekor. TUF also supports key rotation without requiring client updates, enhancing its long-term resilience. The talk highlights how the client downloads various metadata roles (root, timestamp, snapshot, targets) to ensure the integrity and freshness of the repository and finally verifies the artifact's hash and size against the signed metadata before deeming it safe to deploy.

Demo / Proof of Concept

▶ Watch: Emergency release scenario with valid but flawed attestations (7:00)

The speakers presented a compelling demonstration built around a fictional software project, illustrating the shortcomings of identity-based signing alone and the power of a multi-layered approach.

The demo began with the release of V1.0 of their software, which went through a GitHub Actions pipeline, generating a Docker image and Salsa build attestations wrapped by Witness. All steps were signed, and the pipeline showed green, signifying an apparently successful release.

The plot thickened with an "emergency release" of V1.1 to fix a panic bug. The developers "smashed merge" on a Pull Request (PR), triggering another GitHub Actions workflow. Again, attestations were created for every step, all signatures passed validation, and the pipeline lights were green. However, when a Witness policy was applied, it flagged an error related to the linting step. The policy revealed that the PR had, unbeknownst to a casual review, removed the panic but also introduced a change to run the container as root and, critically, added an --ignore flag to bypass the Dockerfile linter. This subtle manipulation meant that while the linter ran, it didn't enforce the organization's security standards, yet the pipeline still passed, and the artifact was signed. This perfectly illustrated the talk's core premise: green lights and valid signatures are insufficient if the underlying processes are subverted.

To rectify this, a V1.2 release was prepared. This time, the PR was carefully reviewed, removing the root user and the linter --ignore flag. The GitHub Actions pipeline ran, generating attestations that were stored in Archavista. When the Witness policy was applied, it passed, indicating that the software now met the defined security expectations.

The final, crucial part of the demo focused on secure distribution using The Update Framework (TUF). With V1.2 now deemed secure, the team used TUF to manage the distribution. They first updated the TUF metadata to remove V1.1, the buggy and compromised version. John Kjell then signed this updated metadata using Sigstore, ensuring its integrity. A client attempting to download V1.1 would now be prevented from doing so by the TUF client, even if the image was still physically present in the registry. Subsequently, V1.0 was also removed from the TUF metadata, leaving only V1.2 as the officially approved and verifiable version. This demonstrated TUF's powerful ability to perform revocation and ensure that users only consume trusted artifacts, effectively "un-signing" problematic versions from the distribution channel.

The talk also briefly touched upon offline capabilities, noting that Archavista can be run locally, Sigstore is moving towards a bundle format with inclusion proofs and signed timestamps for offline validation, and TUF (especially its variant Optane, used in automotive updates) is designed for air-gapped or intermittently connected environments, allowing metadata to be validated offline.

Defensive Implications

▶ Watch: Policy enforcement reveals issues despite valid signatures (8:45)

The insights from this talk provide clear, actionable guidance for defenders looking to strengthen their software supply chain security:

1. Shift Beyond Basic Signature Verification: Defenders must understand that a valid signature only verifies the identity of the signer, not the integrity or safety of the artifact itself. Relying solely on tools like Sigstore for signing, while a critical first step, is insufficient.
2. Implement Comprehensive Attestation Generation: Utilize tools like Witness to generate detailed attestations for every significant step in the software supply chain, not just the build process. This includes pre-build steps (e.g., static analysis, vulnerability scans, dependency checks, code reviews) and post-build steps (e.g., integration tests, license compliance checks). These attestations provide the necessary context to make informed trust decisions.
3. Enforce Policies with Attestations: Crucially, integrate policy enforcement mechanisms (e.g., Witness policies written in Rego) that evaluate these attestations against an organization's specific security and compliance standards. This layer can detect subtle compromises, misconfigurations, or intentional bypasses that might otherwise result in "green" pipelines and valid signatures for insecure artifacts.
4. Adopt a Secure Distribution and Revocation Framework: Implement The Update Framework (TUF) to manage the distribution of software artifacts. TUF provides robust protection against rollback attacks, ensures repository consistency, and, most importantly, offers a reliable revocation mechanism. This allows defenders to quickly "un-sign" and prevent the deployment of compromised or buggy versions, even if they persist in registries or caches.
5. Monitor Transparency Logs: While Rekor is append-only, defenders should still monitor the Rekor transparency log for unexpected certificates issued under their organization's identity. Tools for Rekor monitoring can alert security teams to potential compromises of OIDC providers or Fulcio itself.
6. Decouple Policy from Build Failures (Strategically): Consider decoupling strict policy failures from immediate CI/CD pipeline failures. While critical security violations should halt builds, some policy checks can happen later, allowing developers to iterate quickly while security teams enforce standards before deployment. This balances developer velocity with security rigor.
7. Plan for Offline and Air-Gapped Environments: For organizations operating in highly restricted or disconnected environments, explore the offline capabilities of these projects. TUF is particularly well-suited for such scenarios (e.g., Optane), and Sigstore is evolving its bundle format to support offline verification.
8. Understand the Full Attack Surface: Recognize that the software supply chain is a complex ecosystem. Attacks can occur at any point, from source code injection to compromised build environments, malicious dependencies, or subverted distribution channels. A multi-layered defense strategy is essential to address this broad attack surface.

Key Takeaways

• Identity-based signing (Sigstore) is a necessary but insufficient foundation. It simplifies key management and provides auditability via transparency logs (Rekor) but doesn't guarantee software safety or prevent the signing of compromised artifacts.
• Comprehensive attestations (Witness) are crucial for context. Generate attestations for all supply chain steps—from code review to testing—to provide a verifiable record of how software was produced, complementing basic build provenance.
• Policy enforcement (Witness/Rego) is vital to interpret attestations. Policies allow organizations to define expectations and automatically detect deviations or subtle compromises that pass simple signature checks and green CI/CD pipelines.
• Secure distribution and revocation (TUF) are essential for artifact lifecycle management. TUF protects against rollback attacks, ensures consistency, and enables explicit revocation of compromised or unwanted software versions, preventing their deployment even if physically available.
• A multi-layered approach is key to robust supply chain security. Combining Sigstore for identity, Witness for attestations, Archavista for storage, and TUF for distribution and revocation creates a resilient defense against a wide range of supply chain threats.
• Trust must extend beyond "who signed it" to "how it was built and distributed." True confidence in software requires verifying the entire provenance and distribution pipeline, not just the final signature.

About the Speaker(s)

John Kjell is a prominent figure in the cloud-native security space. He recently joined Control Plane and serves as a co-chair for the CNCF Security TAG. John is also a maintainer on Witness and Archavista, both sub-projects under in-toto, demonstrating his deep involvement in software supply chain security. His contributions extend to projects like Salsa, further solidifying his expertise in securing the software development lifecycle.

Kairo De Araujo is an open-source software engineer dedicated to enhancing software integrity. He is a maintainer of the TUF Python library, a critical component of The Update Framework, and also contributes as a maintainer to Archavista, another in-toto project. Kairo's work focuses on building robust, resilient systems for secure software distribution and provenance.