Rethinking Trust in Forge-Based Git Security

Rethinking Trust in Forge-Based Git Security

Speakers: Aditya Sirish A Yelgundhalli, PhD student, New York University

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=FA1gEAKJAR0

Overview

In the realm of software supply chain security, the integrity of source code stands as a foundational pillar. This talk, delivered by Aditya Sirish A Yelgundhalli, a PhD student at New York University and a leading contributor to projects like in-toto and GitOp, delves into a critical vulnerability inherent in the prevailing centralized trust model of Git-based development. While Git itself is a distributed version control system, the security controls governing code contribution often reside solely within centralized platforms like GitHub, GitLab, and Bitbucket – commonly referred to as "forges." This centralization creates a single point of failure, where a compromise of the forge, an insider threat, or even a software bug can undermine the very policies designed to protect the source code.

The presentation introduces GitOp, an open-source project designed to fundamentally rethink this trust model. GitOp proposes a distributed approach to Git security by embedding cryptographically signed security policies and activity logs directly within the Git repository itself. This paradigm shift empowers individual developers and automated clients to independently verify the integrity of the code and the adherence to security policies, even in the face of a compromised forge. The implications are profound, offering a more resilient and verifiable security posture for software supply chains, a crucial step given the increasing sophistication and frequency of source code-related attacks.

This work is particularly timely and significant as the industry grapples with the escalating threat landscape surrounding software supply chain integrity. By challenging the implicit trust placed in centralized forges for security enforcement, GitOp offers a practical and backwards-compatible solution that aligns with the distributed nature of Git itself. The project's involvement with the Linux Foundation and the Open Source Security Foundation (OpenSSF) further underscores its potential to drive a paradigm shift in how organizations secure their most critical asset: their source code.

Background

▶ Watch: Introduction to rethinking trust in Git security (0:00)

The modern software supply chain is a complex ecosystem, encompassing every stage from source code creation to deployment. As Aditya Yelgundhalli highlights, this intricate web is fraught with threats, particularly at the source code level. The concern is not hypothetical; numerous high-profile incidents underscore the vulnerability of relying on centralized systems for code integrity. For instance, a few years ago, the PHP project discovered malicious commits attributed to language creators that were not genuinely theirs, traced back to a compromised Git server. Similarly, a security researcher exploited a bug in GitHub's authentication to push a commit to the Ruby on Rails repository, demonstrating that even platform-level vulnerabilities can be leveraged to inject malicious code. Other incidents, such as the Juniper breach and the top.gg incident, further illustrate the persistent nature of these threats.

The problem stems from the standard development workflow in a distributed version control system like Git. While developers maintain local copies of a repository, they typically synchronize their changes through a centralized "forge." These forges (e.g., GitHub, GitLab, Bitbucket) are more than just Git servers; they offer a suite of features including issue trackers, code review systems, release management, and crucially, security controls. These controls are usually categorized into:

1. Policy Declaration: Defining who can make changes, which branches are protected, and specific requirements for code contribution.
2. Activity Tracking: Maintaining audit logs of all actions performed on the repository, such as pushes and code review approvals.
3. Policy Enforcement: Verifying that incoming changes adhere to the declared policies before integration.

The fundamental issue, as articulated in the talk, is that this entire security apparatus is centralized within the forge. This creates a single point of trust and, consequently, a single point of failure. The threat model formally considered in the paper posits that any trusted party – be it a maintainer, a developer, or the forge itself – may be compromised and misbehave. This compromise could manifest as:

• Modification of security policies: Weakening rules to allow unauthorized changes.
• Tampering with activity logs: Deleting records of malicious pushes to obscure an attack.
• Subversion of policy enforcement: Bypassing configured rules due to bugs or malicious intent.

In such a centralized model, individual developers pulling code from the forge are essentially blind to these security attributes. They implicitly trust that the forge has correctly enforced policies and that its activity logs are untampered. This leaves them completely out of the equation when it comes to verifying the security of their source code, making them vulnerable to supply chain attacks that exploit the forge's integrity. The core question posed by the researchers is: if Git is a distributed system, why are its security properties so heavily centralized?

Key Findings

▶ Watch: Understanding the centralized role of forges (GitHub, GitLab) (2:50)

The central finding of this research is that the prevailing trust model for Git-based development, which places a disproportionate amount of trust in centralized forges, is fundamentally flawed and vulnerable to compromise. While Git itself is distributed, the security mechanisms—policy declaration, activity tracking, and enforcement—are typically concentrated within a single entity. This centralization creates a critical chokepoint where a compromised maintainer, a malicious insider at the forge, or even a subtle bug within the forge's software can lead to undetected policy violations and the introduction of malicious code.

To address this, the talk introduces GitOp, a novel approach that re-distributes trust by integrating security properties directly into the Git repository. The key insight is to leverage Git's distributed nature for its security controls, rather than relying on an external, centralized authority. GitOp achieves this by storing cryptographically signed metadata representing policies and activity logs within the repository itself. This means that every developer who pulls the repository also receives a copy of these security artifacts, enabling them to independently verify the integrity of the codebase and the adherence to established policies.

The main contributions and findings of GitOp include:

1. Decentralized Policy Enforcement: Security policies are no longer solely enforced by the forge but are embedded within the repository and verifiable by any client. This shifts enforcement from a single, trusted entity to a continuously verifiable, distributed process.
2. Tamper-Evident Activity Logs: Instead of relying on the forge's internal, potentially alterable audit logs, GitOp introduces a Reference State Log (RSL) that records all significant repository actions (e.g., pushes, branch updates) as cryptographically signed data within the repository. This log is append-only and resistant to undetected tampering.
3. Resilience to Forge Compromise: Even if a forge is compromised or buggy and allows an unauthorized push, any developer or client running GitOp will detect the violation upon pulling the repository and performing local verification against the embedded policy and RSL. This "one honest client" principle significantly enhances the overall security posture.
4. Mitigation of Key Compromise: GitOp extends its policy model to include thresholds for approvals, requiring multiple trusted parties to sign off on changes. This mitigates the risk of a single compromised key, as an attacker would need to compromise multiple keys to bypass policies undetected.
5. Backwards Compatibility: A crucial design principle of GitOp is its seamless integration with existing Git workflows and tooling. It does not require developers to abandon their current forges or fundamentally alter their development practices, making it practical for real-world adoption.

In essence, GitOp's key finding is that by decentralizing trust and making security properties an inherent part of the distributed repository, it's possible to build a more robust, transparent, and resilient security model for source code integrity, moving beyond the inherent vulnerabilities of centralized forge-based security.

Technical Deep Dive

▶ Watch: Challenging the trust model of centralized forges (4:10)

GitOp's technical architecture is built upon the principle of embedding verifiable security metadata directly into the Git repository. This approach leverages cryptographic signatures and Git's content-addressable nature to ensure the integrity and authenticity of policies and activity logs.

At its core, GitOp relies on cryptographically signed metadata that lives alongside the source code within the repository. This metadata comes in two primary forms:

1. GitOp Policy: This defines the rules governing contributions. It starts with root metadata that establishes the trust anchors for the policy itself. This root metadata effectively says, "We trust the following keys to sign policy updates." Subsequent rule files specify fine-grained controls, such as:

• Which branches are protected (e.g., main, release).
• Which developers (identified by their cryptographic keys) are authorized to make changes to specific parts of the repository or to specific branches. For example, a rule might state, "The main branch can only be modified by developers 'Yellow' and 'Orange'."
• Thresholds for approvals: To mitigate key compromise, policies can require a minimum number of approvals for a change to be considered valid. For instance, a policy might dictate that two out of three designated maintainers must approve a pull request before it can be merged. These approvals are recorded as attestations, which are themselves signed metadata.

1. Reference State Log (RSL): This is GitOp's distributed activity tracking mechanism. The RSL is an append-only log that records significant events related to the repository's state, such as:

• Pushes: "Alice pushed the main branch to point to commit A."
• Branch creations/deletions.
• Policy updates.

Crucially, the RSL entries are also cryptographically signed and stored within the repository. This means that every developer who clones or pulls the repository automatically receives the complete, tamper-evident history of these security-relevant actions.

The distributed enforcement mechanism is where GitOp truly shines. When a developer pulls changes from the forge, their GitOp-enabled client performs local verification. It inspects the incoming RSL entries against the current GitOp policy embedded in their local repository. If an RSL entry indicates a push that violates the policy (e.g., an unauthorized developer pushed to a protected branch, or a change was merged without the required number of attestations), the client can detect this anomaly. The speaker emphasizes that "you only really need one honest client that's performing this continuous policy enforcement to detect something's gone wrong." This contrasts sharply with the centralized model, where a single compromised forge could allow malicious changes to propagate undetected.

GitOp also addresses the critical issue of key compromise. If an attacker gains access to a trusted developer's key, they could theoretically impersonate that developer. However, by leveraging policy thresholds, GitOp can require multiple independent signatures for critical actions. If a policy demands both "Yellow" and "Orange" to approve a change, and an attacker compromises "Yellow"'s key, they still cannot unilaterally approve the change. Any GitOp client pulling the repository would see the RSL entry for the push but note the absence of "Orange"'s required attestation, thus detecting the compromise. The Q&A session also clarified that key revocation is handled "in-band" through policy updates. If a key is compromised, the GitOp policy itself can be updated (following its own approval rules) to revoke trust in that key, and this change is recorded in the RSL, propagating to all clients.

The project itself, GitOp, is open source, released under the Apache 2.0 license, and is a sandbox project within the Linux Foundation's Open Source Security Foundation (OpenSSF). It is a collaborative effort between academics (like Aditya Yelgundhalli), industry professionals, and independent developers. A critical design goal was backwards compatibility with the existing Git ecosystem. GitOp is designed to be a "drop-in replacement" for a developer's machine, transparently recording and verifying data without forcing users to abandon existing Git tooling or forge platforms like GitHub or GitLab. Early pilot programs, such as one at Bloomberg, suggest minimal performance overhead, with per-push operations adding "0.something seconds" to the workflow.

Future technical directions for GitOp include developing features for cross-repository policy management to scale to large enterprises with thousands of repositories, extending the same distributed security properties to read access control, and improving the experience of working in mixed environments where some developers use GitOp clients and others do not.

Demo / Proof of Concept

▶ Watch: Formal threat model: compromised parties and centralization (5:40)

While the talk did not feature a live, interactive demonstration, Aditya Yelgundhalli confirmed that the GitOp project includes a comprehensive simulation and demo environment. This environment was submitted for artifact evaluation as part of the paper's review process and successfully earned all three badges for reproducibility, availability, and functional correctness.

The purpose of this simulation is to concretely illustrate all the properties and scenarios discussed in the talk, including:

• How GitOp policies are declared and stored within a repository.
• The mechanism by which the Reference State Log (RSL) records pushes and other critical events.
• The process of local client-side verification, where a GitOp client checks the RSL against the policy.
• Demonstrations of how policy violations (e.g., unauthorized pushes, missing approvals) are detected, even when the centralized forge is compromised or buggy.
• The efficacy of thresholds and attestations in mitigating key compromise scenarios.

Although the specifics of the demo's execution were not detailed in the presentation due to time constraints, its successful artifact evaluation underscores the practical feasibility and verifiable nature of GitOp's proposed distributed security model. It serves as a concrete proof of concept that the theoretical framework can be implemented and validated in a controlled environment.

Defensive Implications

▶ Watch: Posing the question: What if we take a distributed approach? (6:30)

The GitOp project introduces several critical defensive implications for organizations seeking to bolster their software supply chain security, moving beyond the inherent vulnerabilities of centralized trust.

Firstly, GitOp empowers organizations to implement a distributed and verifiable trust model for their source code. Instead of implicitly trusting that a forge (like GitHub or GitLab) has correctly enforced policies and maintained tamper-proof audit logs, defenders can now rely on the cryptographic integrity of policies and activity logs embedded directly within their repositories. This means that even if a forge's infrastructure is compromised, or an insider with privileged access attempts to manipulate logs or bypass policies, the malicious activity can be detected by any GitOp-enabled client. This shift fundamentally reduces the attack surface associated with a single, centralized point of failure.

Secondly, GitOp provides a robust mechanism for continuous, client-side policy enforcement. Every time a developer pulls changes, their GitOp client automatically verifies the integrity of the incoming code against the established policies and the Reference State Log (RSL). This creates a distributed network of "watchdogs," where the detection of a policy violation no longer depends on the vigilance of the forge but on the independent verification capabilities of each participating client. This significantly enhances the organization's ability to detect zero-day exploits targeting forge vulnerabilities or sophisticated insider threats that might otherwise go unnoticed.

Thirdly, the implementation of multi-party approval thresholds within GitOp policies offers a strong defense against key compromise. By requiring multiple cryptographic signatures for critical actions (e.g., merging code to a protected branch, updating security policies), organizations can mitigate the impact of a single compromised developer key. An attacker would need to compromise multiple keys to circumvent these controls, raising the bar significantly for successful attacks. This aligns with best practices in security, such as separation of duties and least privilege, by distributing authority and responsibility across multiple entities. Furthermore, the ability to revoke compromised keys through in-band policy updates ensures that the security posture can be dynamically adapted to emerging threats.

Finally, GitOp's backwards compatibility is a major advantage for defensive adoption. Organizations do not need to overhaul their entire development infrastructure or force developers to abandon familiar tools. GitOp can be integrated into existing Git workflows, enhancing security without disrupting productivity. The ongoing pilot at Bloomberg and the focus on scaling policies across thousands of repositories demonstrate its potential for practical enterprise deployment. By adopting GitOp, defenders gain an additional layer of verifiable security that complements existing forge-based controls, providing a critical safety net against a wide array of source code integrity threats in the software supply chain.

Key Takeaways

• Centralized Trust is a Critical Vulnerability: Relying solely on Git forges (e.g., GitHub, GitLab) for security policy enforcement and activity logging creates a single point of failure vulnerable to compromise, insider threats, or software bugs.
• GitOp Decentralizes Trust: The project proposes embedding cryptographically signed security policies and an append-only activity log (Reference State Log - RSL) directly within the Git repository, leveraging Git's distributed nature for security.
• Client-Side Verification is Key: GitOp-enabled clients independently verify incoming changes against the embedded policy and RSL, allowing for the detection of policy violations even if the centralized forge has been compromised or is buggy.
• Enhanced Resilience Against Key Compromise: Policies can enforce multi-party approval thresholds (e.g., requiring two signatures for a merge), significantly mitigating the risk associated with a single compromised developer key.
• Practical and Backwards Compatible: GitOp is an open-source project (Apache 2.0, OpenSSF) designed to integrate seamlessly with existing Git workflows and tooling, minimizing disruption for adoption in real-world environments.
• Proactive Supply Chain Security: By providing verifiable integrity for source code and its contribution process, GitOp offers a crucial defense mechanism against sophisticated software supply chain attacks targeting code integrity.

About the Speaker(s)

Aditya Sirish A Yelgundhalli is a PhD student at New York University, where his research is primarily focused on software supply chain security. He is a prominent figure in this domain, actively involved in practical work with organizations like the Linux Foundation. His contributions extend to significant projects within the software supply chain security ecosystem, including helping to lead the development and maintenance of the in-toto specification and, of course, GitOp, the subject of this talk. His work aims to translate academic research into practical, deployable solutions for securing the software development lifecycle.