Large-Scale Exposure Of Orphaned Commits On Major Git Platforms by Kumar Ashwin

Large-Scale Exposure Of Orphaned Commits On Major Git Platforms

Speakers: Ashwin Kumar, Lead Researcher and Consultant, Rent Labs

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=WBSX80KnGvg

Overview

Ashwin Kumar's Nullcon talk, "Large-Scale Exposure Of Orphaned Commits On Major Git Platforms," sheds critical light on a pervasive yet often overlooked security vulnerability: the persistence of sensitive data within dangling commits (also known as orphaned commits) on major Git hosting platforms like GitHub. The presentation meticulously details how seemingly "deleted" commits, intended by developers to remove mistakenly pushed secrets or sensitive information, do not truly vanish but remain accessible through specific API interactions. This research is significant because it challenges common misconceptions about Git's deletion mechanisms and exposes a vast attack surface for credential harvesting and intellectual property theft.

Kumar, leading research and consulting at Rent Labs, brings his expertise in web, cloud, and supply chain security to uncover the true scale of this problem. The talk is targeted at security professionals, developers, and organizations utilizing Git, urging them to reconsider their commit hygiene practices and understand the inherent persistence of data within version control systems. It matters immensely because it reveals that even after developers attempt to "clean" their repositories, critical secrets can linger, creating a permanent risk that traditional secret scanning tools often miss due to the commits being unreferenced by any active branch.

The core contribution of this work lies in its methodology for identifying and enumerating these hidden commits at scale. By leveraging GitHub's event API data and a sophisticated engineering pipeline, Kumar and his team demonstrated that hundreds of thousands of live secrets remain exposed. This large-scale study provides concrete evidence of the widespread nature of the issue, offering a stark reminder that the decentralized and immutable nature of Git, while powerful for version control, also presents unique challenges for data sanitization, especially when coupled with the operational specifics of remote Git platforms.

Background

▶ Watch: Introduction: Large-scale exposure of secrets on Git platforms (0:00)

Git, as a decentralized Version Control System (VCS), allows developers to collaborate on projects by maintaining local copies of repositories and tracking changes through commits. A commit represents a snapshot of the repository at a specific point in time, uniquely identified by a SHA-1 hash. Developers frequently interact with Git, but a common misconception arises when attempting to "delete" unwanted commits, particularly those containing sensitive data like API keys, credentials, or proprietary information.

The conventional wisdom for removing a recent commit from history often involves commands like git reset. Developers believe that by resetting to a previous state, the problematic commit, along with its sensitive contents, is permanently erased. However, Kumar illustrates that this is fundamentally untrue on remote Git platforms. While git reset removes the commit from the active branch history, the commit object itself, containing all its data, persists within the repository's internal object store. These unreferenced commits are what are termed dangling commits. They are still part of the repository's data but are not directly reachable from any branch or tag, making them invisible to standard Git commands that traverse history.

This problem is exacerbated by the fact that Git platforms, in their default configurations, do not automatically purge these dangling commits. When Rent Labs initially reached out to major Git platforms about this issue, they were informed that there were "no plans to fix it" and were directed to manual removal guidance. Tools like git filter-repo or BFG cleaner are often recommended for rewriting history to remove sensitive files. However, Kumar demonstrated that even after running git filter-repo locally, the dangling commit persisted on the remote platform, still accessible via its direct SHA-1 hash. This highlights a critical gap between local Git operations and the behavior of remote hosting services.

The reason this problem exists is inherent in Git's design. Git is designed to preserve history, making it robust against accidental data loss. When a commit is "removed" from a branch, it's merely the reference (the branch pointer) that moves, not the underlying commit object itself. On local machines, these dangling objects might eventually be garbage collected, but on remote platforms, they often remain indefinitely, waiting for a direct access attempt. This creates a persistent risk for any sensitive data that was ever committed, even if momentarily.

Key Findings

▶ Watch: Demonstration conclusion: Introducing dangling commits concept (3:30)

The research conducted by Ashwin Kumar and Rent Labs unveiled the significant scale and impact of dangling commits on major Git platforms. The core discovery was the ability to systematically enumerate and analyze these otherwise hidden commits, demonstrating their widespread exposure of sensitive data.

The study's methodology involved analyzing an extensive dataset comprising 5 years of Git commits data, totaling approximately 5.4 billion GitHub commits. From this massive corpus, after applying sophisticated filtering techniques, the team successfully identified 66 million dangling commits. This staggering number underscores the prevalence of such unreferenced data across the GitHub ecosystem.

Within these 66 million dangling commits, the researchers uncovered a critical security vulnerability: over 500,000 secrets. These secrets, ranging from API tokens to cloud credentials, were found to be present in various states of verification – some were actively working, others unverified, but all represented potential exposure. This finding directly contradicts the assumption that "deleted" commits are harmless.

Further insights from the large-scale analysis revealed interesting patterns:

• Peak Day for Dangling Commits: The study found that Friday accounted for approximately 22% of all dangling commits. This suggests a common scenario where developers push changes, potentially including sensitive data, towards the end of the week, only to realize their mistake and attempt to "reset" or "clean" their history before the weekend or at the start of the next week.
• Top Contributing Organizations: Several prominent companies were identified as having a high number of dangling commits associated with their non-free email domains. While this doesn't automatically imply secret exposure, it highlights organizations where developer practices might inadvertently lead to this issue. Notable mentions included Snake, N/A, QQ, Checkmarx, Red Hat, and 163.
• Common Commit Messages: The research also cataloged frequently occurring commit messages associated with dangling commits. "Release 4.0.0" was surprisingly popular, alongside phrases like "added chat GPT data," "saving release notes," and "test." These messages often indicate changes made quickly, potentially leading to errors, or experimental commits that were later discarded.
• Types of Exposed Secrets: The variety of secrets discovered was broad, encompassing critical credentials and tokens for various services. These included:
• Cloudflare API tokens
• ChatGPT tokens
• Docker Hub tokens
• GitHub tokens
• AWS credentials (including active, working credentials)
• Hugging Face tokens
• Etherscan keys
• GitLab tokens
• Google Cloud credentials

These key findings collectively demonstrate that dangling commits are not merely an academic curiosity but a significant, large-scale source of sensitive data exposure that demands immediate attention from both platforms and developers.

Technical Deep Dive

▶ Watch: Git platforms' response and failed git filter-repo attempt (5:15)

The core innovation of this research lies in its ability to programmatically identify and analyze dangling commits at an unprecedented scale, leveraging publicly available data streams in a novel way. Kumar explained that while dangling commits are not directly visible through typical Git history traversal, they can be inferred and enumerated by analyzing GitHub's event API.

The initial hypothesis for identifying potential dangling commits centered around force pushes and commits reported with a zero commit count in GitHub events. Force pushes often indicate that a developer has rewritten history, potentially discarding commits. However, this initial filter was too broad, as force pushes can occur for legitimate reasons, such as branch deletion, tag creation, or empty pushes.

To filter this noise effectively, the team developed a more precise mechanism:

1. Leveraging GitHub's Branch Commit Endpoint: The crucial step involved querying GitHub's Branch Commit Endpoint. This endpoint allows verification of whether a specific commit SHA belongs to any active branch within a repository. If a commit exists in the repository's object store but the Branch Commit Endpoint returns no associated branch, it definitively indicates a dangling commit. This method mirrors GitHub's internal logic for displaying the "abandon me" banner on unreferenced commit pages, which is loaded via JavaScript.

1. Historical Data Acquisition: A significant challenge was GitHub's policy of only storing 30 days of public event data. To perform a large-scale, 5-year study, Rent Labs utilized its proprietary historical dataset of GitHub commits, which provided the necessary depth of event data. This dataset was instrumental in overcoming the temporal limitations of GitHub's public API.

The engineering pipeline designed for this research was robust and distributed, capable of processing billions of events:

• Data Storage: The initial raw force push event data was stored in Elasticsearch.
• Distributed Processing: This data was then distributed across 20 different nodes to enable parallel processing and analysis.
• Two-Phase Scanning: Each node performed two primary tasks:
• Commit Scanning: This phase involved extracting metadata from each potential dangling commit, such as commit details, author emails, and timestamps.
• Secret Scanning: This critical phase utilized Rent Labs' internal regex patterns specifically designed to identify various types of sensitive information (e.g., API keys, cloud credentials, tokens).
• Output and Analysis: The results from both scanning phases were then formatted into JSON objects and stored in a separate Elasticsearch cluster for further aggregation, correlation, and analysis.
• Monitoring: The entire process was continuously monitored using Samard (presumably a custom or specific monitoring dashboard/tool) to ensure smooth operation and identify any issues.

This combination of GitHub event pattern recognition and at-scale engineering allowed the researchers to move beyond theoretical understanding to practical, verifiable enumeration of dangling commits and the secrets they contained. The technical sophistication of the approach was key to uncovering the true impact of this hidden vulnerability across the vast landscape of public Git repositories.

Demo / Proof of Concept

▶ Watch: Even git filter-repo fails; GitHub's manual removal process (6:00)

Ashwin Kumar provided a clear and compelling demonstration during the talk to illustrate how dangling commits are created and persist, even after attempts to "delete" them. The live demonstration effectively debunked common developer assumptions about Git's behavior.

The demonstration unfolded in several steps:

1. Initial Commit with Secret: Kumar started by creating a new Git repository and committing a file with an intentionally obvious name like "oh_oops," containing a simulated secret. This commit was then pushed to a public GitHub repository.
2. Observing the Commit on GitHub: On the GitHub web interface, two distinct commits were visible in the repository's history, including the "oh_oops" commit. Kumar then opened the specific page for this "oh_oops" commit in a separate browser tab, preserving its direct URL (which contains the commit's SHA-1 hash).
3. Attempting Deletion with git reset: Back in the local terminal, Kumar executed a git reset command, typically used to revert to a previous state and "undo" a commit. This action locally removed the "oh_oops" commit from the active branch history.
4. Verifying Local Deletion: Upon refreshing the GitHub repository's main page, the "oh_oops" commit was no longer visible in the branch history. This confirmed that git reset had successfully removed the commit from the active lineage, aligning with developers' expectations of deletion.
5. Revealing the Dangling Commit: The crucial part of the demo involved switching back to the separate browser tab where the "oh_oops" commit's direct URL was still open. Instead of a "404 Not Found" error or a blank page, GitHub displayed a prominent yellow banner with a message indicating that the commit was "abandoned." This visually confirmed that despite being removed from the branch history, the commit object itself, with its secret content, still existed on GitHub's servers and was directly accessible if one knew its SHA-1 hash.
6. Ineffectiveness of git filter-repo: Kumar further extended the demonstration by explaining that even after running tools like git filter-repo (which is designed to rewrite history and remove specific files permanently), the dangling commit would still persist on the remote platform, displaying the same yellow banner. This reinforced the point that local history rewriting tools do not guarantee the removal of data from the remote Git service's object store.

This practical demonstration served as a powerful proof of concept, making the abstract concept of dangling commits tangible and immediately understandable. It clearly showed that the "deleted" commits are merely unreferenced pointers, leaving the underlying data intact and potentially exposed.

Defensive Implications

▶ Watch: Methodology: Filtering noise using GitHub's Branch Commit Endpoint (8:25)

The findings presented by Ashwin Kumar highlight critical areas where organizations and individual developers must strengthen their security posture to mitigate the risks posed by dangling commits and exposed secrets. The defensive implications are multi-layered, addressing platform control, automated detection, and crucial human factors.

1. Self-Hosted Git Management Systems:

• Benefit: Hosting a private Git management system (e.g., self-hosted GitLab, Gitea) provides organizations with full control over their Git platforms, including commit policies, data retention, and garbage collection schedules. This allows for direct intervention and custom configurations to manage dangling commits more effectively.
• Challenges: This approach is costly and complex to scale, requiring significant infrastructure, maintenance, and expertise. More importantly, it does not address the problem of personal public repositories on platforms like GitHub, where employees might inadvertently leak company secrets outside the organizational perimeter.

1. Automated Detection and Monitoring:

• Proactive Scanning: Organizations should implement automated secret scanning solutions that can detect sensitive information in all Git data, including dangling commits. The research methodology presented by Rent Labs (analyzing GitHub event APIs and branch commit endpoints) provides a blueprint for developing such advanced detection logic.
• Immediate Rotation: Upon detection of a leaked secret, an immediate and automated secret rotation process is paramount.
• Platform Notifications: While some cloud providers (e.g., AWS) offer services to notify users if their credentials are found in public Git repositories, Kumar noted a critical gap: GitHub does not provide similar notifications for leaked GitHub tokens. This emphasizes the need for organizations to implement their own comprehensive scanning, rather than relying solely on platform-provided alerts.

1. Developer Awareness and Training (The Human Factor):

• Education: This is arguably the most crucial defensive measure. Developers must be educated about the true nature of Git's history and the persistence of data, especially regarding git reset, git filter-repo, and other "deletion" mechanisms. They need to understand that once sensitive data is pushed to a remote public repository, it is exceptionally difficult, if not impossible, to fully purge it.
• Secure Commit Hygiene: Promote and enforce secure commit hygiene practices. This includes:
• Never committing secrets directly into code or configuration files.
• Using environment variables or dedicated secret management solutions (e.g., HashiCorp Vault, AWS Secrets Manager) for sensitive data.
• Being vigilant about what is staged and committed.
• Enforce Git Hooks and Rules:
• Pre-commit hooks: Implement client-side or server-side pre-commit hooks that scan for common secret patterns before a commit is allowed. While client-side hooks can be bypassed, server-side pre-receive hooks (especially on self-hosted instances) are more robust.
• Signed commits: Encourage or enforce signed commits to verify author identity and prevent history tampering.
• Branch protection rules: Utilize branch protection rules on platforms like GitHub to prevent force pushes to critical branches and enforce requirements like pull request reviews.

The core challenge remains the "edge cases" where employees commit company secrets to their personal, public repositories, which are outside the organization's direct control. While GitHub provides secret scanning for organization repositories, it often doesn't extend to personal accounts, creating a blind spot. Therefore, a holistic defense strategy must combine technical controls with continuous developer education and a culture of security awareness.

Key Takeaways

• Git's "Deletion" is Misleading: Commands like git reset and tools like git filter-repo do not permanently remove commits from remote Git platforms; they merely make them unreachable from active branches, leaving them as dangling commits.
• Dangling Commits are Discoverable and Dangerous: These hidden commits are not truly gone and can be enumerated and accessed by leveraging Git platform event APIs and direct commit SHA-1 hashes, presenting a significant attack surface for exposed secrets.
• Widespread Exposure: A large-scale study revealed over 66 million dangling commits and 500,000+ exposed secrets (including active AWS, GitHub, and other tokens) on GitHub alone, indicating a pervasive and unaddressed security risk.
• Friday is Peak Leak Day: A notable finding was that 22% of dangling commits occur on Fridays, suggesting a pattern of rushed work or last-minute "cleanups" that inadvertently leave sensitive data exposed.
• Multi-faceted Defense is Essential: Effective mitigation requires a combination of technical controls (advanced automated secret scanning, robust Git management systems), strict process enforcement (pre-commit hooks, branch protection), and continuous developer education on secure commit hygiene.
• The Human Factor is Critical: Lack of developer awareness and the ease of accidentally pushing secrets to personal public repositories remain major contributors to this problem, often falling outside organizational security controls.

About the Speaker(s)

Ashwin Kumar is a prominent figure in the cybersecurity research community, currently serving as a Lead Researcher and Consultant at Rent Labs. His professional focus spans critical areas including web security, cloud security, and supply chain security, where he contributes to both research and practical consulting engagements. Kumar is a recognized speaker and trainer, having shared his insights at various prestigious cybersecurity conferences globally, including Black Hat, Zeron, and now Nullcon. His work, as demonstrated by this talk, often delves into uncovering large-scale vulnerabilities and providing actionable intelligence for improving security postures across the industry.