Self-Hosted GitHub CI/CD Runners: Continuous Integration, Continuous Destruction

Self-Hosted GitHub CI/CD Runners: Continuous Integration, Continuous Destruction

Speakers: Adnan Khan, Security Engineer, Researcher, and Bug Bounty Hunter; John Stawinski, Red Team Security Engineer and CI/CD Researcher

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=GJhab1qXNig

Overview

This talk, "Self-Hosted GitHub CI/CD Runners: Continuous Integration, Continuous Destruction," delivered by Adnan Khan and John Stawinski at Black Hat USA, exposes a critical and systemic vulnerability class impacting organizations globally: the insecure configuration of self-hosted GitHub Actions runners. The speakers highlight a pervasive lack of awareness regarding the security implications of these CI/CD agents, which, when misconfigured, can serve as a direct conduit for attackers to launch devastating supply chain attacks. They assert that public GitHub repositories, often overlooked as a potential attack surface, are becoming a primary entry point for such compromises.

Khan and Stawinski substantiate their claims with compelling real-world examples, including a critical vulnerability they identified in a major domestic chip manufacturer, which granted them enterprise admin privileges over the company's GitHub Enterprise Cloud tenant. This level of access could have led to the deletion of 120,000 repositories or the exposure of highly sensitive intellectual property. Their research also uncovered a critical flaw in GitHub's own actions runner images repository, underscoring the widespread nature and severity of these issues across the GitHub ecosystem. The talk aims to educate the security community on these easily exploitable vulnerabilities and their potential to inflict widespread damage, including altering the economic balance of power.

The importance of this research cannot be overstated in an era increasingly defined by sophisticated supply chain attacks, such as SolarWinds. By demonstrating how simple misconfigurations in GitHub Actions workflows can be leveraged by external attackers via seemingly innocuous pull requests, the speakers reveal a significant blind spot in many organizations' security postures. Their work, which evolved from prior Red Team engagements and the development of the open-source tool Gato for self-hosted runner discovery, provides a stark warning and actionable insights for defenders grappling with the complexities of modern CI/CD security.

Background

▶ Watch: Opening story: Hacking a chip manufacturer via GitHub Actions (0:00)

The landscape of cybersecurity has dramatically shifted, with supply chain attacks emerging as one of the most potent and insidious threats. While the industry has become increasingly aware of supply chain risks following high-profile incidents like SolarWinds, attackers are continually evolving their tactics, identifying new and often overlooked entry points. Adnan Khan and John Stawinski contend that a significant, yet under-recognized, frontier for these attacks lies within public GitHub repositories and their associated CI/CD infrastructure, specifically self-hosted GitHub Actions runners.

The problem stems from a fundamental lack of awareness and understanding regarding the security posture required for these agents. Organizations, particularly those at the technological forefront, often deploy self-hosted runners to gain greater control over their build environments, utilize specialized hardware, or run workflows in private networks. However, this flexibility comes with increased responsibility for security, a responsibility frequently unmet. When a self-hosted runner is exposed to a public repository and configured to automatically execute workflows from external pull requests, it creates a critical vulnerability. An attacker can fork the repository, inject malicious code into a workflow file, and submit a pull request. If the self-hosted runner is configured to process this PR without adequate safeguards, the attacker's malicious code is executed on the runner's host, often within the organization's network perimeter.

The speakers' journey into this vulnerability class began in August 2022 during a Red Team engagement. They gained initial network access to a client by leveraging a GitHub token and subsequently exploiting a self-hosted runner to establish persistence within the network. This initial success catalyzed further research, leading to a talk titled "Phantom of the Pipeline" at ShmooCon. Concurrently, Adnan Khan, Mason Davis, and Matt Chaikowski developed Gato, an open-source tool designed to automate the discovery of self-hosted runners, thereby enabling security researchers and Red Teams to identify these vulnerable assets more efficiently. John Stawinski later joined the team, further deepening the research into GitHub Actions post-exploitation techniques. A pivotal discovery, attributed to Adnan, was the realization that even seemingly benign actions, such as fixing a typo in a public repository, could be weaponized. By modifying the workflow file in a forked repository and creating a pull request, an attacker could gain persistent access to the attached self-hosted runner, setting the stage for more advanced attacks. This particular insight led to a critical vulnerability disclosure to GitHub itself, concerning their actions runner images repository, highlighting the systemic nature of the issue.

Key Findings

▶ Watch: Systemic vulnerability class enabling supply chain attacks (1:07)

The central and most alarming finding presented by Khan and Stawinski is the widespread systemic lack of awareness around the security of self-hosted CI/CD agents, particularly in the context of GitHub Actions. This oversight is exposing even the world's most technologically advanced organizations to critical supply chain attacks. Their research demonstrates that misconfigured self-hosted runners, when associated with public GitHub repositories, can be easily exploited to achieve devastating outcomes.

The core mechanism of exploitation revolves around an attacker's ability to submit a pull request (PR) from a forked repository containing a modified workflow file. If the target repository's self-hosted runner is configured to automatically execute workflows triggered by pull_request events from external contributors without sufficient validation or approval, the malicious code within the PR's workflow file will execute on the runner's host machine. This grants the attacker arbitrary code execution within the organization's environment where the runner resides.

The severity of these exploits is profound, as illustrated by several key impacts:

1. Enterprise Admin Privileges: In one disclosed case, the speakers were able to escalate to enterprise admin privileges over a major chip manufacturer's GitHub Enterprise Cloud tenant. This level of access confers complete control over the organization's GitHub presence, including the ability to:

• Access and exfiltrate highly sensitive intellectual property (IP).
• Make any private repository public, leading to massive data breaches.
• Delete entire GitHub organizations, resulting in an immediate and catastrophic loss of over 120,000 repositories.

1. Supply Chain Compromise: The ability to execute arbitrary code on a self-hosted runner often means gaining a foothold within the corporate network that hosts the runner. This can lead to lateral movement, privilege escalation, and ultimately, a full-scale supply chain attack akin to SolarWinds, affecting not only the target organization but also its customers and partners.
2. Widespread Vulnerability: The research revealed that this vulnerability class is not isolated to a few specific organizations. The speakers explicitly stated that they used this "systemic vulnerability class to hack a lot of organizations," often achieving sufficient access to cause full-scale supply chain attacks. Furthermore, their successful critical vulnerability disclosure to the GitHub actions runner images repository indicates that even GitHub's own infrastructure or nearly "every GitHub customer that uses hosted runner images" could have been susceptible to similar attacks.
3. Ease of Exploitation: A critical finding is the relative ease with which these attacks can be executed. The speakers repeatedly emphasize that "some of these attacks are very easy to pull off," requiring little more than a GitHub account and knowledge of how to manipulate workflow files in a pull request scenario. This low barrier to entry makes the threat particularly dangerous, as it expands the pool of potential attackers beyond highly sophisticated Advanced Persistent Threat (APT) groups.

In essence, the research uncovers a gaping security hole where the convenience of automated CI/CD workflows, combined with a lack of robust security hygiene for self-hosted infrastructure, creates a direct and easily exploitable pathway for attackers to compromise critical corporate assets and initiate far-reaching supply chain disruptions.

Technical Deep Dive

▶ Watch: Systemic lack of CI/CD agent security awareness (2:00)

The technical foundation of these vulnerabilities lies in the interplay between GitHub Actions, self-hosted runners, and the inherent trust model (or lack thereof) when processing contributions from external users in public repositories.

GitHub Actions is a CI/CD platform that allows developers to automate software workflows directly within their GitHub repositories. These workflows are defined in YAML files (typically located in the .github/workflows/ directory) and can be triggered by various events, such as pushes, issues, or, crucially, pull requests. When a workflow is triggered, it runs a series of jobs, each executed on a designated runner.

Runners can be either GitHub-hosted runners (managed by GitHub) or self-hosted runners. While GitHub-hosted runners offer convenience and isolation, self-hosted runners provide organizations with greater control over the hardware, operating system, and software environment. They are typically machines (physical or virtual) within an organization's private network, configured with the GitHub Actions runner application to listen for and execute jobs.

The core vulnerability mechanism exploited by Khan and Stawinski leverages a common misconfiguration:

1. Public Repository Exposure: An organization hosts a public repository on GitHub.
2. Self-Hosted Runner Assignment: This public repository (or the organization/enterprise it belongs to) is configured to use a self-hosted runner for its CI/CD workflows. The critical aspect here is that the runner is often situated within the organization's trusted network and may have elevated permissions or access to internal resources.
3. Untrusted Pull Request Execution: The repository's GitHub Actions workflow is set up to trigger on pull_request events, allowing external contributors to submit code for review. Crucially, the workflow is configured to execute jobs from the forked repository's code, rather than just the base branch's code, or it lacks sufficient checks to prevent untrusted code execution.
4. Malicious Workflow Injection: An attacker, with a standard GitHub account, forks the target public repository. In their forked copy, they modify an existing workflow file (e.g., build.yml or test.yml) or create a new one. Within this workflow, they embed malicious commands (e.g., shell scripts, Python scripts) designed to exfiltrate data, establish persistence (e.g., reverse shell, SSH key deployment), or perform reconnaissance on the runner's host.
5. Pull Request Submission: The attacker then creates a pull request from their malicious fork to the original public repository.
6. Arbitrary Code Execution: When this pull request is opened, the GitHub Actions service, following the repository's configuration, triggers the workflow. Because a self-hosted runner is assigned and configured to process these PRs, the malicious workflow code from the attacker's fork is executed on the organization's self-hosted runner. This grants the attacker arbitrary code execution on the underlying machine hosting the runner.

The GitHub Actions GITHUB_TOKEN is also a critical component. By default, each workflow run is provided a unique GITHUB_TOKEN with a limited set of permissions. While GitHub has implemented measures to restrict the GITHUB_TOKEN's permissions for pull_request events from forks (setting it to read-only), this does not prevent arbitrary code execution on the runner itself. Once code execution is achieved on the self-hosted runner, the attacker can bypass GITHUB_TOKEN restrictions by interacting directly with the host operating system, installing persistent backdoors, or leveraging other credentials found on the runner.

The speakers also highlighted their tool, Gato, which was developed to automate the discovery of self-hosted runners. While specific technical details of Gato's operation were not provided in the transcript, such tools typically scan GitHub for public repositories, analyze their workflow files for indications of self-hosted runner usage (e.g., runs-on: self-hosted), and potentially identify misconfigurations that would allow the described exploitation.

The vulnerability found in the GitHub actions runner images repository underscores the broad applicability of this attack vector. If GitHub's own infrastructure or the images used by their hosted runners could be compromised via a similar pull request mechanism, it implies a fundamental design or configuration challenge that affects a vast number of users and organizations relying on GitHub Actions. This particular finding was accepted as a critical report, indicating its significant security impact.

Demo / Proof of Concept

▶ Watch: Origin of research: Red Team engagement and Gato tool (2:35)

While the talk did not include a live, step-by-step demonstration in the provided transcript, the speakers opened with a compelling and detailed proof of concept in the form of a real-world exploit they conducted against "one of the largest domestic chip manufacturers in the United States." This initial story serves as a powerful illustration of the attack's feasibility and devastating impact.

The scenario unfolded as follows:

1. Vulnerability Identification: The speakers identified a GitHub Actions misconfiguration within a public repository owned by the target chip manufacturer. This misconfiguration allowed external pull requests to trigger workflows on a self-hosted runner.
2. Exploitation via Pull Request: Anyone with a GitHub account could have exploited this. The speakers, acting as ethical hackers, created a pull request to the vulnerable public repository. Although not explicitly stated, the implication is that this pull request contained a modified workflow file designed to execute malicious commands on the self-hosted runner.
3. Privilege Escalation: The successful execution of their crafted workflow on the self-hosted runner granted them enterprise admin privileges over the company's GitHub Enterprise Cloud tenant. This is a critical escalation, moving beyond just access to the runner's host to full administrative control over the company's entire GitHub presence.
4. Access to Sensitive Intellectual Property: With enterprise admin privileges, the attackers gained access to "some of that company's most sensitive intellectual property." This highlights the direct link between CI/CD misconfigurations and the compromise of core business assets.
5. Catastrophic Potential: The speakers emphasized the extent of their control, stating they "had the privileges to make every single repository public or even delete their GitHub organizations entirely." Deleting the organization would have triggered "an immediate loss of over 120,000 repositories," demonstrating the potential for massive data destruction and operational paralysis.
6. Responsible Disclosure: Crucially, this was not an attack by an APT group but a responsible disclosure by the speakers. They notified the company, which subsequently remediated the vulnerability.

This detailed case study effectively serves as a powerful proof of concept, demonstrating how a seemingly minor misconfiguration in a public repository's CI/CD setup can be leveraged by an external entity to achieve complete control over an organization's GitHub environment, with far-reaching implications for data integrity, intellectual property, and business continuity. The speakers also mentioned conducting "three case studies covering exploits that we conducted through our research," reinforcing that this was not an isolated incident but a repeatable attack pattern.

Defensive Implications

▶ Watch: Critical GitHub Actions runner image vulnerability disclosure (3:55)

The findings presented by Adnan Khan and John Stawinski demand immediate attention from organizations utilizing GitHub Actions, especially those with self-hosted runners. The defensive implications are broad, requiring a multi-faceted approach to secure CI/CD pipelines and prevent the devastating supply chain attacks demonstrated.

1. Increase Awareness and Education: The primary defensive measure is to address the "systemic lack of awareness." Development, DevOps, and security teams must be educated on the unique security risks posed by self-hosted runners, particularly when interacting with public repositories. This includes understanding how workflow configurations can create attack vectors.
2. Strict Review of Workflow Configurations for Public Repositories:

• External PRs and Self-Hosted Runners: Organizations must critically re-evaluate any GitHub Actions workflow in a public repository that triggers on pull_request events and uses a self-hosted runner. This is the most direct path to exploitation.
• Required Approval: Implement required approval for workflows originating from external forks before they are allowed to execute on self-hosted runners. GitHub offers features like pull_request_target combined with explicit approval steps, or workflow_run events that can be used to scan PRs before triggering a build on a sensitive runner.
• Isolated Environments: If external PRs must be built on self-hosted infrastructure, ensure these runners are highly isolated, ephemeral, and run in a severely restricted network segment with no access to sensitive internal resources or credentials.

1. Principle of Least Privilege:

• Runner Permissions: Self-hosted runners should operate with the absolute minimum necessary permissions. Their associated GitHub Apps or personal access tokens should only have the scope required for their specific tasks.
• Host Environment: The host machine running the runner agent should be hardened, with restricted network access, limited user privileges for the runner process, and minimal installed software.
• Ephemeral Runners: Whenever possible, self-hosted runners should be ephemeral, meaning they are provisioned for a single job and then destroyed. This prevents attackers from establishing persistence.

1. Credential Management:

• Secrets Storage: Avoid storing sensitive credentials directly on runner hosts. Utilize secure secret management solutions (e.g., GitHub Secrets, Azure Key Vault, AWS Secrets Manager) and ensure secrets are injected at runtime, not hardcoded.
• GITHUB_TOKEN Scrutiny: While GitHub limits GITHUB_TOKEN permissions for forks, administrators should still be aware of its default permissions and explicitly restrict them further if possible.

1. Continuous Monitoring and Logging:

• Runner Activity: Implement robust logging and monitoring for all self-hosted runner activity. Look for unusual commands, unexpected network connections, file modifications, or attempts to access unauthorized resources.
• GitHub Audit Logs: Regularly review GitHub Enterprise Cloud/Server audit logs for suspicious activities, such as changes to runner configurations, organization settings, or repository visibility.

1. Utilize Security Tools:

• Static Application Security Testing (SAST): Integrate SAST tools into CI/CD pipelines to scan workflow files and repository code for common misconfigurations and vulnerabilities.
• Runner Discovery Tools: Tools like Gato (which the speakers developed) can be used defensively to discover and inventory self-hosted runners within an organization's GitHub ecosystem, helping identify potential attack surfaces.

1. Regular Security Audits and Penetration Testing: Conduct periodic security audits and Red Team exercises specifically targeting CI/CD pipelines and self-hosted runners. This will help uncover misconfigurations and vulnerabilities before attackers do.
2. Consider GitHub-Hosted Runners for Public Repositories: For public repositories, prefer GitHub-hosted runners for external contributions, as they offer a higher degree of isolation and security management by GitHub, reducing the burden on the organization.

By adopting these defensive strategies, organizations can significantly mitigate the risk of self-hosted GitHub CI/CD runners becoming a "continuous destruction" pathway for supply chain attacks.

Key Takeaways

• Self-hosted GitHub CI/CD runners represent a critical and often overlooked attack surface, enabling sophisticated supply chain attacks when misconfigured.
• Public GitHub repositories are a primary entry point for attackers to exploit these misconfigurations, leveraging seemingly benign actions like pull requests from forks.
• An attacker can achieve arbitrary code execution on an organization's self-hosted runner by injecting malicious commands into a workflow file within a forked repository and submitting a pull request.
• The impact of such exploits can be catastrophic, leading to enterprise admin privileges, access to sensitive intellectual property, and the potential for massive data destruction (e.g., deleting 120,000 repositories).
• There is a systemic lack of awareness within even advanced technological organizations regarding the security implications of self-hosted CI/CD agents, making them highly vulnerable.
• Defenders must implement strict workflow reviews, least privilege principles, ephemeral runners, and robust monitoring to secure their CI/CD pipelines against these threats. Tools like Gato can aid in discovery.

About the Speaker(s)

Adnan Khan is a security engineer by profession, who also dedicates his time to security research and bug bounty hunting. He resides in Baltimore, Maryland, and is a key contributor to understanding and exposing vulnerabilities in CI/CD systems. His work includes the co-development of the Gato tool and identifying critical flaws in the GitHub Actions ecosystem.

John Stawinski serves as a Red Team security engineer and specializes in CI/CD research. In his past, he was a collegiate wrestler. He works remotely, which allows him flexibility in his professional pursuits. John joined the research efforts on GitHub Actions vulnerabilities, contributing to the discovery and exploitation techniques discussed in the talk.