Sub:jugation - Hijacking Cloud Identities by Recycling Namespaces in Global OIDC Issuers

Sub:jugation - Hijacking Cloud Identities by Recycling Namespaces in Global OIDC Issuers

Speakers: Tal Verer, Head of Research, Asec Security

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=aVLG_PATkmQ

Overview

In a revealing presentation at fwd:cloudsec, Tal Verer, Head of Research at Asec Security, unveiled "Sub:jugation," a novel class of vulnerability impacting nearly all major CI/CD providers. What initially appeared to be a straightforward issue—the potential for reclaiming deleted namespaces—escalated into a critical security concern upon deeper analysis from an attacker's perspective. The talk meticulously detailed how a fundamental design flaw in modern CI/CD authentication, specifically the interplay of global OIDC issuers and reclaimable namespaces, leads to the creation of "phantom cloud identities" that are ripe for exploitation.

Sub:jugation highlights a significant blind spot in current cloud security practices, demonstrating that even when adhering to modern OIDC-based authentication standards, subtle architectural weaknesses can introduce severe vulnerabilities. The research presented by Verer showcased concrete evidence of this vulnerability's prevalence across real-world environments, affecting major platforms like GitHub Actions, GitLab, and Terraform Cloud. This talk serves as a crucial wake-up call for organizations relying on CI/CD pipelines, urging a re-evaluation of how external identity providers are trusted and managed within their cloud ecosystems.

The core of the issue lies in the violation of the OpenID Connect specification's requirement for the sub (subject) claim to be locally unique and never reassigned. By combining this non-compliance with the widespread use of global OIDC issuers, attackers can passively identify and exploit abandoned cloud identities, gaining unauthorized access to sensitive cloud resources. Verer's research not only quantifies the scale of this problem but also offers actionable defensive strategies and outlines the proactive steps major providers are taking to mitigate this pervasive threat.

Background

▶ Watch: Introduction to Sub:jugation vulnerability in CI/CD OIDC (0:00)

Traditional CI/CD environments often relied on long-lived, hard-coded secret tokens or credentials embedded directly into workflow configurations. While simple to implement, this approach presented significant security risks, as these tokens, if compromised, could grant persistent and broad access to cloud environments. Modern CI/CD authentication has largely shifted towards more secure paradigms, primarily leveraging OpenID Connect (OIDC) and JSON Web Tokens (JWTs) to issue short-lived, scoped credentials.

Using GitHub Actions as a primary example, Verer illustrated the typical flow: a CI/CD workflow, granted a special id-token permission, requests a JWT from GitHub Actions' OIDC issuer. This issuer, defined by a specific global domain (e.g., token.actions.githubusercontent.com), returns a JWT containing various claims about the workflow. Crucially, the sub claim (subject claim) within this JWT identifies the entity requesting the token. Cloud providers, such as AWS, then verify this JWT by checking its signature against the OIDC issuer's public keys and evaluating conditions on its claims. A common and recommended best practice is to condition access on specific values within the sub claim, such as the GitHub organization and repository name (e.g., repo:my-org/my-repo:ref:refs/heads/main). This ensures that only workflows originating from a specific, trusted source can assume a particular cloud role, thereby granting short-lived, fine-grained access to cloud resources. This method is generally considered a highly secure approach compared to static credentials.

However, Verer's research uncovered two critical design choices that undermine this security model:

1. Global OIDC Issuers: CI/CD providers typically use a single, global OIDC issuer domain to issue tokens for all workflows across all their customers and tenants. This means that an OIDC issuer domain like token.actions.githubusercontent.com is universally trusted by any cloud account configured to accept tokens from GitHub Actions. There is no tenant-specific issuer.
2. Reclaimable Namespaces: The sub claim, which the cloud provider uses to establish trust, is constructed using elements that are part of a reclaimable namespace. For GitHub Actions, this often includes the GitHub organization or user name. If an organization or user deletes their account or renames it, that specific namespace (the original name) becomes available for anyone else to register. The RFC 7519 (JSON Web Token) specification explicitly states that "The sub (subject) claim identifies the principal that is the subject of the JWT. The claims in a JWT are normally statements about the subject. The subject value MUST be locally unique and never reassigned within the context of the issuer or be globally unique." This fundamental principle is violated by the reclaimable nature of these namespaces.

The combination of a universally trusted global issuer and a sub claim built upon reclaimable namespaces creates phantom cloud identities. These are cloud roles or identities that continue to trust a specific namespace (e.g., a GitHub organization name) even after that namespace has been deleted or abandoned by its original owner. If an attacker can reclaim such a namespace, they can then mint a JWT with the expected sub claim and assume the associated cloud identity, gaining unauthorized access.

Key Findings

▶ Watch: Identifying global OIDC issuers and reclaimable sub-claims (3:00)

Initially, the concept of reclaimable namespaces leading to potential identity takeover seemed "not really groundbreaking" to the Asec team. It was assumed that organizations would diligently offboard associated cloud identities when namespaces were deleted. However, adopting an attacker's mindset quickly revealed the true severity and pervasiveness of the issue, leading to the coining of the term "Sub:jugation."

Asec's research confirmed the existence of this vulnerability across all major CI/CD providers investigated:

• GitHub Actions: The sub claim often includes repository_owner, which is reclaimable. GitHub has a 3-month cooldown period before a deleted namespace can be re-registered.
• GitLab: The sub claim often includes project_path, which is reclaimable. GitLab has immediate namespace reclamation.
• Terraform Cloud (HashiCorp): The sub claim often includes organization_name, which is reclaimable. Terraform Cloud has a 1-month cooldown.

To quantify the real-world impact, Asec leveraged its unique vantage point, monitoring hundreds of thousands of cloud identities across major cloud providers (AWS and Azure) and numerous customer accounts. Their methodology was straightforward:

1. Extract all monitored cloud identities.
2. Filter for those configured to trust global OIDC issuers (e.g., GitHub Actions).
3. Extract the specific namespace (e.g., GitHub organization name) from the sub condition in the cloud trust policy.
4. Check if this namespace still actively exists on the CI/CD provider's platform (typically via a simple API request or GET to the public profile).

The results were striking:

• In AWS environments, 14% of the cloud identities configured to trust GitHub Actions had sub conditions referencing namespaces that no longer existed on GitHub.
• In Azure environments, this figure was even higher, at 24%.
• On average, 12 different cloud identities were found to trust the same phantom namespace. This indicates that when a group or organization deletes its namespace, multiple associated cloud roles or service accounts are often left behind, multiplying the potential impact of a single namespace reclamation.

These findings unequivocally demonstrated that the problem of phantom cloud identities is substantial and widespread, indicating a significant gap in offboarding practices.

Technical Deep Dive

▶ Watch: Thesis: Global issuers + reclaimable namespaces = phantom cloud identities (4:00)

An attacker aiming to exploit Sub:jugation requires two primary pieces of information:

1. The cloud identity identifier: This is the specific ARN (for AWS), Client ID (for Azure), or Service Account name (for GCP) that the vulnerable CI/CD workflow is configured to assume.
2. The namespace target: This is the specific reclaimable string within the sub claim that the cloud identity trusts (e.g., a GitHub organization or user name, GitLab project path, Terraform organization name) and the associated resource name (e.g., repository name).

The critical question is how an attacker, with zero prior access to an organization's internal systems, can acquire this information. The answer lies in the common practice of defining CI/CD workflows in code, which is frequently hosted in publicly accessible repositories.

Verer explained that attackers can search for "telltale signs" within public CI/CD workflow definitions. These include:

• The permission: id-token declaration, indicating the workflow's ability to request OIDC tokens.
• References to common actions or modules used for cloud authentication, such as aws-actions/configure-aws-credentials or similar for Azure/GCP, often with parameters like role-to-assume.

While best practices dictate using variables for sensitive identifiers like cloud ARNs, the research revealed that a significant percentage of workflows still hard-code these values directly. This makes them discoverable via code search.

Asec developed and utilized an "attacker's funnel" based on these principles:

1. Code Search: Employing tools like GitHub Code Search, attackers can query public repositories for specific patterns. Verer demonstrated using qualifiers like path:github/workflows combined with keywords (aws-actions/configure-aws-credentials, azure/login, google-github-actions/auth) to pinpoint relevant CI/CD workflows.
2. Extract Identifiers and Namespaces: From the identified workflows, the attacker extracts the cloud identity identifier (e.g., AWS ARN) and the reclaimable namespace (e.g., GitHub organization/user name) from the sub condition or the hard-coded role definition.
3. Monitor Namespace Deletion: The attacker then passively monitors these extracted namespaces for deletion. This can be done by periodically checking the availability of the namespace on the respective CI/CD platform (e.g., attempting to register the name or checking a public profile).
4. Collect Vulnerable Phantom Cloud Identities: Once a monitored namespace becomes available for reclamation, the attacker knows that any cloud identity associated with that namespace in their collected data set is now vulnerable. The attacker can then register the deleted namespace, create a malicious workflow that mints a JWT with the expected sub claim, and assume the targeted cloud identity.

Asec ran this funnel for over six months, yielding stark results:

• Approximately 28,000 distinct repositories matched at least one of their search patterns.
• About 25% of these were verified to use one of the major cloud providers (AWS, Azure, GCP).
• Crucially, over 30% of these workflows had their cloud identity identifiers (e.g., ARNs) hard-coded in cleartext within the workflow definition, making them directly discoverable by an external attacker. This percentage varied by platform, with Azure developers showing better adherence to best practices, while AWS and GCP had worse statistics.

The "mathematics of subjugation" further illuminated the scale of the problem:

• Asec observed a growth of approximately 9,000 new OIDC-enabled namespaces being created and monitored per month, translating to roughly 1 million per year.
• Within their monitored set, about 60 namespaces were deleted per month (roughly two per day).
• Combining these figures with the 30% cleartext exposure rate, Verer concluded that approximately **8 new phantom cloud identities become vulnerable to Sub:jugation with zero prior access *every month***.

This number represents a lower bound, as the research focused primarily on GitHub Actions and utilized only GitHub Code Search for reconnaissance. Other OIDC issuers exist, and alternative recon techniques (such as using the Internet Archive to find already deleted namespaces) could uncover many more vulnerable identities. The implication is profound: an attacker can passively collect a continuous stream of entry points into various cloud environments, including those belonging to large organizations, simply by monitoring public code and namespace availability.

Demo / Proof of Concept

▶ Watch: Subjugation found in GitHub, GitLab, and Terraform (6:00)

While the talk did not feature a live, interactive demonstration of an exploit, the comprehensive research methodology and the "attacker's funnel" described by Tal Verer effectively served as a proof of concept for how the Sub:jugation vulnerability can be exploited by an external attacker with no prior access. The step-by-step process of discovering vulnerable configurations through public code search, monitoring namespace deletions, and then being able to reclaim those namespaces to forge trusted OIDC tokens, conceptually demonstrates the attack chain.

The core of the "demo" was the presentation of real-world data and statistics gathered by Asec Security, which validated the feasibility and prevalence of the attack. By showing that 14-24% of monitored cloud identities were linked to non-existent namespaces and that approximately 8 new exploitable identities emerge monthly, Verer provided compelling evidence that the attack is not merely theoretical but a tangible and ongoing threat. The funnel’s output—a stream of exploitable phantom cloud identities—is the ultimate proof of concept for an attacker.

Defensive Implications

▶ Watch: Research question and methodology for real-world impact (6:50)

Asec Security responsibly disclosed their findings to GitHub, GitLab, and HashiCorp, who have all demonstrated cooperation in addressing the vulnerability.

• GitHub: Just prior to the presentation, GitHub rolled out a fix. New namespaces created in GitHub will now have a unique, immutable identifier attached to their sub claim. This means that even if the human-readable part of the namespace (e.g., organization name) is deleted and reclaimed, the underlying immutable identifier in the sub claim will be different, preventing the new owner from assuming the old cloud identity. Existing namespaces will retain their current sub claim format, requiring organizations to actively manage their trust policies. A blog post detailing this change was expected shortly after the talk.
• GitLab: GitLab released a mitigating fix around the time of the talk, which prevents the reclamation of namespaces that are currently in use. They are also tracking a CVE for the issue, particularly for local GitLab instances, and plan to implement a more comprehensive solution, likely akin to GitHub's immutable identifier approach.
• HashiCorp (Terraform Cloud): HashiCorp released an update to its language and warnings, advising users when changing or deleting their organization (the main namespace in Terraform Cloud). They are also planning a more robust solution, probably involving immutable identifiers, to fully address the issue.

For other CI/CD providers or existing configurations that remain vulnerable, Tal Verer provided clear recommendations for defenders:

1. Identify Global Issuer Trusts: Find all cloud identities (e.g., AWS IAM roles, Azure service principals, GCP service accounts) that trust global OIDC issuers. Look for issuer domains that are not unique to your tenant but are shared across all users of a particular CI/CD provider (e.g., token.actions.githubusercontent.com, oauth2.googleapis.com/token, gitlab.com).
2. Extract Namespaces: From the sub condition in these trust policies, extract the specific namespace components (e.g., organization name, repository owner, project path) that are used for identification.
3. Verify Namespace Activity: Test if these extracted namespaces are still active on the respective CI/CD platform. This usually involves a simple API request or a GET request to check for the existence of the organization, user, or project.
4. Remediate Inactive Namespaces: For any inactive or non-existent namespaces identified:

• Delete the associated cloud identity: This is the most secure and recommended action to revoke trust entirely.
• Reclaim the namespace (if possible and practical): If your organization still needs to use that specific namespace and it's available, you could reclaim it to prevent malicious actors from doing so. However, deleting the cloud identity is generally preferred to avoid any lingering trust issues.

Key Takeaways

• Sub:jugation is a new class of vulnerability arising from the combination of global OIDC issuers and reclaimable namespaces in CI/CD authentication, violating the JWT RFC's sub claim uniqueness requirement.
• Widespread Real-World Impact: Asec's research showed that 14-24% of monitored cloud identities in AWS and Azure environments were vulnerable to Sub:jugation, with an average of 12 identities trusting the same phantom namespace.
• Attacker's Advantage: Attackers can passively discover vulnerable cloud identities and their associated reclaimable namespaces by leveraging public code search (e.g., GitHub Code Search) for hard-coded credentials in CI/CD workflows.
• Continuous Exploitation Opportunity: The study estimated that approximately 8 new phantom cloud identities become exploitable with zero-access every month, providing a continuous stream of potential footholds into cloud environments.
• Provider Mitigations Underway: Major CI/CD providers (GitHub, GitLab, HashiCorp) are rolling out fixes, primarily by introducing immutable identifiers into the sub claim for new namespaces or blocking namespace reclamation.
• Proactive Defender Actions Needed: Organizations must audit their OIDC trust policies, identify cloud identities trusting global issuers, verify the activity of trusted namespaces, and promptly offboard or remediate any identified phantom cloud identities.

About the Speaker(s)

Tal Verer is the Head of Research at Asec Security, where he leads a team of talented researchers in uncovering novel security vulnerabilities. With a decade of experience in hacking and security research, Tal has a deep understanding of complex security challenges. Prior to his role at Asec, he contributed significantly to automotive security. Tal is passionate about all things related to hacking and security, and encourages engaging discussions on these topics. He gave a special shout-out to his colleague, Yav Raman, who was the original researcher who discovered the initial observation that kickstarted the Sub:jugation research.