Attacking & Defending Supply Chains. How we got Admin in your Cloud,...

Attacking & Defending Supply Chains. How we got Admin in your Cloud,...

Speakers: Mike Ruth

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=QN3fLLSfAgs

Overview

This talk, presented by Mike Ruth, a Security Engineer at Rippling, delves into the critical and often overlooked security vulnerabilities within modern software supply chains, specifically focusing on Continuous Integration/Continuous Deployment (CI/CD) pipelines. Titled "Attacking & Defending Supply Chains. How we got Admin in your Cloud,..." it serves as an extension of a previous talk given in 2022 that primarily focused on Terraform. This iteration expands the scope to include other prevalent CI/CD tools such as GitHub Actions and Buildkite, demonstrating how misconfigurations and inherent design choices can lead to administrative access in cloud environments.

The presentation highlights a fundamental disconnect between developer experience and security best practices, where the ease of creating and managing CI/CD resources can inadvertently expose sensitive credentials and allow for unauthorized code execution. Ruth meticulously details several attack vectors, including secret exfiltration and pre-merge remote code execution, across these popular platforms. More importantly, he proposes practical defensive strategies and discusses recent advancements from vendors aimed at mitigating these risks, emphasizing the need for practitioners to innovate in securing these complex systems.

The talk is highly relevant for security engineers, DevOps professionals, and cloud architects who are responsible for designing, implementing, and securing CI/CD pipelines. It underscores the pervasive nature of supply chain attacks, not just from upstream open-source dependencies, but from the very tools and processes organizations use to deliver software from source to production. By illustrating how an attacker can leverage common CI/CD patterns to gain administrative access, Ruth provides actionable insights for strengthening an organization's security posture against sophisticated insider threats and compromised developer accounts.

Background

▶ Watch: Introduction: Attacking & Defending Software Supply Chains (0:00)

The proliferation of CI/CD tools and software supply chains is a direct response to the exponential growth of engineering workforces. As companies scale from tens to hundreds or thousands of employees, the traditional model of engineers directly managing production credentials and deploying artifacts becomes untenable. This leads to "teams stomping over one another" and an inability to maintain productivity. Automation through CI/CD tools becomes a necessity, offering significant benefits in terms of efficiency and consistency.

From a security perspective, CI/CD pipelines offer the advantage of centralizing credentials, removing them from individual engineering endpoints, and allowing engineers to interact with familiar tools like GitHub for version control. However, the scope of this talk deviates from the typical focus on upstream open-source contributions and libraries. Instead, it zeroes in on the most common services used within an organization's CI/CD pipeline—GitHub, GitHub Actions, Buildkite, and Terraform—and how software moves from source code to production environments using these tools.

Ruth outlines the reference architectures for these tools:

• GitHub: Emphasizes repositories as logical boundaries for source code and organizations as supersets for scalable access control. The ability for engineers to manually create repositories and configure settings (collaborators, webhooks, protection rules) is highlighted as a common practice for developer experience, but also a potential vulnerability.
• GitHub Actions: Natively integrated with GitHub, it uses workflows defined in .github/workflows/*.yml files. Each job runs in its own GitHub worker with a provisioned GitHub token for identity. This token grants access to repository and organization secrets. More elegantly, it supports OpenID Connect (OIDC) for federating identity to cloud environments. Workflows are triggered by events like on push or on pull_request.
• Buildkite: Similar to GitHub Actions, it uses pipelines configured via a YAML step file in a reserved path (.buildkite/). It requires a webhook for integration and uses an agent as its identity, which can access secrets, IAM roles, and service accounts.
• Terraform: Primarily a Continuous Deployment (CD) tool, it manages infrastructure as code. Workspaces maintain state and manage resources, often mapped one-to-one with a repository or subdirectory (e.g., dev, prod). State files track the last known good configuration, enabling diffs during plan operations for code review. Providers are plugins that interact with cloud APIs (AWS, GCP, Azure, Kubernetes). Permissions govern how users and upstream integrations interact with Terraform.

The typical user flow involves:

1. A user submits a Pull Request (PR) to a repository.
2. Configuration files (e.g., .buildkite.yml, .github/workflows/*.yml) trigger a job or webhook.
3. A worker (GitHub Actions worker, Buildkite agent) fetches and runs the job.
4. The worker's identity fetches necessary secrets to perform pre-plan actions (linting, testing).
5. Code review occurs, and upon merge, right-based actions (building artifacts, pushing to repositories) are performed.
6. For Terraform, a PR triggers a plan (dry run), followed by code review and eventual resource management in cloud environments.

Three key assumptions underpin the problem statement:

1. Most organizations allow engineers to push PRs or code to any repository.
2. Branching strategies are largely flat, with one protected branch (e.g., main) and everything else.
3. Service owners are predominantly responsible for managing their own workflows and pipelines, rather than relying on a centralized infrastructure team for "white glove services."

These assumptions lead to the core problem statement: "Engineers have access to create repositories, PRs, CI/CD resources unilaterally in our organizations and as such they can create and Trigger dry runs for new and existing workflows, pipelines and plans for all of our environments." This seemingly innocuous capability, when combined with common configurations, opens the door to significant security vulnerabilities.

Key Findings

▶ Watch: Problem Statement: Unilateral Engineer Access to CI/CD (12:00)

Mike Ruth's presentation uncovers several critical vulnerabilities stemming from the common practices and default configurations of CI/CD tools, particularly when engineers have broad permissions to create and modify resources.

1. Default Exposure of Organization Secrets (GitHub Actions):

• When a new repository is created, it automatically gains access to all GitHub organization secrets that are configured to be accessible by all repositories.
• This means a malicious actor or compromised account can create a new repository, define a GitHub Actions workflow, and immediately access these sensitive organization-wide secrets.
• Secret Exfiltration Technique: While GitHub attempts to mask secrets in logs, a memory dump technique (e.g., using cat /proc/self/environ or similar methods to access environment variables, then base64 encoding them) can bypass this masking, revealing the secret's value and the GitHub token of the worker.

1. Arbitrary Agent Definition and Role Assumption (Buildkite):

• Buildkite pipelines, by default, allow arbitrary definition of the agent that executes the pipeline. This means a pipeline in one repository can specify an agent typically associated with another, more privileged repository.
• By leveraging this, an attacker can assume the IAM roles or access the secrets associated with that privileged agent, even if the current repository should not have such access.
• The demonstration showed assuming a PR agent role and performing a get-caller-identity to prove the unauthorized role assumption.

1. Pre-Merge Remote Code Execution (RCE) - The Core Problem:

• GitHub's documentation states that if an individual has write access to a repository, they effectively have read access to all secrets within that repository. This fundamentally undermines the intended security boundaries of CI/CD tools.
• This finding implies that an attacker doesn't even need to create new repositories or pipelines. If they can push a PR to any existing repository, they can modify or create new workflow files within their feature branch.
• Crucially, these modified configuration files are honored and executed during the PR process (e.g., during pre-merge checks or dry runs). This allows an attacker to run arbitrary code with the permissions of the CI/CD worker's identity, effectively achieving RCE without code review or merging.

1. New Attack Use Cases Enabled by Pre-Merge RCE:

• OIDC-based Federation Exploitation: If a repository uses OIDC to federate identity to cloud IAM roles (e.g., AWS IAM roles), a malicious workflow in a PR can assume these roles. The attacker can craft a workflow to generate a JSON Web Token (JWT), exchange it for cloud credentials, and gain access to cloud resources. This is particularly dangerous as these federated roles often have significant permissions.
• Terraform Apply-on-Plan Bypass: Terraform, especially when using unsafe built-in functions or providers (e.g., null_resource with local-exec or external data sources), can be manipulated. An attacker can embed commands within a terraform plan that escape the plan context and execute arbitrary code on the underlying file system. This allows them to perform a terraform apply (creating or modifying cloud resources) during the plan phase, completely bypassing code review and merge requirements. Since Terraform workers often have administrative privileges to manage cloud resources, this can lead to full administrative access to the cloud environment.

In summary, the key findings reveal that common CI/CD configurations, combined with broad developer permissions, create a fertile ground for insider threats or compromised accounts to exfiltrate secrets, impersonate privileged agents, and execute arbitrary code, ultimately leading to administrative access in cloud environments without proper oversight or approval.

Technical Deep Dive

▶ Watch: Terraform Apply-on-Plan Bypass Demonstration (22:00)

The technical deep dive illustrates the specific mechanisms through which the identified vulnerabilities can be exploited across GitHub Actions, Buildkite, and Terraform.

GitHub Actions Secret Exfiltration:

The core vulnerability here lies in the default accessibility of GitHub organization secrets. When an organization defines a secret and sets its visibility to "Accessible to all repositories," any new repository created within that organization automatically inherits access to this secret.

An attacker would perform the following steps:

1. Create a new repository: Leveraging the common assumption that engineers can unilaterally create repositories.
2. Define a malicious GitHub Actions workflow: Within the .github/workflows/ directory of the newly created repository, a YAML file is crafted. This workflow would include a job designed to exfiltrate secrets.
3. Attempt direct echo (and bypass): A naive attempt might be echo "${{ secrets.ORG_SECRET }}". GitHub Actions logs are designed to mask secrets, so this direct approach would show *** in the logs.
4. Memory Dump for Exfiltration: The more sophisticated technique involves dumping the environment variables of the GitHub Actions worker. Secrets are often injected as environment variables. A command like cat /proc/self/environ (on Linux-based runners) can dump these variables. The output is then typically base64 encoded to prevent logging filters from catching sensitive strings and to facilitate easy transfer.

Decoding the base64 blob would reveal the actual value of ORG_SECRET and the GITHUB_TOKEN provisioned for the worker, which itself can be used to interact with the GitHub API.

Buildkite Agent Impersonation:

Buildkite's flexibility in agent definition presents a similar secret exfiltration and privilege escalation vector.

1. Create a new Buildkite pipeline: Again, assuming engineers can create pipelines.
2. Connect to a repository: The pipeline is linked to a newly created or existing repository via a webhook.
3. Arbitrary Agent Definition: The crucial step is defining the agent for the pipeline. Instead of using an agent specific to the current repository, the attacker specifies an agent that is typically used by a more privileged pipeline (e.g., a prod-deploy-agent).

By defining agents with a specific queue or other metadata, the pipeline can effectively "borrow" the identity and permissions of any agent registered to that queue, allowing it to assume roles or access secrets it shouldn't. The aws sts get-caller-identity command serves as a proof of concept to demonstrate the assumed identity.

Pre-Merge Remote Code Execution (RCE) with OIDC Federation:

This attack leverages the fact that workflow files in a feature branch are executed during a PR.

1. OIDC Provider Setup: To enable OIDC federation, an identity provider (e.g., AWS IAM OIDC provider) is configured in the cloud environment, trusting token.actions.githubusercontent.com.
2. IAM Role with Trust Policy: An IAM role is created in the cloud (e.g., GitHubActionsRole). Its trust policy is configured to allow sts:AssumeRoleWithWebIdentity from the OIDC provider. Crucially, the trust policy includes conditions on the JWT claims, such as sub (subject), which typically specifies the GitHub organization and repository (e.g., repo:my-org/my-repo:*).
3. Malicious Workflow in a PR: An attacker creates a feature branch, modifies an existing GitHub Actions workflow, or creates a new one. This workflow is designed to assume the federated role.

When this PR is opened, the workflow runs. The aws-actions/configure-aws-credentials action requests a JWT from GitHub, which is then used to assume the GitHubActionsRole. Because the workflow is running in a PR, it bypasses the merge requirement, and the get-caller-identity confirms the attacker has assumed the role. The permissions of GitHubActionsRole then dictate the extent of cloud access.

Terraform Apply-on-Plan Bypass:

This is a particularly insidious attack that exploits unsafe Terraform resources to execute arbitrary commands during the plan phase, bypassing the intended code review for apply.

1. Unsafe Terraform Resources: The attack leverages resources like null_resource combined with the local-exec provisioner, or external data sources. These allow arbitrary commands to be executed on the Terraform worker.
2. Malicious Terraform Configuration in a PR: An attacker creates a PR to a Terraform repository, adding or modifying a .tf file. This file includes a resource designed to perform an apply during the plan.

The terraform apply -auto-approve command within the local-exec provisioner forces an apply operation. The null_resource is often used because it doesn't manage external infrastructure directly, making its presence potentially less suspicious in a plan output.

1. Execution during Plan: When a PR containing this malicious .tf file triggers a terraform plan, the local-exec provisioner within the null_resource is executed. This, in turn, runs the embedded terraform apply, creating or modifying resources (like the aws_s3_bucket in the example) without any explicit apply command being issued or code review being completed. The Terraform worker's credentials (which are often administrative) are used for this unauthorized apply.

These deep dives illustrate how seemingly minor misconfigurations or overlooked execution contexts can be chained together to achieve significant compromise within CI/CD pipelines.

Demo / Proof of Concept

▶ Watch: Mitigation: GitHub Environments & Buildkite Clusters (26:00)

The talk included a compelling demonstration of the Terraform apply-on-plan bypass, showcasing how an attacker could create cloud resources without any code review or explicit terraform apply command. The scenario involved a hypothetical company, "cryptoexchange.co," using Terraform Cloud to manage their Dev and Prod AWS environments. Changes from GitHub trigger plans, and after review and merge, resources are deployed.

The demonstration proceeded as follows:

1. Objective: Create an S3 bucket named remove-this-test-bucket in an AWS account.
2. Initial State Verification: Mike Ruth first navigated to the AWS console to confirm that an S3 bucket with that specific name did not currently exist in the target account. This established the baseline.
3. Malicious PR Creation: A Pull Request (PR) was created against the Terraform repository. This PR contained a malicious Terraform configuration file.

• The configuration included a null_resource with a local-exec provisioner.
• Inside the local-exec block, a command was specified to execute terraform apply -auto-approve -no-color -input=false -var-file=terraform.tfvars.json. This command forces an apply operation.
• Additionally, an aws_s3_bucket resource definition for remove-this-test-bucket was included in the same configuration.

1. Plan Trigger and Execution: The creation of the PR automatically triggered a terraform plan operation, as is typical in CI/CD pipelines for infrastructure-as-code changes.
2. Bypass in Action: During the execution of this plan, the local-exec provisioner within the malicious null_resource was invoked. This, in turn, executed the embedded terraform apply command.

• The output of the plan showed that the malicious null_resource was being "created" (in the context of the plan).
• Crucially, the logs also indicated that the terraform apply command embedded within it had completed, and resources were created.

1. Verification of Unauthorized Resource Creation: Immediately after the plan completed, Mike Ruth returned to the AWS console. Upon refreshing the S3 bucket list, the remove-this-test-bucket was visibly present.

This demonstration vividly illustrated that the S3 bucket was created entirely within the context of a terraform plan triggered by an unmerged PR. There was no explicit code review of the apply operation, no +1 approval, and no merge to the main branch. The attacker effectively bypassed all intended security controls and deployed infrastructure directly into the cloud environment using the administrative permissions of the Terraform worker. This highlights the severe implications of the "apply-on-plan bypass" and the pre-merge RCE vulnerability.

Defensive Implications

▶ Watch: GitHub Push Rule Sets (Beta) for Granular Control (38:00)

Addressing the vulnerabilities identified requires a multi-faceted approach, balancing developer experience with robust security controls. Mike Ruth outlined several remediations, ranging from immediate configuration changes to aspirational platform features.

1. Implement Config as Code / Developer Portal:

• Goal: Remove the ability for engineers to manually create repositories, pipelines, and workflows unilaterally.
• Mechanism: Instead of manual creation, enforce the use of Infrastructure as Code (IaC) or a developer portal (e.g., Backstage, or a custom wizard). This modularizes resource creation, making it scalable and secure by default. Templates can ensure that all new CI/CD resources adhere to security baselines.

1. Leverage GitHub Environments:

• Goal: Create an affinity between secrets, the GitHub branch, and the GitHub Actions workflow content, mitigating secret exfiltration and OIDC attacks.
• Mechanism: GitHub Environments allow defining secrets that are only accessible under specific conditions.
• Post-Merge Environment: Contains highly privileged secrets (e.g., production credentials). Pinned to protected branches (e.g., main). Only workflows running on these branches can access these secrets.
• Pre-Merge Environment: Contains read-only or less privileged secrets. Accessible by non-protected branches (e.g., feature branches, PRs).
• This prevents malicious workflows in PRs from accessing production secrets, as they would be running in a pre-merge context. The demo showed a red rocket icon indicating a failed attempt to access a post-merge secret from a pre-merge environment. This also applies to OIDC federation, where IAM roles can be configured with trust policies that include conditions on the GitHub environment claim.

1. Utilize Buildkite Clusters (Introduced Feb 2024):

• Goal: Mitigate the ability to arbitrarily choose any agent and access its associated IAM roles and secrets.
• Mechanism: Buildkite Clusters establish an affinity between a repository, its Buildkite pipeline, and the specific agents that can run jobs for that pipeline. This prevents a pipeline from one repository from leveraging a privileged agent queue intended for another. Prior to this feature, achieving this affinity was significantly more challenging, often requiring complex bootstrapping layers.

1. Establish an Opinionated Branching Strategy:

• Goal: Enforce a clear separation of privileges based on branch protection.
• Mechanism:
• Pre-Merge Branches (e.g., feature branches): Should only have read-only access to resources and secrets. These are accessible by non-protected branches.
• Post-Merge Branches (e.g., main): Should have more privileged credentials and are only accessible by protected branches.
• This strategy, combined with GitHub Environments and Buildkite Clusters, ensures that even if pre-merge RCE occurs, the impact is limited to read-only actions, preventing unauthorized deployments or secret exfiltration.

1. Mitigate Terraform Plan Attacks:

• Goal: Prevent the "apply-on-plan bypass" by checking for dangerous resources before a terraform plan executes.
• Mechanism (Open Source/Atlantis): For self-hosted solutions, serialize webhooks. Prior to running a plan, implement custom code or static code analysis tools to scan the Terraform configuration for:
• null_resource with local-exec or external data sources.
• Other unsafe built-in functions or remote providers that allow arbitrary command execution.
• If found, the plan should be stopped, and an alert raised. A mechanism for admins to review and explicitly approve legitimate uses of such resources can be established.
• Mechanism (Terraform Enterprise/Cloud): Leverage Run Tasks. Historically, Run Tasks (similar to Sentinel policies) only ran prior to an apply. However, they now support pre-plan functionality. This allows organizations to define policies that check for dangerous resources before a plan is executed, effectively blocking the apply-on-plan bypass.

Aspirational / Vendor Improvements:

Mike Ruth also proposed several features that would further enhance security:

• Granular Access Control for Repository Settings (GitHub): The ability to define specific collaborators or teams that can make changes to protected file paths (e.g., .github/workflows/, .buildkite/). This would block pushes that violate these rules, preventing malicious workflow modifications at the source.
• Granular Scoping of GitHub Secrets: Allow existing organization or repository secrets to be scoped to specific GitHub Environments without requiring a full rotation and migration.
• Admission Control for CI/CD: Inspired by Kubernetes admission controllers, a service that intercepts CI/CD resource deployments/changes and checks their configuration against policies before allowing them to proceed.
• GitHub Actions: Custom deployment protection rules (e.g., a GitHub App that validates workflow changes).
• Buildkite: Agent hooks (though noted as less elegant than native solutions).
• Terraform: Enhanced Run Tasks (now with pre-plan support).

Recent Development: GitHub Push Rule Sets (Beta, April 2024):

Just days before the talk, GitHub released Push Rule Sets in beta, directly addressing the granular access control for file paths.

• Functionality: Allows defining restricted file paths (e.g., /.github/workflows/**) at the repository or organization level.
• Approvers: Specific users or teams can be designated as approvers or bypass list members.
• Enforcement: If a push violates these rules (i.e., a non-approved user tries to modify a restricted path), the push is blocked before any other triggers occur.
• Evaluation Mode: A "dry run" mode allows organizations to test rule sets and identify potential breakage before full enforcement.

This feature is a significant step towards mitigating pre-merge RCE by preventing unauthorized modifications to CI/CD configurations at the source.

Key Takeaways

• Pre-Merge Remote Code Execution (RCE) is a Pervasive Threat: The ability for engineers to push PRs and modify CI/CD workflow files in feature branches, which are then executed during the PR process, represents a critical and widely exploitable vulnerability across GitHub Actions, Buildkite, and Terraform.
• Secret Exfiltration is a Default Risk: Default configurations in GitHub Actions (org secrets accessible to all repos) and Buildkite (arbitrary agent definition) make secret exfiltration straightforward for malicious insiders or compromised accounts.
• Terraform Plan Bypass is a Critical Vulnerability: Leveraging unsafe Terraform resources (e.g., null_resource with local-exec) allows attackers to execute terraform apply commands during the plan phase, completely bypassing code review and gaining administrative access to cloud environments.
• Vendor Solutions are Evolving but Incomplete: While vendors like GitHub (Environments, Push Rule Sets) and Buildkite (Clusters) are introducing features to address these issues, a clear, universally adopted "paved path" for securing CI/CD supply chains is still emerging. Practitioners must actively implement and innovate.
• Defensive Strategies Require Layered Controls: Effective defense involves a combination of Config as Code for resource creation, GitHub Environments and Buildkite Clusters for secret and agent affinity, opinionated branching strategies for privilege separation, and pre-plan checks for Terraform.
• Granular Access Control is Key: Recent advancements like GitHub's Push Rule Sets offer promising capabilities for enforcing granular access control over CI/CD configuration files, preventing unauthorized modifications at the source and significantly reducing the attack surface.

About the Speaker(s)

Mike Ruth is a Security Engineer at Rippling. He specializes in security engineering, with a particular focus on the intricacies of software supply chains and CI/CD pipeline security. This talk is an extension of his previous work in 2022, demonstrating his ongoing commitment to researching and presenting on these critical areas of cybersecurity.