The Illusion of Isolation: How Isolation Failures in CI/CD Servers Lead to RCE and Privacy Risks

The Illusion of Isolation: How Isolation Failures in CI/CD Servers Lead to RCE and Privacy Risks

Speakers: Tenjo, Yuwan Wong (Students, University of Chinese Academy of Sciences; Never Stop Exploding aka NSE CTF Team)

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=VdzduFZi-GI

Overview

In this compelling Black Hat Asia talk, "The Illusion of Isolation," researchers Tenjo and Yuwan Wong from the University of Chinese Academy of Sciences delve into a critical, yet often overlooked, area of modern software development security: isolation failures within Continuous Integration/Continuous Deployment (CI/CD) servers. While traditional CI/CD security research frequently focuses on vulnerabilities that grant access to build workers or agents, this presentation shifts the spotlight to the more insidious and impactful realm of server-side compromise. The speakers meticulously demonstrate how fundamental design flaws and inadequate isolation mechanisms in popular CI/CD platforms can lead to remote code execution (RCE) and significant privacy risks directly on the CI/CD server itself.

The talk systematically dissects vulnerabilities across several prominent CI/CD systems, including Atlassian Bamboo, Go CD, OneDev, and GitLab Runners. By exploiting weaknesses in how these platforms interact with Source Code Management (SCM) systems and process user-supplied code, Tenjo and Wong unveil novel attack paths that bypass intended security boundaries. Their research underscores a crucial paradigm shift: attackers can leverage seemingly innocuous features, such as repository cloning or configuration-as-code, to elevate privileges, read sensitive files, and ultimately achieve RCE on the very heart of an organization's development infrastructure, exposing not just project code but also server-side secrets and user data.

This research is highly significant for several reasons. CI/CD pipelines are the backbone of modern software delivery, automating critical processes from code commit to deployment. A compromise at the server level can have catastrophic consequences, enabling attackers to inject malicious code into production systems, steal intellectual property, or gain a foothold for broader network penetration. The findings challenge the implicit trust often placed in CI/CD server-side operations and provide actionable insights for developers, security professionals, and platform vendors to fortify these essential systems against sophisticated attacks that exploit the "illusion" of isolation.

Background

▶ Watch: Introduction: The illusion of isolation in CI/CD (0:00)

The CI/CD pipeline is an indispensable component of modern software development, streamlining the process from code creation to deployment. A typical workflow involves developers writing code locally, committing and pushing it to a remote Source Code Management (SCM) system (like Git, GitLab, GitHub), after which the CI/CD platform automatically triggers a series of steps. These include continuous integration (CI) stages like running tests and building artifacts, followed by continuous deployment (CD) steps such as publishing and deploying the application.

CI/CD platforms are generally structured into two main components: the server side and the worker side. The server acts as the orchestrator, integrating with SCM systems, managing audit logs, defining pipelines, dispatching commands to workers, and maintaining build records. Workers, also known as agents or runners, are the execution environments that perform the actual jobs defined in a pipeline. They can be physical machines, virtual machines, containers, or pods, and typically maintain a one-to-many relationship with the server. A fundamental assumption in this architecture is robust isolation between these components and between different projects or users. This isolation is conceptualized along two dimensions: file system level isolation, where code is built in isolated containers and separated at the file system level, and data level isolation, achieved through physical boundaries, virtualized environments, and Role-Based Access Control (RBAC) policies.

Despite these intended isolation mechanisms, CI/CD environments have become prime targets for attackers. Common attack vectors include gaining initial access through weak passwords, leveraging supply chain attacks by injecting malicious code into dependencies, or searching for leaked secrets on cloud platforms. A notable attack is Poison Pipeline Execution (PPE), where attackers inject malicious code into a code repository or pipeline configuration, effectively "poisoning" the pipeline. These attacks typically grant an attacker a shell on a worker machine, providing limited access to resources like the code repository and secrets. This initial access often necessitates further actions like lateral movement or container escape to reach more valuable targets.

The motivation behind Tenjo and Wong's research stems from this limitation. They question whether it's possible to bypass these worker-centric attacks and directly target the server side of CI/CD platforms, leveraging the very isolation mechanisms that are supposed to provide security. Their goal is to achieve command execution on the server, which represents a significantly higher impact compromise than worker access. The key feature identified as a primary attack surface for server-side compromise is the Source Code Management (SCM) integration. SCM systems are inherently dangerous because:

1. User-controlled Parameters: Repository URLs, branch names, and other SCM-related parameters are often configurable by users, presenting opportunities for injection.
2. Client Interactions: SCM systems frequently interact with repositories using client tools that execute corresponding commands (e.g., git clone, git pull). This opens avenues for command injection or parameter injection.
3. Attacker-controlled Content: Attackers have full control over the content within a code repository. Malicious files, scripts, or configurations can be stored in the repository and subsequently processed or stored on the target CI/CD server, potentially chaining with other vulnerabilities.

Understanding these inherent risks associated with SCM integration forms the bedrock of their server-side attack methodology, illustrating how the "illusion of isolation" can be shattered by exploiting the critical interaction points between the CI/CD server and the code it manages.

Key Findings

▶ Watch: Common CI/CD attacks only target workers (4:40)

The research uncovered critical isolation failures and novel attack techniques across several widely used CI/CD platforms, demonstrating that server-side vulnerabilities are prevalent and highly impactful.

Atlassian Bamboo

Bamboo is a CI server from Atlassian. The researchers identified multiple vulnerabilities stemming from its Bamboo Specs feature, which allows defining build plans as code.

• Arbitrary File Read (AFR): Bamboo Specs clones the repository to the server's local file system. The parsing logic for bamboo.yml (or .yaml) does not adequately handle symbolic links. By creating a symbolic link named bamboo.yml pointing to a sensitive file (e.g., /etc/passwd) within a malicious repository, the server would attempt to read and parse it. Although parsing would fail due to invalid YAML content, the resulting exception message logged by Bamboo would contain the contents of the linked sensitive file, leading to information disclosure.
• Environment Variable Injection leading to RCE (LD_PRELOAD): Bamboo's integration with Perforce repositories allows users to specify environment variables for the Perforce client. This input lacked validation, enabling environment variable injection. When Bamboo invokes a Linux command (e.g., BASH) for Perforce operations, an attacker could inject an LD_PRELOAD environment variable. To achieve RCE, a malicious shared object (.so) library needs to be present on the server. The researchers leveraged Bamboo Specs to upload this .so file via a cloned repository.
• Path Disclosure: To make the LD_PRELOAD attack reliable, the absolute path of the cloned repository on the server was needed. The snakeyaml library used by Bamboo supports an !include tag. By crafting a malformed YAML file with a path traversal attempt in an !include tag, Bamboo's parsing logic would throw an exception, and the error message would reveal the absolute path of the directory where the repository was cloned, including the repository ID.
• File Persistence via Java Specs Sleep: A challenge for the LD_PRELOAD RCE was that cloned repositories are deleted after Bamboo Specs processing. The researchers found that Java Specs (another Bamboo Specs option) allows writing Java code to define plans. By inserting Thread.currentThread().sleep(60000); into a Java Spec, they could delay the deletion of the cloned repository for 60 seconds, providing enough time for the LD_PRELOAD attack to execute.
• Server Push Attack (RCE): This novel technique exploits Git's behavior with local bare repositories and hooks. When Bamboo's "run branch" functionality performs a git merge and git push operation, it can be directed to a local bare Git repository (created via Bamboo Specs) using the file:// protocol. Pushing to a local bare repository triggers Git server-side hooks (like pre-receive) on the client machine – which, in this case, is the Bamboo server. By embedding a malicious Git hook script within a bare repository cloned to the server, an attacker could achieve RCE when Bamboo performs a merge/push.
• Privilege Escalation: A bonus vulnerability allowed escalation from a "repo user" to a "plan user." During the Bamboo Specs process, permissions were incorrectly set to system, enabling an attacker to overwrite any configuration of a given project.

Go CD

Go CD is a free and open-source continuous delivery server.

• XML External Entity (XXE) Injection: Go CD's "create configuration repository" feature loads configuration from remote repositories, parsing JSON or YAML by default. However, it also undocumented support for XML parsing (via plugin-id: go-cd-xml), which was vulnerable to XXE injection, allowing arbitrary file reads or SSRF from the server.
• Repository Path Disclosure & Lack of File Isolation: Similar to Bamboo, malformed JSON during configuration repository parsing could disclose the absolute path of cloned repositories. Crucially, if an error occurred during the repository checkout process, the files in the repository would be retained on the server, again demonstrating a lack of file-level isolation.
• RCE via Backup Script: Combining the path disclosure and file persistence, attackers could upload a malicious script to a repository, determine its absolute path, and then configure Go CD's post-backup script setting to execute this script, leading to RCE.
• Privilege Escalation: Go CD uses JRuby and a Rails app. The researchers found that while admin routes typically enforce permission checks, it was possible to directly access underlying Rails handlers for administrative functionalities (like updating the config XML) without proper permission checks, allowing a regular user to become an administrator.

OneDev

OneDev is a Git server with CI/CD, Kanban, and packages.

• Parameter Injection (RCE): The pull from remote step executes git pull remote-URL refs. OneDev attempts to validate remote URLs to only allow HTTP/HTTPS, but this validation could be bypassed by editing the .1dev-species.yml file. This allowed injecting parameters like --upload-pack into the git pull command, leading to RCE.
• Server Push Attack (RCE): OneDev also clones repositories to the local file system and then mounts them into Docker containers for CI tasks, rather than cloning directly into the container. This design flaw, coupled with a lack of restrictions on allowed Git protocols and predictable repository paths, made it vulnerable to the same server push attack described for Bamboo. An attacker could use a checkout step to place a malicious bare Git repository on the server, then use a push to repository step to trigger its embedded Git hooks for RCE.

GitLab Runners

For GitLab Runners, the focus was on data isolation failures rather than RCE.

• Data Isolation Failure in Shell Executor: When using the shell executor, GitLab does not provide effective data isolation between different projects running on the same runner. Attackers with access to one project could simply use ls and cat commands to access private projects belonging to other users on the same shared runner. This highlights a significant privacy risk and potential for information leakage. The speakers strongly recommend using Docker-based solutions for runners to ensure adequate data isolation.

Technical Deep Dive

▶ Watch: Motivation: Executing commands on the CI/CD server (5:00)

The core of the research lies in exploiting the intricate ways CI/CD servers interact with Source Code Management (SCM) systems and process user-controlled input. The vulnerabilities often stem from a fundamental misunderstanding or oversight of file system and process isolation, particularly when dealing with repository content on the server side.

Bamboo: Exploiting Bamboo Specs and Git Interactions

Bamboo Specs is a powerful feature that allows users to define CI/CD plans using Java or YAML. The critical vulnerability here is that when Bamboo processes these specs, it first clones the remote repository to the local file system of the Bamboo server. This crucial detail breaks the illusion of isolation.

1. Arbitrary File Read via Symbolic Link:

• Bamboo's bamboo.yml parsing logic uses a function like readFileToScreen.
• Git, by default (core.symlinks=true), respects symbolic links.
• An attacker creates a Git repository containing a symbolic link bamboo.yml -> /etc/passwd.
• When Bamboo clones this repo, it creates the symbolic link on its server.
• During parsing, Bamboo attempts to read bamboo.yml. It follows the symlink, reads /etc/passwd.
• Since /etc/passwd is not valid YAML, a parsing exception is thrown.
• The key insight: this exception message, caught by an outermost catch statement, would log the content of the file it tried to parse, thus disclosing /etc/passwd.

1. RCE via LD_PRELOAD and Environment Variable Injection:

• Bamboo's Perforce integration allows users to input environment variables for the Perforce client. This user-controlled input is passed directly to a BASH command (BSH in the talk's code snippet) without sufficient sanitization.
• This enables environment variable injection. Specifically, the LD_PRELOAD environment variable can be used to load an arbitrary shared object (.so) library before a program starts, thereby executing code within the context of that program.
• Challenge: How to get the malicious .so file onto the server and determine its absolute path?
• File Upload: The Bamboo Specs cloning mechanism can be reused. An attacker commits their malicious .so file into a repository and uses Bamboo Specs to clone it.
• Path Disclosure: The snakeyaml library used by Bamboo supports an !include tag. A specially crafted YAML file with an !include referencing a path traversal (!include ../../../foo.yml) would trigger an exception. The resulting error message would contain the absolute path to the directory where the repository was cloned (e.g., /var/atlassian/application-data/bamboo/xml-data/build-dir/_git-repositories-cache/<repo_ID>/checkout/...). This provides the necessary absolute path for LD_PRELOAD.
• File Persistence: Cloned repositories are typically deleted. However, Bamboo's Java Specs feature allows arbitrary Java code execution within a Docker container. While direct server-side RCE isn't possible here, a simple Thread.currentThread().sleep(60000); within the Java Spec can delay the repository deletion for 60 seconds, providing a window for the LD_PRELOAD attack to execute.
• Execution Flow:

1. Prepare malicious .so and Java Spec with sleep.
2. Use Java Spec to clone repo and delay deletion.
3. Use malformed YAML to get absolute path.
4. Create Perforce repo, inject LD_PRELOAD=/absolute/path/to/evil.so into env vars.
5. Click "Test Connection" for Perforce repo, triggering BASH command that loads evil.so.

1. Server Push Attack (RCE):

• This attack leverages the behavior of Git hooks and bare repositories. Git hooks are scripts that run automatically before or after certain Git events. Crucially, pushing to a local bare Git repository (using file:// protocol) triggers server-side hooks (like pre-receive) on the client machine that initiated the push. In this case, the Bamboo server is both the "client" pushing and the "server" hosting the bare repo.
• Mechanism:

1. An attacker creates a special Git repository that contains a bare repository within it (e.g., a .git directory with a hooks subdirectory containing a malicious post-checkout or pre-receive script).
2. This malicious repository is cloned to the Bamboo server using Bamboo Specs.
3. Bamboo's "run branch" functionality includes git merge and git push operations. The attacker configures this to push to the local bare repository on the Bamboo server (identified via path disclosure).
4. When Bamboo executes git push file:///path/to/local/bare_repo, the malicious Git hook within the bare repo is triggered on the Bamboo server, resulting in RCE.

Go CD: XXE and Rails Route Bypass

1. XXE Injection in Configuration Repository:

• Go CD's configuration repository feature loads pipeline configurations from remote Git repositories. While documented to parse JSON or YAML, it undocumented also supported XML parsing if the plugin-id was set to go-cd-xml.
• This XML parser was vulnerable to XXE, allowing attackers to perform arbitrary file reads (e.g., /etc/passwd) or Server-Side Request Forgery (SSRF) by crafting an XML payload within the configuration repository.

1. RCE via Backup Script and Privilege Escalation:

• Go CD suffered from similar file isolation issues: malformed JSON could disclose repository paths, and errors during checkout could leave repository files persisted on the server.
• Go CD has a feature to configure a post-backup script that executes on the server after a backup.
• Execution Flow:

1. Upload a malicious script to a repository.
2. Trigger a configuration repository processing with malformed JSON to get the absolute path of the uploaded script.
3. Due to checkout errors, the script persists on the server.
4. Privilege Escalation: Go CD uses JRuby and a Rails application. The researchers found that admin-only Rails routes could be directly accessed without proper permission checks. By directly calling the handler for updating the configuration XML, a regular user could elevate their privileges to admin.
5. As an admin, configure the post-backup script to point to the malicious script on the server.
6. Trigger a backup, leading to RCE.

OneDev: Parameter Injection and Server Push

1. Parameter Injection (RCE):

• OneDev's pull from remote step executes git pull remote-URL refs. It attempted to validate remote-URL to only allow HTTP/HTTPS.
• However, this validation could be bypassed by editing the .1dev-species.yml file, which defines CI/CD steps. This allowed an attacker to inject arbitrary parameters into the git pull command.
• A payload like --upload-pack=/usr/bin/id could be injected, causing git pull to execute /usr/bin/id instead of its intended upload-pack process, effectively achieving RCE.

1. Server Push Attack (RCE):

• OneDev's CI steps clone repositories to the local file system of the server and then mount them into Docker containers. This means the raw repository content, including Git metadata and hooks, resides on the server.
• The lack of restrictions on allowed Git protocols (permitting file://) and predictable repository paths made OneDev vulnerable to the same server push attack as Bamboo. An attacker could use a checkout step to place a malicious bare Git repository (with embedded hooks) on the server, then use a push to repository step to trigger those hooks for RCE.

GitLab Runners: Data Isolation in Shell Executor

• The issue here is not RCE on the server, but a severe data isolation failure between users sharing the same GitLab Runner configured with the shell executor.
• Unlike Docker or Kubernetes executors, which provide strong container-level separation, the shell executor executes jobs directly on the underlying host.
• The researchers found that projects from different users would often be cloned into adjacent directories on the same runner machine. An attacker with access to one project could simply use standard Linux commands (ls, cat) to browse and exfiltrate files from other users' private projects, leading to significant privacy risks and intellectual property theft.

These detailed technical breakdowns highlight a recurring theme: the CI/CD server, often assumed to be a trusted and isolated orchestrator, frequently becomes an execution environment for user-controlled code and data, with inadequate safeguards.

Demo / Proof of Concept

▶ Watch: SCM as a dangerous attack surface for CI/CD (6:10)

While the talk did not feature a live demonstration, the speakers presented clear screenshots of command execution and explained the step-by-step process of their proof-of-concept exploits.

For the Bamboo RCE via Server Push Attack, the presentation included a screenshot showing the output of a Git command run from a malicious hook. Specifically, the researchers demonstrated a bare Git repository with a post-checkout hook. When cloned and then "checked out" (as part of a git checkout master command within the malicious repo's own directory structure), the hook script containing echo "arbitrary evil code goes here" was executed, confirming the ability to run arbitrary commands. The subsequent demonstration of full RCE on Bamboo showed the output of commands like id and whoami, confirming server-side code execution.

Similarly, for OneDev, a screenshot depicting the successful execution of an arbitrary command via parameter injection or the server push attack was presented, validating the RCE capabilities on that platform.

The systematic approach to building these exploits, from path disclosure to file persistence and finally command execution, implicitly serves as a detailed proof-of-concept methodology. The combination of code snippets, architectural diagrams, and command output screenshots provided a robust demonstration of the presented vulnerabilities.

Defensive Implications

▶ Watch: Real-world case: Bamboo Species vulnerability via local cloning (7:50)

The findings presented in "The Illusion of Isolation" carry profound implications for the security posture of CI/CD environments. Organizations and vendors must re-evaluate their approaches to isolation and trust boundaries within their development pipelines.

1. Strict Isolation for SCM Operations: The most critical takeaway is that repository operations (cloning, merging, pushing) must never be performed directly on the CI/CD server without robust sandboxing and extreme privilege reduction. If repository content must reside on the server for processing (e.g., for configuration-as-code features), it should be done in highly ephemeral, isolated environments (e.g., dedicated, single-use containers with strict resource limits and network policies) that are immediately purged after use.
2. Comprehensive Input Validation: All user-supplied input, especially for SCM-related parameters like repository URLs, branch names, environment variables, and configuration file content (YAML, JSON, XML), must undergo stringent validation. This includes protocol enforcement, character whitelisting, and preventing parameter or command injection. Parsing libraries should be carefully chosen and configured to mitigate risks like XXE.
3. Prioritize Containerized Runners: For CI/CD jobs that execute user-provided code, Docker-based or Kubernetes-based executors are strongly recommended over shell-based runners. Containerization provides a significantly higher degree of process, file system, and network isolation between different projects and users. If shell executors must be used, organizations should implement dedicated runners per project or user, combined with aggressive OS-level isolation (e.g., chroot, user namespaces, AppArmor/SELinux profiles) to prevent data leakage and lateral movement.
4. Least Privilege Principle: CI/CD processes, particularly those interacting with SCM, should operate with the absolute minimum necessary permissions. This applies to both the server process itself and the user accounts under which jobs are executed. Incorrect permission assignments (like system permissions during Bamboo Specs processing) can lead to privilege escalation.
5. Secure Configuration Management: Audit all configuration parsing logic within CI/CD platforms for vulnerabilities. This includes being aware of undocumented features (like XML parsing in Go CD) and ensuring that configuration files (e.g., .1dev-species.yml, bamboo.yml) cannot be weaponized to bypass security controls or inject malicious parameters.
6. Monitor for Anomalies: Implement robust logging and monitoring for unusual activities on CI/CD servers. This includes detecting unexpected file system operations (e.g., creation of .so files in temporary directories, unusual Git bare repositories), suspicious network connections, attempts to access administrative endpoints without proper authentication, and any errors that might disclose sensitive system information (like absolute paths).
7. Review Git Hook Policies: Be aware of the implications of Git hooks, especially when dealing with local or bare repositories. If CI/CD platforms perform Git operations that could trigger hooks on the server, ensure that these operations are either sandboxed or that hooks are explicitly disabled/sanitized.
8. Consider Cloud-Based SaaS Solutions: The speakers noted that "cloud-based SaaS has a natural advantage in implemented isolation mechanisms." While not a panacea, reputable SaaS CI/CD providers often invest heavily in multi-tenancy isolation at infrastructure, network, and application layers, which can be harder to achieve effectively in on-premise deployments. Organizations using on-premise solutions must dedicate significant resources to implementing and maintaining these complex isolation controls.

By addressing these defensive implications, organizations can move beyond the "illusion" and build more resilient and genuinely isolated CI/CD pipelines, safeguarding their software supply chain from sophisticated server-side attacks.

Key Takeaways

• Server-Side Vulnerabilities are Critical: Contrary to common focus on worker-side attacks, CI/CD server-side vulnerabilities are prevalent and can lead to RCE and severe data breaches, affecting the core of the software supply chain.
• Isolation Failures are the Root Cause: A fundamental lack of file system, process, and data isolation when CI/CD servers interact with Source Code Management (SCM) systems is the primary enabler for these attacks.
• User-Controlled SCM Content is a Weapon: Repository content, including configuration-as-code files, Git hooks, and environment variables, can be weaponized to exploit parsing flaws, command/parameter injection, and Git's internal mechanisms.
• Shell Executors are Inherently Insecure for Multi-Tenant Use: GitLab's shell executor demonstrated a critical lack of data isolation, allowing attackers to access other users' private projects on shared runners. Containerized executors (Docker, Kubernetes) are essential for secure multi-tenancy.
• Path Disclosure and File Persistence Enable Chaining: Vulnerabilities that disclose absolute file paths on the server, coupled with mechanisms that allow malicious files to persist (even temporarily), are critical primitives for chaining exploits to achieve RCE.
• Rigorous Input Validation and Least Privilege are Paramount: Strict validation of all user-supplied input and adherence to the principle of least privilege for CI/CD processes are non-negotiable to prevent injection attacks and privilege escalation.

About the Speaker(s)

Tenjo and Yuwan Wong are talented researchers and students from the University of Chinese Academy of Sciences. They are active participants in the competitive hacking scene, playing Capture The Flag (CTF) with the team Never Stop Exploding (aka NSE). Their research interests primarily lie in the field of web security, as demonstrated by their detailed investigation into CI/CD server vulnerabilities.