Infrastructure as Remote Code Execution: How to abuse Terraform to elevate access

Infrastructure as Remote Code Execution: How to abuse Terraform to elevate access

Speakers: Mike McCabe, President, Cloud Security Partners

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=XXcGuhCq_dc

Overview

This talk, presented by Mike McCabe, President of Cloud Security Partners, delves into the often-overlooked security implications of Infrastructure as Code (IaC) tools, specifically focusing on Terraform. While IaC is widely lauded for its ability to provision consistent, codified, and repeatable infrastructure, McCabe highlights how its inherent power and common misconfigurations can be abused to achieve remote code execution (RCE) and elevate access within cloud environments. The presentation is crucial for cloud architects, security professionals, and DevOps teams who rely on Terraform for deployments, offering a deep dive into the specific vulnerabilities and architectural pitfalls that attackers or malicious insiders can exploit.

McCabe's central thesis is that despite Terraform's significant benefits in improving infrastructure security through standardization and preventative controls, its widespread adoption often comes with critical security blind spots. These include over-privileged CI/CD pipelines, the storage of sensitive information in Terraform state files, and the presence of built-in features and API interactions that, while intended for legitimate purposes, can be weaponized. The talk underscores the necessity of treating IaC pipelines with the same rigor as application code, advocating for a multi-layered defense strategy that extends beyond basic configuration management to encompass advanced policy enforcement and continuous monitoring.

Background

▶ Watch: Introduction to the talk and Terraform basics (0:00)

Infrastructure as Code (IaC) has revolutionized how organizations manage and deploy their cloud resources. Tools like Terraform allow engineers to define infrastructure in human-readable configuration files, which are then interpreted and provisioned programmatically. This approach offers numerous benefits over traditional "ClickOps" – manual configuration via cloud consoles – by ensuring consistency, enabling version control, and facilitating the application of security controls at the provisioning stage. McCabe emphasizes that IaC, when properly implemented, can significantly reduce the "mess in the cloud" caused by inconsistent deployments and undocumented changes, making it a powerful preventative security control.

However, the very power that makes Terraform so beneficial also introduces significant security challenges. A primary concern is the common practice of granting IaC pipelines highly privileged roles, often with administrative access, to deploy and manage a wide array of cloud resources. This over-privileging creates a single point of failure: if an attacker gains control of the pipeline or introduces malicious code, they inherit these elevated permissions, enabling widespread compromise. Furthermore, Terraform, while designed for infrastructure provisioning, contains features and interacts with APIs in ways that can inadvertently facilitate code execution on remote systems or the Terraform runner itself. This talk aims to expose these pathways, illustrating how a tool designed for infrastructure can become a vector for remote code execution and data exfiltration, transforming "Infrastructure as Code" into "Infrastructure as Remote Code Execution."

Key Findings

▶ Watch: Common security challenges: pipeline privileges and RCE (2:30)

McCabe's presentation uncovers several critical security findings and attack vectors inherent in Terraform's design and common deployment patterns:

1. Terraform State File as a Secret Store: Terraform's state file, essential for tracking deployed infrastructure, often inadvertently stores sensitive information like database master passwords, SSH keys, and other secret values. Despite being a known issue since at least 2014 (GitHub issue), it remains unaddressed by HashiCorp, making these state files prime targets for attackers who can then pivot to other systems, as seen in a Sysdig investigation.

1. Over-privileged Pipelines: A pervasive issue is granting CI/CD pipelines, which execute Terraform, excessively broad IAM permissions (e.g., admin access). This enables an attacker who compromises the pipeline to perform virtually any action within the cloud environment.

1. Extensible Providers as Attack Surface: Terraform's reliance on providers (e.g., AWS, Azure), which are essentially large codebases interacting with cloud APIs, introduces a vast attack surface. These providers, including community-contributed ones, can contain edge cases, vulnerabilities, or even malicious code, similar to risks associated with third-party open-source libraries in application development.

1. Terraform API Abuse: The Terraform Cloud/Enterprise API presents another attack surface. McCabe highlights the potential for a malicious actor to use terraform import to take control of critical production resources (e.g., all production RDS instances) and then execute terraform destroy, causing catastrophic damage.

1. STS Credential Logging: When verbose logging is enabled, Terraform can dump the STS assume role session credentials (access key, secret key, session token) to logs. If developers can set variables, they can intentionally enable verbose logging, exfiltrate these highly privileged credentials, and gain direct access to the role assumed by the Terraform runner.

1. remote-exec Provisioner for Endpoint RCE: Despite HashiCorp's warnings, the remote-exec provisioner allows arbitrary scripts to be run on the provisioned instance (e.g., an EC2 machine). This bypasses traditional privilege access management (PAM) controls, enabling an attacker to establish reverse shells (e.g., via Netcat), uninstall security agents, or execute any command as the user on the endpoint.

1. local-exec Provisioner for Runner RCE: The local-exec provisioner executes scripts on the machine where Terraform is running (the pipeline runner). This allows an attacker to compromise the Terraform runner itself, steal its highly privileged STS credentials (by curling the metadata endpoint), and exfiltrate them, effectively giving them the full permissions of the pipeline.

1. Data Sources for Reconnaissance and Exfiltration: Terraform's data sources, used to fetch information about existing resources, can be abused. The http data source, for example, can be used to query the instance metadata service to retrieve STS credentials. The external data source allows executing arbitrary scripts (e.g., Python) to pull sensitive data from services like Secrets Manager using Terraform's role and then exfiltrate it.

1. "After Unknowns" Bypass for Policy-as-Code: A critical finding for policy enforcement is the concept of "after unknowns." These are values that Terraform cannot determine until the apply phase, such as ARNs for newly created resources. If policy-as-code tools like Sentinel or OPA run checks against the plan output, they can fail to validate these "after unknown" values, creating a bypass for security controls. An attacker can craft Terraform to intentionally make sensitive parameters "after unknown," thereby circumventing policy checks.

Technical Deep Dive

▶ Watch: How IaC improves cloud security posture management (4:15)

Terraform's operational model revolves around three core phases: plan, apply, and state. When a user writes Terraform configuration files (HCL – HashiCorp Configuration Language), they define desired infrastructure. The terraform plan command compares this desired state with the actual current state of the cloud environment and generates an execution plan detailing what changes will be made (create, update, destroy). The terraform apply command then executes this plan, interacting with cloud provider APIs to provision or modify resources. Finally, terraform state is a crucial file that maps the deployed cloud resources back to the Terraform configuration, enabling drift detection and managing dependencies.

The talk highlights how each of these phases and components can be manipulated for malicious purposes.

Terraform State and Secret Exposure

The terraform state file is a persistent record of all resources managed by Terraform, including their attributes and configuration. Critically, if Terraform is used to provision resources that contain secrets – such as an AWS RDS instance with a master password, an SSH key pair, or values stored in AWS Secrets Manager – these sensitive values will be stored unencrypted within the state file. McCabe points out a long-standing GitHub issue on the Terraform repository, opened in 2014, acknowledging this problem, yet it remains unresolved by HashiCorp. This means that if an attacker gains access to the Terraform state file (e.g., from an S3 bucket or a local Jenkins server backup), they immediately gain access to a treasure trove of credentials, enabling horizontal movement within the cloud environment. A real-world example cited is a Sysdig investigation where an attacker used secrets from a Terraform state file to pivot to other systems.

Providers and API Attack Surface

Terraform's functionality is extended through providers, which are plugins that interact with specific cloud services (AWS, Azure, Google Cloud, etc.) or other APIs. These providers are vast, complex codebases (e.g., the AWS provider can be hundreds of thousands of lines of code). Each provider exposes a unique API for interacting with its respective service. This extensibility, while powerful, dramatically expands the attack surface. Malicious or poorly written community providers could introduce vulnerabilities, and even official providers can have "interesting edge cases" that, when combined with other Terraform features, can lead to exploits.

Beyond providers, the Terraform Cloud/Enterprise API itself is a privileged interface. McCabe warns about the terraform import command, which allows existing, manually provisioned infrastructure to be brought under Terraform management. An attacker with API access could hypothetically import critical production resources (like all production RDS databases) into their state and then execute a terraform destroy command, leading to catastrophic data loss or service disruption.

STS Credential Logging

A subtle but critical vulnerability arises from Terraform's logging capabilities. When Terraform assumes an IAM role in AWS (via STS assume role), it receives temporary credentials (access key, secret key, session token) to interact with AWS APIs. If verbose logging is enabled, these credentials can be dumped directly into the logs. Attackers or malicious insiders who can control logging verbosity (e.g., by setting environment variables in a CI/CD pipeline) can exfiltrate these highly privileged credentials. Once obtained, these STS credentials grant the attacker direct, programmatic access to the AWS environment with the full permissions of the compromised Terraform role, bypassing the pipeline entirely.

Remote Code Execution via Provisioners

Terraform is primarily an infrastructure provisioning tool, not an endpoint configuration management system like Ansible or Chef. However, it includes provisioners that allow executing scripts on instances or the local machine. HashiCorp explicitly advises against using these for general configuration, yet they are built-in and often overlooked.

1. remote-exec: This provisioner allows running arbitrary commands on a newly provisioned remote instance (e.g., an EC2 instance) after it boots. McCabe demonstrates how an attacker could provision an EC2 instance and then use remote-exec to launch a Netcat listener, establishing a reverse shell back to a controlled server. This technique effectively bypasses traditional Privileged Access Management (PAM) systems, allowing an attacker to gain interactive shell access to production systems even if direct SSH is restricted for developers. Commands executed via remote-exec run with the permissions of the user specified in the connection block, often root or a highly privileged user.

1. local-exec: This provisioner executes commands on the machine where Terraform itself is being run – typically the CI/CD pipeline runner. This is a direct path to compromising the pipeline environment. McCabe illustrates how an attacker could use local-exec to curl the instance metadata service (e.g., http://169.254.169.254/latest/meta-data/iam/security-credentials/) to retrieve the STS credentials of the Terraform runner's IAM role. These credentials can then be exfiltrated (e.g., via another curl command to an attacker-controlled endpoint), giving the attacker the full privileges of the pipeline outside the monitored environment.

Data Exfiltration and Reconnaissance

Terraform's data sources are designed to fetch information about existing infrastructure. However, they can be weaponized for reconnaissance and data exfiltration.

1. http Data Source: This data source allows making basic HTTP GET requests. An attacker can use it to query the instance metadata service to retrieve STS credentials, similar to the local-exec example, but often running earlier in the plan phase. This technique mirrors the SSRF (Server-Side Request Forgery) vulnerability exploited in the Capital One hack.

1. external Data Source: This powerful data source allows executing an arbitrary external program or script and using its JSON output as data within Terraform. McCabe demonstrates a scenario where an external data source invokes a Python script. This Python script, running with the Terraform runner's permissions, could access sensitive information (e.g., from AWS Secrets Manager) and then exfiltrate it to an attacker-controlled server. This combines the power of Terraform's IAM role with arbitrary code execution.

Bypassing Policy-as-Code with "After Unknowns"

To mitigate these risks, organizations increasingly adopt Policy-as-Code tools like Semgrep, Open Policy Agent (OPA) with its Rego language, or HashiCorp's Sentinel. These tools scan Terraform code (HCL) or the generated Terraform plan to enforce security policies (e.g., disallowing public S3 buckets, enforcing instance types, preventing remote-exec).

• Semgrep: A fast, open-source static analysis tool that uses pattern matching to find issues in code. It can effectively detect the presence of remote-exec or local-exec provisioners directly in the HCL. McCabe provides a simple Semgrep rule example to detect provisioner "local-exec".
• OPA/Sentinel: These tools operate on the Terraform plan, allowing for more complex, logic-driven policy enforcement. They can check resource attributes, relationships, and ensure compliance with internal security baselines (e.g., CIS benchmarks). McCabe demonstrates a Sentinel rule to restrict EC2 instance types.

However, a significant bypass exists: "after unknowns." These are values that Terraform cannot determine during the plan phase and only become known after the apply phase (e.g., the ARN of a newly created resource, or values derived from complex module interdependencies). If policy-as-code tools like Sentinel or OPA evaluate policies against the plan output, and a critical security-related parameter is marked as "after unknown," the policy check effectively bypasses it. An attacker could intentionally craft Terraform configurations to ensure sensitive attributes are "after unknown," thereby circumventing security policies entirely. This issue was reportedly not well-understood by HashiCorp when brought to their attention, highlighting a gap in how policy-as-code interacts with Terraform's execution model.

Demo / Proof of Concept

▶ Watch: Visualizing a typical Terraform Cloud pipeline workflow (6:00)

While the talk did not feature a live, interactive demonstration, Mike McCabe provided several clear and concise code examples within his presentation that serve as effective proofs of concept for the discussed vulnerabilities. These included:

• A Terraform configuration snippet for an AWS EC2 instance using the remote-exec provisioner to execute netcat and establish a reverse shell.
• A Terraform configuration using the local-exec provisioner to curl the AWS metadata endpoint and exfiltrate STS credentials.
• A Semgrep rule demonstrating how to detect the local-exec provisioner.
• A Terraform configuration utilizing the http data source to query the metadata service.
• A Terraform configuration employing an external data source to execute a Python script for exfiltrating a secret from AWS Secrets Manager.
• Examples of Sentinel policies for validating EC2 instance types and IAM roles.
• A visual representation of Terraform plan output showing resources with after_unknown attributes, illustrating the policy bypass.

These examples effectively illustrate how the theoretical attack vectors can be implemented in practice using standard Terraform features, providing concrete evidence of the RCE and data exfiltration capabilities.

Defensive Implications

▶ Watch: Deconstructing Terraform's plan and apply process (6:50)

Securing Terraform and IaC pipelines requires a multi-layered, defense-in-depth approach, moving beyond basic configurations to robust policy enforcement and continuous monitoring.

1. Strict IAM Policies (Least Privilege): This is the foundational defense.

• Principle of Least Privilege: Grant Terraform CI/CD pipelines only the absolute minimum permissions required to provision specific resources. Avoid granting administrative access or overly broad permissions (e.g., actions on resources).
• Granular Permissions: Define IAM policies that explicitly list allowed actions and resources (e.g., ec2:RunInstances on specific EC2 types, s3:PutObject on specific S3 buckets).
• Conditional Policies: Utilize IAM conditions (e.g., aws:SourceVpc, aws:PrincipalArn) to restrict where and by whom Terraform roles can be assumed or used, further limiting the blast radius of a compromised role.
• Monitor IAM Changes: Continuously monitor changes to IAM roles and policies, especially those associated with CI/CD pipelines. Alerts should be triggered for any unauthorized modifications.

1. Hardened Patterns and Guardrails:

• Standardized Modules: Develop and enforce the use of hardened, security-vetted Terraform modules for common services (e.g., a secure RDS module, a compliant S3 bucket module). These modules should embed security best practices (encryption, logging, network isolation) by default.
• Internal Control Mapping: Map these hardened patterns directly to internal and external compliance frameworks (e.g., CIS Baselines, ISO 27001, SOC 2), ensuring that security controls are consistently applied.
• Centralized Module Repository: Restrict modules to be pulled from trusted, internal repositories to prevent the use of unvetted or potentially malicious third-party modules.

1. Automated Code Review (Policy-as-Code):

• Static Analysis (HCL Level): Implement tools like Semgrep early in the pipeline to scan Terraform HCL code for known malicious patterns or anti-patterns.
• Ban Provisioners: Create rules to explicitly block the use of remote-exec and local-exec provisioners, as they are direct RCE vectors.
• Detect Data Source Abuse: Develop rules to identify suspicious http data source calls (e.g., to metadata endpoints) or external data sources executing unapproved scripts.
• Custom Rules: Leverage Semgrep's flexibility to create custom rules for specific organizational policies (e.g., naming conventions, resource tagging, approved module sources).
• Plan-Time Policy Enforcement (OPA/Sentinel): Use Open Policy Agent (OPA) with Rego or HashiCorp Sentinel to evaluate policies against the Terraform plan output.
• Fine-grained Checks: Enforce granular controls like allowed EC2 instance types, mandatory encryption for S3 buckets, network configurations, and IAM role validations.
• "After Unknowns" Mitigation: Critically, implement strict Sentinel/OPA rules that explicitly ban any "after unknown" values for security-sensitive parameters. This forces Terraform to resolve all critical values at plan time, preventing attackers from bypassing policy checks.
• Unit Testing for Policies: Treat security policies as code themselves. Write unit tests for all Semgrep, OPA, and Sentinel rules, including both good and bad examples, to ensure they function as intended and prevent regressions.

1. Secure Terraform State Management:

• Treat State as Secrets Manager: The Terraform state file must be treated with the highest level of security, akin to a dedicated secrets manager.
• Secure Backend: Use a secure backend (e.g., S3 with strong encryption, versioning, and strict bucket policies, or Terraform Cloud/Enterprise's managed state) that limits access to only the authorized Terraform runner.
• Strict Access Controls: Implement stringent IAM policies on the state backend to restrict who can read or write to the state file.
• Monitoring: Monitor access and changes to the state file.

1. Comprehensive Logging and Monitoring:

• Audit Trails: Enable detailed logging for all Terraform operations, cloud API calls, and CI/CD pipeline activities.
• Alerting for Anomalies: Configure alerts for:
• Unexpected infrastructure changes (e.g., new resources in unapproved regions, unauthorized modifications to existing resources).
• Changes to IAM roles or trust policies, especially for CI/CD systems.
• Terraform state file access or modification outside of normal operations.
• Execution of remote-exec or local-exec (if not explicitly banned and needed for legacy reasons).
• Attempts to exfiltrate credentials (e.g., curl to metadata service).
• Integrate with SIEM/CSPO: Forward logs to a Security Information and Event Management (SIEM) or Cloud Security Posture Management (CSPM) solution for centralized analysis and threat detection.

By combining these proactive measures, organizations can significantly harden their Terraform environments, transform IaC into a powerful security enabler, and mitigate the risks of it becoming an RCE vector.

Key Takeaways

• Terraform is a powerful tool for infrastructure consistency but introduces unique security challenges. Its ability to manage vast cloud resources makes it a high-value target for attackers.
• Over-privileged CI/CD pipelines are a critical vulnerability. Granting administrative IAM access to Terraform runners creates a single point of failure that can lead to widespread compromise.
• Terraform state files often contain sensitive secrets. These files must be treated with the same security rigor as a dedicated secrets manager, with strict access controls and encryption.
• Built-in features like remote-exec and local-exec provisioners are RCE vectors. They enable arbitrary code execution on provisioned instances or the Terraform runner, bypassing traditional security controls.
• Data sources and verbose logging can be abused for reconnaissance and exfiltration. Attackers can use http or external data sources, or exploit verbose logging, to steal credentials or sensitive data.
• Policy-as-Code tools are essential but can be bypassed. While Semgrep, OPA, and Sentinel are crucial for enforcing security policies, the "after unknowns" issue can circumvent plan-time checks, requiring explicit rules to ban such occurrences for critical parameters.
• A multi-layered defense-in-depth strategy is crucial. This includes strict IAM, hardened patterns, automated code review, secure state management, and comprehensive monitoring.

About the Speaker(s)

Mike McCabe is the President of Cloud Security Partners, a specialized firm dedicated to assisting companies with their cloud migrations, cloud security challenges, and application security needs across modern web, IoT, and mobile spaces. With a deep passion for Infrastructure as Code (IaC) as a means to enhance infrastructure security, Mike brings extensive experience from working with large financial institutions to build secure, structured deployment pipelines using tools like Terraform. He is also the host of the "Relating to DevSecOps" podcast, where he shares insights and discussions on cloud security topics.