How I hacked a cloud production environment with external Terraform manipulation

How I hacked a cloud production environment with external Terraform manipulation

Speakers: Uri Aronovici, CTO, Zest

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=SAz9lFA3--g

Overview

In his compelling BSides NYC talk, "How I hacked a cloud production environment with external Terraform manipulation," Uri Aronovici, CTO and co-founder of Zest, unveiled critical security risks inherent in the widespread use of Terraform providers and modules. Aronovici, drawing from over a decade of experience in vulnerability management and cloud security, specializing in both offensive and defensive strategies, highlighted how these foundational infrastructure-as-code components can become vectors for significant cloud environment compromise. His presentation served as a stark reminder that even seemingly innocuous or officially sanctioned tools can harbor vulnerabilities exploitable for privilege escalation, data exfiltration, and complete system control.

The core of Aronovici's message revolved around a crucial blind spot in many organizations' security postures: the uncritical adoption and insufficient scrutiny of third-party or even official Terraform components. He demonstrated through real-world scenarios how vulnerable providers or cleverly crafted malicious modules could bypass security controls, leading to unauthorized access and devastating impacts on production systems. The talk underscored the urgent need for robust due diligence, continuous scanning, and stringent mitigation strategies to safeguard cloud infrastructure managed by Terraform.

This talk is particularly relevant for DevOps engineers, cloud architects, security professionals, and anyone involved in managing cloud infrastructure with Terraform. Aronovici's insights challenge the assumption of inherent security in popular tools, pushing practitioners to adopt a more proactive and skeptical approach to their infrastructure code dependencies. By dissecting the attack surface, presenting concrete exploitation paths, and offering actionable defensive measures, he provided invaluable guidance for securing critical cloud environments against these often-overlooked threats.

Background

▶ Watch: Introduction and talk agenda (0:00)

Terraform, HashiCorp's open-source infrastructure-as-code (IaC) tool, has become a cornerstone for provisioning and managing cloud resources. Its power lies in its ability to define infrastructure in human-readable configuration files, allowing for version control, automation, and consistent deployments. However, this power also introduces a significant attack surface, primarily through its reliance on providers and modules.

Providers are Terraform's fundamental plugins, written in Go, that enable interaction with various cloud services (like AWS, Azure, GCP) and other API targets. They define the resource and data types that Terraform can manage, communicating via RPC and HTTPS with the target APIs. This plugin architecture, while extensible, means that each provider introduces a potential point of failure or vulnerability. Modules, on the other hand, are self-contained packages of Terraform configurations that allow users to organize and reuse infrastructure code. For instance, a module might define a specific EC2 instance configuration or a VPC setup, bundling multiple resources together for convenience and consistency. The communication flow typically involves a user defining desired resources via modules, Terraform Core translating these requests, and providers interacting with the respective cloud APIs to create or modify resources.

HashiCorp maintains a public Terraform Registry where providers and modules are stored, categorized into three tiers:

• Official: Maintained directly by HashiCorp, with HashiCorp responsible for their security.
• Partner: Developed by other companies that are part of HashiCorp's partner program.
• Community: Contributed by individual developers or other teams, making them the most diverse and potentially least scrutinized.

A critical, yet often overlooked, warning from HashiCorp itself states: "Terraform configuration will have full access to the Variable and Terraform state within a workspace. Terraform cannot prevent the malicious Providers and modules for exfiltrating this sensitive Data." This explicit caution highlights the inherent trust placed in these components and the potential for them to be exploited if compromised or maliciously designed.

Aronovici's research revealed a profound disconnect: while millions of organizations use Terraform in production, a significant majority (based on an audience poll where only one hand was raised) are not actively scanning their providers or modules for vulnerabilities. This creates a massive "measure blind spot," where widely used components, downloaded millions of times (especially community modules), operate without adequate security scrutiny. This situation is further exacerbated by the manual and expensive nature of remediating issues once discovered, making proactive security paramount. The attractiveness of Terraform as a target, combined with this prevalent security oversight, sets the stage for the types of attacks demonstrated in the talk.

Key Findings

▶ Watch: Understanding Terraform providers and modules (2:38)

Uri Aronovici's analysis and demonstrations brought several critical findings to light, challenging common assumptions about Terraform security and exposing significant blind spots in cloud infrastructure management:

1. Widespread Vulnerability in Terraform Providers: A static analysis scan performed on the top ten providers from each of the Terraform Registry's tiers revealed a concerning distribution of critical vulnerabilities. The community tier exhibited the highest risk, with 60% of critical vulnerabilities. The partner tier followed with 29%, but perhaps most alarmingly, even the official tier — maintained by HashiCorp itself and generally assumed to be secure — contained 12% of critical vulnerabilities. This finding directly contradicts the implicit trust often placed in official components and underscores that no tier is immune to security flaws.

1. Massive Attack Surface Due to High Adoption: The sheer volume of downloads for Terraform providers and modules represents an enormous attack surface. Aronovici highlighted that community modules alone account for millions of downloads monthly, with official and partner modules seeing even higher figures. This scale means that a single vulnerability or malicious injection can potentially impact a vast number of organizations, making these components highly attractive targets for attackers.

1. Significant Security Blind Spot Among Users: An informal poll conducted during the talk revealed that very few attendees (only one hand raised) actively scan their Terraform providers and modules for vulnerabilities. This indicates a widespread "measure blind spot" where organizations are deploying critical infrastructure using components whose security posture is largely unknown or unverified. This lack of scrutiny creates fertile ground for exploitation, as vulnerabilities can persist unnoticed for extended periods.

1. Exploitable Vulnerabilities in Official Providers (CVE Example): Aronovici presented a real-world scenario involving a CVE from 2021 within the Vault provider, an official HashiCorp component. This vulnerability demonstrated that even trusted providers can have critical flaws, in this case, an incorrect mapping issue that allowed for authentication bypass and privilege escalation. This particular flaw allowed an attacker with read-only access to a GCP environment to authenticate to Vault, effectively gaining full control and enabling lateral movement or service disruption.

1. Sensitive Information Exposure Beyond CVEs: The talk also highlighted that providers can inadvertently expose sensitive information in Terraform state files or logs, even without an official CVE. Examples included issues found in the Console and Akamai providers, where sensitive data could be leaked, despite the absence of a CVE (Console even closed its GitHub issue without assigning one). This emphasizes the need for deep code review and monitoring of GitHub repositories for open issues, as relying solely on CVE databases is insufficient.

1. Ease of Creating Malicious Modules: Aronovici demonstrated how straightforward it is for an attacker to create a seemingly legitimate Terraform module that, in reality, injects a backdoor into deployed cloud resources. By leveraging common features like user_data in EC2 instances and encoding malicious scripts, attackers can easily publish modules to the Terraform Registry or GitHub, tricking victims into deploying compromised infrastructure. This capability represents a significant supply chain risk, as trust in module sources can be easily abused.

These findings collectively paint a picture of a critical and under-addressed security challenge in the realm of infrastructure-as-code, urging organizations to re-evaluate their security practices for Terraform dependencies.

Technical Deep Dive

▶ Watch: Analysis of vulnerabilities across provider tiers (5:20)

Terraform’s architecture, while robust for infrastructure management, inherently exposes several technical vulnerabilities, particularly through its provider and module ecosystem. Aronovici delved into these aspects, illustrating how they form critical attack vectors.

At its core, Terraform operates using providers, which are essentially plugins written in Go. These providers are responsible for understanding resource types (e.g., aws_instance, azurerm_resource_group) and data types specific to the cloud or service they integrate with. Communication between Terraform Core and providers happens via RPC, while providers interact with their respective target APIs (like AWS, Azure, GCP, or Vault) over HTTPS. This architecture means that the security of a Terraform deployment is intrinsically linked to the security of each provider used. A vulnerability in any provider's Go codebase or its handling of API interactions can have far-reaching consequences.

Modules, on the other hand, serve as logical groupings of Terraform configurations (.tf files) that define a set of related resources. They promote reusability and organization, allowing DevOps teams to encapsulate complex infrastructure patterns. The user defines what resources they want using modules, Terraform Core then translates this into calls to the appropriate providers, which in turn execute the desired actions against the cloud APIs. The widespread use of modules, often sourced from the public Terraform Registry or GitHub, introduces a supply chain risk: if a module is compromised or malicious, any infrastructure built with it inherits that compromise.

Aronovici detailed two primary attack scenarios to demonstrate these risks:

Scenario 1: Exploiting a Vulnerable Official Provider (CVE-2021-XXXX in Vault Provider)

This scenario highlighted a critical vulnerability (CVE-2021-XXXX, though the specific CVE ID wasn't explicitly stated in the talk, it was referenced as a real 2021 CVE) in the Vault provider, an official HashiCorp-maintained component. The vulnerability stemmed from an incorrect mapping issue within the provider itself.

Vault, HashiCorp's secret management tool, uses authentication roles (e.g., auth/gcp/role/admin) that can be configured with restrictions like bound_labels and bound_services to limit which service accounts can authenticate. The problem was that when the Vault provider processed a Terraform configuration defining such a role with specific bound_labels, it incorrectly mapped the bound_label field to an empty value during the terraform apply phase.

Attack Steps:

1. Initial Access: An attacker first needs to obtain read-only access to the target cloud environment (e.g., a GCP project).
2. Vulnerability Identification: The attacker identifies that the organization is using the vulnerable Vault provider and has configured a Vault authentication role via Terraform, expecting bound_labels to restrict access. A typical configuration might look like:

1. Exploitation: Despite the bound_labels = ["admin"] configuration, the vulnerable provider's internal mapping error meant that the bound_label was effectively empty. This allowed any service account to authenticate with the admin role. The attacker could then execute a command like:

This command would bypass the intended restrictions, granting the attacker the privileges associated with the admin role in Vault.

1. Privilege Escalation & Control: Once authenticated to Vault, the attacker could then leverage Vault's capabilities, potentially using its Google API authentication to gain full control over the cloud environment. This could lead to lateral movement, data exfiltration, or service disruption. Aronovici emphasized that this was a real incident he experienced, highlighting the severity of such flaws even in official components.

Scenario 2: Malicious Terraform Modules

This scenario demonstrated a supply chain attack involving a seemingly legitimate Terraform module that secretly injects a backdoor.

Attack Steps:

1. Module Creation & Publication: An attacker crafts a Terraform module, for example, one designed to create an EC2 instance. The module appears benign, defining standard parameters like AMI, instance type, and security groups. However, it incorporates a hidden malicious payload, typically via the user_data field, which is base64-encoded to obscure its contents.

The var.malicious_script would contain commands to establish a reverse shell, download malware, or exfiltrate data. The attacker then publishes this module to the Terraform Registry or a public GitHub repository, making it available for unsuspecting victims.

1. Victim Downloads & Imports: A victim, looking for a convenient way to provision an EC2 instance, downloads and imports this module into their Terraform configuration file (.tf file). For example:

1. Execution & Backdoor Establishment:

• The victim runs terraform plan. Because user_data is base64-encoded, the malicious script is not immediately obvious in the plan output, making it appear legitimate.
• The victim proceeds with terraform apply. Terraform provisions the EC2 instance as requested, but critically, it also executes the base64-decoded user_data script upon instance launch.
• This script establishes a backdoor, giving the attacker unauthorized access to the newly provisioned EC2 instance and potentially the wider cloud environment.

Aronovici stressed that this is a simple, yet highly effective, method for attackers to compromise systems through what appears to be standard IaC practices. The ease of publishing modules and the lack of deep code scrutiny by many users make this a significant threat.

Demo / Proof of Concept

▶ Watch: Diving into two real attack scenarios (8:30)

While Uri Aronovici's presentation did not include live coding demonstrations, he effectively walked through two compelling proof-of-concept (PoC) scenarios using code snippets and detailed explanations, illustrating how these vulnerabilities could be exploited in real-world cloud production environments. These illustrative scenarios served as powerful demonstrations of the attack vectors discussed.

Proof of Concept 1: Exploiting the Vault Provider Vulnerability (CVE-2021-XXXX)

Aronovici detailed a "real story" from his own career, outlining the exploitation of a critical vulnerability in the official Vault provider. The steps he presented clearly articulated the attack chain:

1. Initial Read-Only Access: The prerequisite for this attack was an attacker already having read-only access within the cloud environment. This is a common starting point for many attackers, seeking to escalate privileges.
2. Identifying Vulnerable Configuration: The attacker would then identify that the organization was using the Vault provider and that a Vault authentication role had been configured via Terraform. The example shown involved a configuration for a GCP authentication role (auth/gcp/role/admin) that was intended to be restricted using bound_labels (e.g., bound_labels = ["admin"]) and bound_services. The Terraform code itself appeared to follow best practices, making it difficult to spot the underlying flaw without deep technical understanding of the provider's internal logic.
3. The Provider's Flaw in Action: Aronovici explained that the critical issue lay in the Vault provider's internal mapping. When terraform apply was executed, the communication between Terraform Core and the provider resulted in the bound_label being incorrectly processed and effectively mapped to an empty value, despite the explicit configuration. This meant the intended restriction was nullified.
4. Authentication Bypass: With this flaw, the attacker could bypass the authentication restrictions. Aronovici demonstrated the command an attacker would use: vault write auth/gcp/login role=admin. Because the bound_label was effectively empty, this command would successfully authenticate, granting the attacker the privileges associated with the admin role in Vault.
5. Privilege Escalation and Control: Having successfully authenticated to Vault, the attacker could then leverage Vault's capabilities, specifically its ability to authenticate with Google APIs. This final step allowed for a full privilege escalation, giving the attacker complete control over the environment. Aronovici emphasized the grave implications, including lateral movement and service disruption, underscoring that this scenario was a personal experience that led to significant compromise.

Proof of Concept 2: Malicious Terraform Module Injection

The second PoC illustrated a supply chain attack using a maliciously crafted Terraform module designed to inject a backdoor into an EC2 instance.

1. Attacker Module Publication: Aronovici described how an attacker would first publish a seemingly legitimate module to a repository like GitHub or the Terraform Registry. This module, for instance, would be named something innocuous like ec2-instance and appear to simply provision an AWS EC2 instance.
2. Victim Module Import: A victim, unaware of the module's true intent, would then import and use it in their Terraform configuration (.tf file). The example shown was a standard module block, specifying the source and defining common EC2 parameters like ami_id and instance_type`.

1. Backdoor Injection via user_data: The "malicious" aspect of the module was hidden within its internal definition. Aronovici displayed the module's source code, showing a standard aws_instance resource block, but with a critical addition: a user_data field that was base64-encoded.

He explained that this base64-encoded string, when decoded, contained a malicious script designed to establish a backdoor (e.g., a reverse shell, data exfiltration script) upon the EC2 instance's launch. The use of base64 encoding makes the malicious payload less conspicuous during a quick review of the Terraform plan.

1. Execution and Compromise: When the victim executed terraform plan, the output would primarily show the creation of an EC2 instance with its standard attributes. The base64-encoded user_data would likely be present but not immediately revealing. Upon terraform apply, the EC2 instance would be provisioned, and the user_data script would execute, establishing the backdoor and giving the attacker control over the newly created cloud resource.

These PoCs powerfully illustrated the ease with which attackers can leverage both vulnerabilities in trusted components and the supply chain risks of third-party modules to achieve significant compromise within cloud production environments.

Defensive Implications

▶ Watch: Exploiting a CVE in an official Vault provider (9:00)

Uri Aronovici provided a comprehensive set of defensive strategies and best practices to mitigate the risks associated with vulnerable or malicious Terraform providers and modules. These recommendations span proactive due diligence, continuous monitoring, and robust architectural controls.

Proactive Due Diligence

1. Read Documentation Thoroughly: While HashiCorp provides excellent documentation, it's crucial to extend this scrutiny to the providers and modules themselves. Understand their intended functionality, potential security implications, and any known limitations.
2. Review Source Code: For any provider or module used, especially those from community or partner tiers, perform a thorough review of its source code. Look for:

• Injections: Any potential for command injection or arbitrary code execution.
• Sensitive Operations: If the provider executes command-line tools or interacts with sensitive system components.
• Data Handling: How it processes and stores sensitive data, particularly in state files or logs.

1. Examine Community Feedback and Issues: Don't rely solely on official CVE databases. Check the GitHub repositories for providers and modules for open issues, discussions, and community feedback. Many security concerns, like sensitive data exposure, might be discussed there without an assigned CVE (as seen with the Console provider example).
2. Continuous Vulnerability Scanning: Implement static analysis security testing (SAST) tools to scan the source code of providers and modules for known vulnerabilities. While exploiting provider vulnerabilities can be challenging, identifying them indicates the developer's security posture and potential for future issues.

Secure Configuration and Management

1. Pin Provider and Module Versions: Always pin specific, immutable versions of your providers and modules in your Terraform configurations (e.g., version = "~> 3.0", source = "hashicorp/aws?ref=v3.74.0"). This prevents automatic updates to potentially vulnerable or malicious new versions without explicit review and testing.
2. Enable State File Locking: Terraform state files contain sensitive information about your infrastructure. Enable state file locking (e.g., using S3 with DynamoDB for AWS) to prevent concurrent modifications and protect against manipulation by unauthorized actors.
3. Version Control for terraform.lock.hcl: Place your terraform.lock.hcl file under version control. This file records the exact versions of providers used by your configuration, offering transparency and a clear audit trail.
4. Audit Terraform Plan and State Files: Regularly audit the output of terraform plan for unexpected changes or misconfigurations before applying. Crucially, review the Terraform state file (terraform.tfstate) for any sensitive information exposure or unintended resource definitions.
5. Integrate IaC Security Tools: Utilize IaC security scanning tools (ee.g., Checkov, Terrascan, Bridgecrew) to identify misconfigurations within your own Terraform code. However, Aronovici stressed the importance of also scanning the providers themselves with static analysis tools, a step often overlooked.

Runtime Mitigation and Controls

1. Least Privilege IAM Roles and Policies: Implement strict IAM roles and policies for your CI/CD systems and the identities used by Terraform. Restrict access to only the necessary resources (tfstate files, cloud APIs) and enforce the principle of least privilege.

• Temporary Credentials: Always use temporary credentials (e.g., AWS IAM Roles for Service Accounts in Kubernetes, OIDC with GitHub Actions) instead of long-lived secrets for Terraform deployments.

1. Network Restrictions: Employ robust network segmentation and restrictions.

• VPC Load Balancers: Use VPC load balancers to control and monitor communication between services.
• Restrict Inbound/Outbound Traffic: Avoid broad 0.0.0.0/0 inbound or outbound rules. Implement specific firewall rules to limit network access to only what is absolutely necessary.

1. Cloud Workload Protection Platforms (CWPP) and Static Application Security Testing (SAST):

• CWPP: Deploy CWPP solutions to detect malicious communication or anomalous behavior from cloud workloads that might indicate a backdoor or compromise.
• SAST: While primarily for application code, SAST can be adapted or integrated into pipelines for provider/module code review where applicable.

1. Cloud Monitoring and Alerting: Configure CloudWatch (for AWS) or similar cloud-native monitoring and alerting systems (e.g., GCP Cloud Logging, Azure Monitor) to detect suspicious activities related to Terraform state files. This includes unauthorized attempts to read, write, or modify state files, as well as unusual API calls from Terraform identities.

By implementing these comprehensive defensive measures, organizations can significantly reduce their exposure to the risks posed by vulnerable or malicious Terraform providers and modules, moving from a blind spot to a position of proactive security.

Key Takeaways

• Terraform providers and modules are a significant, often overlooked, attack surface. Despite widespread use in production environments, their security is frequently not adequately scrutinized, creating a major blind spot.
• Vulnerabilities exist across all Terraform Registry tiers, including official ones. Even HashiCorp-maintained providers can harbor critical flaws, as demonstrated by the CVE in the Vault provider, allowing for authentication bypass and privilege escalation.
• Malicious modules pose a critical supply chain risk. Attackers can easily publish seemingly legitimate modules that inject backdoors (e.g., via user_data in EC2 instances), compromising cloud resources upon deployment.
• Sensitive data can be exposed without formal CVEs. Some providers may inadvertently expose sensitive information in state files or logs, requiring proactive monitoring of GitHub issues and code reviews beyond just CVE databases.
• Robust due diligence and continuous scanning are essential. Organizations must actively read documentation, review source code, check community feedback, and use static analysis tools to scan both their IaC and the providers/modules themselves.
• Implement strong architectural and operational controls. This includes version pinning, state file locking, strict IAM policies with temporary credentials, network restrictions, and comprehensive cloud monitoring to detect and prevent exploitation.

About the Speaker(s)

Uri Aronovici is the CTO and co-founder of Zest, a company focused on cloud security. He brings over ten years of extensive experience in the fields of vulnerability management and cloud security, with a specialized background in both offensive and defensive security strategies. Prior to his current role, Aronovici served as a security architect, where he encountered numerous security challenges in his career, some of which directly inspired the insights shared in this talk. His practical experience and deep understanding of cloud environments from both an attacker's and defender's perspective make him a credible voice in the realm of cloud security.