Breaching AWS Accounts Through Shadow Resources

Breaching AWS Accounts Through Shadow Resources

Speakers: Yakir Kadkoda, Michael Katchinskiy, Ofek Itach, Security Researchers at Aqua Security Team Nautilus

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=iya0VJ-9qg0

Overview

This Black Hat USA talk, "Breaching AWS Accounts Through Shadow Resources," presented by Yakir Kadkoda, Michael Katchinskiy, and Ofek Itach from Aqua Security Team Nautilus, delves into a critical and often overlooked attack surface within Amazon Web Services: automatically generated "shadow resources." The researchers demonstrate how these resources, which are typically spawned without explicit user intervention and can go unnoticed, can be leveraged through various vulnerabilities to compromise AWS accounts. The most severe of these vulnerabilities allows an attacker to inject an administrative role into a victim's AWS account, granting full control.

The core of the research revolves around exploiting predictable naming patterns of S3 buckets created by AWS services, combined with a technique known as S3 bucket name squatting or bucket sniping. By pre-claiming these predictably named buckets, attackers can intercept sensitive data, modify critical configuration files, and ultimately achieve cross-account privilege escalation. The implications are significant, as these attacks can lead to data leakage, denial of service, data manipulation affecting machine learning models, and even the creation of persistent backdoors, underscoring the importance of understanding and securing every component within an AWS environment, even those automatically provisioned.

Background

▶ Watch: Introduction and breaching AWS accounts via Shadow Resources (0:00)

The journey into Shadow Resources began when the researchers observed that AWS services, particularly CloudFormation, automatically create S3 buckets in the background when users interact with them via the management console. These resources are often "shadowy" because they are not explicitly requested by the user, may not appear immediately obvious in resource lists, and can easily go unnoticed by account owners. This automatic provisioning, while convenient, introduces a potential blind spot for security teams.

A foundational element of this research touches upon a long-standing debate in the AWS security community: whether the AWS account ID – a 12-digit unique identifier for each account – should be treated as a secret. While AWS official documentation advises careful sharing, it doesn't classify the account ID as a secret. However, as security practitioners, the speakers assert that attackers can achieve problematic outcomes simply by knowing an account ID. This talk provides solid evidence to support the argument that account IDs, when combined with other factors, can indeed be a sensitive piece of information.

Central to the attack methodology is the unique nature of S3 bucket names. Every S3 bucket name must be globally unique across all AWS accounts. If a user creates a bucket named "cool-bucket," no other AWS account can claim that name. This global uniqueness is a critical enabler for S3 bucket name squatting or bucket sniping, a technique where an attacker registers a desirable or predictable bucket name before a legitimate user or service can. Orion Mckay's work on this technique is acknowledged as foundational. The researchers expanded this concept to target automatically generated service-specific buckets with semi-predictable naming patterns.

Another crucial concept leveraged in the more advanced attacks is the Time-of-Check, Time-of-Use (TOCTOU) vulnerability. This class of vulnerability arises when there's a delay between when a system checks a condition (e.g., verifying a file) and when it uses that condition (e.g., deploying the file). If an attacker can modify the resource during this window, they can bypass security checks. This was particularly relevant for the CloudFormation template injection attack, building on techniques previously published by Rhino Lab and credited to Matthew Fuller.

Key Findings

▶ Watch: Understanding S3 bucket global naming uniqueness (3:08)

The primary discovery of this research is the widespread existence and exploitability of Shadow Resources in AWS, specifically S3 buckets automatically created by various AWS services with predictable or semi-predictable naming conventions. The researchers found that by identifying these patterns, an attacker could perform S3 bucket name squatting to intercept and manipulate critical service operations.

The most significant contributions and findings include:

1. Identification of Predictable S3 Bucket Naming Patterns: The researchers meticulously uncovered how services like AWS CloudFormation, AWS Glue, and AWS Sagemaker Canvas generate S3 buckets with names that, under certain conditions, can be predicted or discovered by an attacker. For instance, Sagemaker Canvas explicitly uses the account ID in its bucket naming convention (sagemaker-{region}-{account ID}), while CloudFormation uses a cf-templates-{hash}-{region} pattern, where the hash, while not trivially reversible, was found to be discoverable for over 1,000 existing buckets through open-source intelligence.
2. Escalation Paths via Bucket Squatting: By pre-registering these predictable buckets, attackers could achieve multiple levels of compromise:

• Denial of Service (DoS): If an attacker claims a bucket and keeps it private, the legitimate AWS service trying to create the bucket will encounter an error, preventing the victim from using the service in that region for the first time.
• Information Disclosure: By setting a permissive bucket policy, an attacker could capture sensitive files (e.g., CloudFormation templates, Glue scripts, Sagemaker datasets) uploaded by the victim, leading to data leakage.
• Data Manipulation: Attackers could modify the contents of files uploaded to their claimed buckets before the victim's service processes them. This was demonstrated with Sagemaker datasets, leading to potentially inaccurate machine learning models, and with Glue scripts, allowing for arbitrary code execution.
• Cross-Account Admin Role Injection: This was the most severe finding. By exploiting a TOCTOU window in CloudFormation, an attacker could modify a victim's template file to inject an AWS Identity and Access Management (IAM) role with administrative privileges, allowing an external attacker account to assume control of the victim's AWS account.

1. "Invisible Backdoor" in AWS Glue: A critical visual bug was identified in AWS Glue. When a victim views a script stored in an attacker-controlled bucket, they see the original, uncompromised script. However, when the script is actually executed by the Glue service, the modified, malicious version provided by the attacker runs. This creates an insidious "invisible backdoor" where the victim is unaware of the compromise.
2. Development of an Open-Source Tool ("Bucket Monopoly"): To aid in discovering and emphasizing the risk of these vulnerabilities, the researchers developed an open-source tool named "Bucket Monopoly." While the talk doesn't detail its mechanics, its purpose is to help identify more such vulnerable patterns and mitigate risks.

These findings highlight a significant security blind spot in AWS, where automatic resource provisioning, combined with predictable naming and specific service behaviors, creates opportunities for sophisticated cross-account attacks.

Technical Deep Dive

▶ Watch: Unpacking CloudFormation's predictable S3 bucket naming (4:18)

The technical core of the "Breaching AWS Accounts Through Shadow Resources" talk revolves around specific AWS service behaviors, S3 bucket naming conventions, and the exploitation of a Time-of-Check, Time-of-Use (TOCTOU) vulnerability.

AWS CloudFormation Vulnerability

AWS CloudFormation is an Infrastructure as Code (IaC) service that allows users to define AWS resources in template files and deploy them as "stacks." When a user interacts with CloudFormation via the AWS Management Console, especially for the first time in a new region, AWS automatically creates an S3 bucket for internal use.

The naming pattern for these CloudFormation-generated buckets is cf-templates-{hash}-{region}. The cf-templates prefix is constant, and the {region} component is straightforward. The challenge lies with the {hash} component, which is unique per account. Initially, the researchers believed this hash was randomized and unpredictable, posing a significant hurdle to their bucket squatting technique. However, through extensive searching using tools like Google, GitHub, and Sourcegraph, they discovered over 1,000 instances of these buckets. While the exact hash calculation wasn't fully reversed, the sheer number of publicly discoverable bucket names demonstrated that reconnaissance could yield targets. Furthermore, later in the presentation, it's clarified that other services do use the account ID directly in their bucket names, which bolstered the general principle of predictable naming.

The attack against CloudFormation proceeds as follows:

1. Bucket Squatting: An attacker, knowing a potential victim's account ID (or having identified a specific cf-templates-{hash}-{region} pattern through OSINT), pre-creates an S3 bucket with that exact name in their own AWS account.
2. Permissive Policy: To enable interaction, the attacker configures this bucket with a highly permissive resource-based policy that allows s3: actions for (any principal) from * (any AWS account). This step is crucial because S3 buckets block public access by default.
3. Victim Interaction: When the legitimate victim user, for the first time, uses the CloudFormation service in the targeted region via the console and uploads a template file, CloudFormation attempts to create an S3 bucket with the predictable name. Since the attacker has already claimed this name, CloudFormation interacts with the attacker's bucket. The victim's template file is then dropped into the attacker's S3 bucket.
4. TOCTOU Exploitation and Admin Role Injection: This is where the attack escalates to its most severe form. The CloudFormation service exhibits a TOCTOU behavior: it first uploads the template file and then, after the user defines stack parameters and permissions, attempts to deploy the resources defined in that template. This creates a window of opportunity.

• The attacker, having access to the victim's uploaded template in their claimed bucket, can modify it.
• The modification involves injecting an IAM role definition into the template, granting AdministratorAccess and establishing a trust policy that allows the attacker's external AWS account to assume this newly created role.
• When the victim finally submits the stack for deployment, CloudFormation processes the modified template from the attacker's bucket.
• Result: An admin role is created in the victim's account, with a trust policy allowing the attacker to assume it, achieving cross-account administrative access. This requires the initiating CloudFormation user to have sufficient privileges to create such an IAM role in the first place, which is often the case for users managing IaC.

AWS Glue Vulnerability

AWS Glue is a serverless data integration service for data engineers, used primarily for data manipulation tasks. Similar to CloudFormation, when a user creates a new Glue job, an automatic S3 bucket is provisioned to store job scripts and related files. While the exact naming pattern wasn't explicitly detailed as glue-{hash}-{region} in the same way as CloudFormation, the implication is that it also follows a predictable pattern, likely incorporating the AWS account ID.

The attack scenario for Glue mirrors the CloudFormation one:

1. Bucket Squatting: An attacker pre-claims the predictably named Glue S3 bucket.
2. Script Interception and Manipulation: When a victim user creates a Glue job and uploads a script, it is stored in the attacker's controlled bucket. The attacker can then download, modify, and re-upload the script.
3. "Invisible Backdoor" Visual Bug: A critical finding here is a visual bug in the AWS console. When the victim views their Glue job script, the console displays the original script content that they initially uploaded. However, when the Glue job is executed, the service retrieves and runs the modified script from the attacker's S3 bucket. This creates an invisible backdoor, where the victim is completely unaware that malicious code is being executed under their account.
4. Impact: This can lead to arbitrary code execution, data exfiltration, or further privilege escalation within the victim's environment, depending on the modifications made to the Glue script.

AWS Sagemaker Canvas Vulnerability

AWS Sagemaker Canvas is a visual interface within the Sagemaker ecosystem designed for business analysts to build, train, and deploy machine learning models without writing code. This service explicitly creates S3 buckets with a highly predictable naming pattern: sagemaker-{region}-{account ID}. This pattern directly incorporates the victim's AWS account ID, making it straightforward for an attacker to predict the bucket name given the account ID.

The attack flow is as follows:

1. Predictable Naming: The bucket name sagemaker-{region}-{account ID} is fully predictable if the attacker knows the victim's account ID and the region they will use Sagemaker Canvas in.
2. Bucket Claiming: The attacker claims this bucket name before the victim uses Sagemaker Canvas in that region for the first time.
3. Data Leakage: When the victim creates a dataset or uploads files via Sagemaker Canvas, these files are transparently uploaded to the attacker's pre-claimed S3 bucket. The attacker can then download and access the victim's sensitive datasets, leading to data leakage.
4. Data Manipulation: Beyond leakage, the attacker can modify the datasets stored in their bucket and re-upload them. When Sagemaker Canvas uses these manipulated datasets for model training, it can lead to inaccurate or poisoned machine learning models, potentially causing significant operational or business impact without the user's knowledge.

Collectively, these vulnerabilities highlight a systemic issue where the convenience of automatic resource provisioning, combined with predictable naming and specific service implementation details, creates potent attack vectors.

Demo / Proof of Concept

▶ Watch: Escalating to information disclosure via public access (6:10)

The talk features a clear demonstration of the CloudFormation attack chain, showcasing the most severe outcome: cross-account admin role injection.

The demonstration typically follows these steps:

1. Attacker Pre-configuration:

• The attacker first identifies the target victim's AWS account ID.
• They then determine a likely region the victim might use CloudFormation for the first time.
• Based on internal research or public information, the attacker predicts the cf-templates-{hash}-{region} bucket name (or for other services, sagemaker-{region}-{account ID}).
• The attacker creates an S3 bucket with this exact predicted name in their own AWS account.
• Crucially, the attacker configures this newly created bucket with a highly permissive bucket policy. This policy allows s3: actions from (any principal) across * (any AWS account), effectively opening it for cross-account interaction.

1. Victim's Initial Interaction:

• The victim, unaware of the pre-claimed bucket, logs into the AWS Management Console.
• They navigate to the CloudFormation service and initiate the creation of a new stack.
• The victim uploads a legitimate CloudFormation template file (e.g., defining an EC2 instance or a simple S3 bucket) via the console.
• Behind the scenes, the CloudFormation service attempts to store this template. Because the attacker has squatted the bucket name, the template is dropped into the attacker's S3 bucket instead of a bucket in the victim's account.

1. Attacker's Exploitation (TOCTOU Window):

• While the victim is still configuring their stack (e.g., entering stack name, parameters, reviewing permissions), the attacker monitors their claimed S3 bucket.
• Upon detecting the victim's uploaded template file, the attacker immediately downloads it.
• The attacker then modifies this template file. The modification involves adding a new AWS IAM resource definition. This new resource creates an IAM role with AdministratorAccess permissions and, critically, sets a trust policy that allows the attacker's specific AWS account ID to assume this role.
• The modified, malicious template is then re-uploaded to the attacker's claimed S3 bucket, overwriting the original. This action takes place within the Time-of-Check, Time-of-Use (TOCTOU) window before the CloudFormation service fully processes the template for deployment.

1. Victim's Deployment and Compromise:

• The victim completes their stack configuration and proceeds to deploy the CloudFormation stack.
• CloudFormation retrieves the template from the (now attacker-controlled and modified) S3 bucket.
• The service then creates all resources defined in the modified template, including the malicious admin role with the attacker's account as a trusted entity.
• The demonstration then shows the attacker, from their own AWS account, using the sts:AssumeRole API call to assume the newly created admin role in the victim's account, thereby gaining full administrative control.

While the primary focus of the demo is the CloudFormation scenario, the speakers also describe the mechanics of the AWS Glue "invisible backdoor." This involves showing how a victim might view their script in the console and see the original content, only for the modified script (placed by an attacker in a squatted Glue bucket) to execute when the job runs. Similarly, for Sagemaker Canvas, the demonstration would involve showing a victim uploading a dataset, which then appears in the attacker's bucket, leading to potential data leakage or manipulation.

The efficacy of these proofs of concept highlights the severe risk posed by unmanaged "shadow resources" and underscores the critical need for proactive monitoring and strict access controls even on automatically generated infrastructure.

Defensive Implications

▶ Watch: Achieving admin role injection through template modification (7:35)

The vulnerabilities highlighted in this talk present significant defensive challenges, as they exploit inherent behaviors of AWS services that are often considered "behind the scenes." Defenders must adopt a proactive and comprehensive approach to mitigate these risks.

1. Treat AWS Account IDs as Sensitive: The talk strongly argues that AWS account IDs, despite AWS's stance, should be treated with care. While not a secret on its own, its combination with predictable resource naming allows for targeted attacks. Avoid unnecessary public exposure of account IDs.
2. Proactive Discovery and Monitoring of Shadow Resources: Organizations must implement mechanisms to discover and monitor all resources, including those automatically provisioned by AWS services. Tools like the researchers' "Bucket Monopoly" (or similar custom scripts and cloud security posture management (CSPM) solutions) should be used to identify S3 buckets created by services with predictable naming conventions across all regions.
3. Implement Account-Level S3 Public Access Block: This is a crucial first line of defense. Enable Block Public Access settings at the AWS account level. This setting prevents S3 buckets from being made public, even if an attacker claims a bucket name. While it might lead to a Denial of Service (DoS) for the legitimate user (as the service won't be able to access the bucket), it prevents data leakage, manipulation, and admin role injection via public bucket policies.
4. Strict S3 Bucket Policy Auditing: Regularly audit S3 bucket policies, especially those that allow cross-account access or public access. Ensure that Principal: "" or Allow: "" policies are strictly limited and only applied when absolutely necessary, with appropriate conditions. Look for policies that grant broad s3:PutObject or s3:GetObject permissions.
5. Principle of Least Privilege for Service Users: Enforce the principle of least privilege for IAM users and roles that interact with services like CloudFormation, Glue, and Sagemaker. If a user only needs to deploy specific types of resources, their permissions should be scoped narrowly. For instance, a CloudFormation user should only have permissions to create the resources defined in their legitimate templates, not arbitrary IAM roles with AdministratorAccess.
6. Region Strategy and First-Time Usage: Be aware that many of these attacks rely on a victim using a service "for the first time in a new region." Organizations should have a clear strategy for region usage, ideally restricting users to specific, necessary regions. For critical accounts, consider pre-creating service-specific buckets in all necessary regions to prevent squatting.
7. Input Validation and Sanitization for IaC: For services like CloudFormation and Glue, ensure that templates and scripts are rigorously validated and sanitized. Implement security checks in your CI/CD pipelines to scan for malicious code or overly permissive IAM definitions before deployment.
8. Educate Developers and Cloud Engineers: Raise awareness among development and operations teams about the concept of shadow resources, the risks of predictable naming, and the importance of secure configurations for even automatically generated resources.
9. Monitor for Suspicious IAM Activity: Implement robust monitoring for suspicious IAM activities, particularly sts:AssumeRole calls from unexpected external accounts, or the creation of new IAM roles with administrative privileges. AWS CloudTrail logs are essential for detecting such events.
10. Consider Cloud Security Posture Management (CSPM) Tools: Leverage CSPM solutions that can automatically detect misconfigurations, overly permissive policies, and unmanaged resources across your AWS environment, including newly created or "shadow" S3 buckets.

By integrating these defensive strategies, organizations can significantly reduce their exposure to attacks leveraging shadow resources and maintain a stronger security posture in AWS.

Key Takeaways

• Shadow Resources Pose Significant Risk: Automatically generated AWS resources, particularly S3 buckets, often go unnoticed but can be exploited due to predictable naming patterns.
• S3 Bucket Name Squatting is a Potent Attack Vector: By predicting or discovering service-specific S3 bucket names, attackers can pre-claim them, intercepting data and manipulating service operations.
• Cross-Account Admin Role Injection is Possible: Through a Time-of-Check, Time-of-Use (TOCTOU) vulnerability in CloudFormation, attackers can modify legitimate templates to inject an admin role, gaining full control of a victim's AWS account.
• Data Leakage and Manipulation are Common Outcomes: Services like Sagemaker Canvas and Glue can be exploited for sensitive data leakage or manipulation of critical assets (e.g., machine learning datasets, Glue job scripts).
• "Invisible Backdoors" Can Persist Unnoticed: A visual bug in AWS Glue allows attackers to run modified scripts while victims see the original, creating a stealthy and persistent compromise.
• Proactive Defense is Crucial: Implementing account-level S3 Public Access Block, strict policy auditing, least privilege, and continuous monitoring for shadow resources are essential mitigations.

About the Speaker(s)

The talk "Breaching AWS Accounts Through Shadow Resources" was delivered by a team of dedicated security researchers from Aqua Security Team Nautilus:

• Yakir Kadkoda: A security researcher focusing on cloud vulnerabilities, open-source vulnerabilities, and other areas of cloud security.
• Michael Katchinskiy: Also a security researcher with Aqua Security Team Nautilus, contributing to the discovery and analysis of cloud and open-source vulnerabilities.
• Ofek Itach: Completing the trio, Ofek is a security researcher whose work, alongside Yakir and Michael, explores various aspects of cloud and open-source security challenges.

As part of Aqua Security's Nautilus team, their collective expertise lies in uncovering and dissecting vulnerabilities within cloud environments and open-source projects, providing valuable insights into emerging threats and defensive strategies.