AWS CloudQuarry: Digging for secrets in public AMIs

AWS CloudQuarry: Digging for secrets in public AMIs

Speakers: Eduard Agavriloae, Senior Penetration Tester, KPMG Romania; Matei Josephs, Senior Security Researcher, KPMG Romania

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=7k3zOylPKbM

Overview

This talk, "AWS CloudQuarry: Digging for secrets in public AMIs," presented by Eduard Agavriloae and Matei Josephs, unveils a widespread security vulnerability stemming from misconfigured public Amazon Machine Images (AMIs). An AMI serves as a template for launching Amazon EC2 instances, essentially a pre-configured snapshot of a virtual machine containing the operating system, applications, and often, critical data. While AMIs are private by default, the research highlights that a significant number are inadvertently made public, exposing their entire contents to anyone with an AWS account.

The speakers, both from KPMG Romania, detail their extensive research initiative to systematically scan every public AMI across all AWS regions. Their motivation stemmed from a real-world penetration test where a client had five public AMIs, the contents of which they were forbidden to inspect. This sparked the question: "What if other companies have secrets in public AMIs?" The talk meticulously outlines their methodology for collecting, accessing, and analyzing these publicly exposed images, revealing a treasure trove of sensitive information, including valid AWS access keys and other credentials.

This research is critical because it exposes a persistent blind spot in cloud security. Organizations often overlook AMIs as a potential source of data leakage, focusing instead on more commonly audited resources like S3 buckets or EC2 instances. The findings underscore the ease with which sensitive data can be inadvertently published and the significant challenges associated with responsible disclosure and remediation, making this talk highly relevant for cloud security professionals, penetration testers, and anyone managing AWS environments.

Background

▶ Watch: Introduction and the critical public AMI finding (0:00)

Amazon Machine Images (AMIs) are fundamental components of the AWS ecosystem, serving as the blueprint for launching EC2 instances. An AMI is a pre-configured virtual machine image that includes an operating system, application server, and applications. It allows users to quickly deploy new instances with a consistent baseline, saving time and effort in setup. AWS offers two primary types of AMIs: AWS Marketplace AMIs, which are official images from trusted vendors, and Community AMIs, which can be published by any AWS user. While Marketplace AMIs are generally considered safe, Community AMIs carry inherent risks, as their publishers are not necessarily vetted, potentially containing backdoors or malware.

Crucially, AMIs can be either public or private. By default, all AMIs are private, meaning only the owner can use them. Making an AMI public is not a trivial action; it requires explicit configuration, making accidental exposure less likely but still a common occurrence. The core problem addressed by this research is that despite the difficulty in making them public, many organizations inadvertently expose AMIs containing sensitive data. The initial inspiration for this research came from a penetration testing engagement where Eduard Agavriloae discovered a customer with five public AMIs that had been exposed for multiple years. The client's refusal to allow inspection of these AMIs highlighted a potential widespread issue that warranted deeper investigation.

The researchers acknowledged prior work in this domain but emphasized the unique scope and methodology of their project. Dolev Farhi had previously scanned AMIs based on keywords in a single AWS region, which, while valuable, was limited in its global reach and depth of analysis. Ben Morris also conducted research into searching for secrets in public EBS snapshots. However, the speakers highlighted a critical distinction: an AMI being public does not automatically mean its underlying EBS snapshots are also public. This fundamental difference meant that Morris's methodology would not suffice for their goal of comprehensively scanning public AMIs, necessitating a novel approach to access the AMI contents directly. The goal of this research was therefore set: to dig for secrets in every public AMI across all AWS regions, perform responsible disclosure, and potentially earn some bug bounties.

Key Findings

▶ Watch: Defining research goals, including bounty hunting (4:10)

The research embarked on an ambitious journey to scan every public AMI across all AWS regions, yielding significant insights into the prevalence and nature of exposed secrets. The initial collection phase involved using the AWS CLI with the include-deprecated flag to ensure all AMIs, including those no longer visible in the web portal, were captured. This yielded a staggering 3.1 million AMIs. Recognizing the impracticality of scanning such a vast number, the team implemented a multi-stage filtering process:

1. AWS Marketplace AMIs: Removed, reducing the count to 1.5 million. These are generally from trusted vendors and less likely to contain unintentional secrets.
2. AWS-Owned AMIs (non-Marketplace): Further removed, bringing the total down to 1 million. These are also managed by AWS and typically do not pose a secret leakage risk from user error.
3. High-Volume Owners: Owners with more than 50 public AMIs were excluded. The rationale was that such entities are either official publishers or in a "point of no return" regarding security practices, making targeted disclosure less impactful for the research's scope. This aggressive filtering reduced the list to a more manageable 27,000 AMIs.
4. Sanity Filtering: Finally, AMIs with an unusually high number of volumes (e.g., 17 volumes) or excessively large volumes (e.g., 20 terabytes) were removed due to budget constraints and the impracticality of processing such massive data. This left a final, scannable list of 26,000 AMIs.

The distribution of these filtered AMIs varied significantly by region, with US East 1 hosting the majority, while IL Central 1 (Israel Central one) had the smallest number, with only 30 public AMIs.

The technical effort involved in this large-scale scanning was substantial. The researchers developed 1.2 thousand lines of code to automate the process, which ran for 14 days. The total AWS bill for the project amounted to $450, a remarkably efficient cost for such a comprehensive scan. From the analyzed AMIs, the team collected 500 gigabytes of carefully selected files and data suspected of containing secrets.

The most critical finding was the discovery of numerous valid secrets. The very first AWS access keys they found were active and functional. On the same day, they validated 10 other active AWS access keys, confirming the severity and widespread nature of the issue. These findings demonstrated that misconfigured public AMIs are a significant and active source of credential leakage.

Beyond discovery, the team engaged in a responsible disclosure process, attempting to notify affected companies. They sent approximately 70 disclosure emails, often having to scour the internet for contact details and resort to unconventional methods like scheduling sales calls or contacting national CERTs. Despite these efforts, they received fewer than 10 responses, and after a 90-day disclosure period, only 10 of the compromised AWS keys were invalidated. This highlights a critical industry-wide challenge in vulnerability response, where even direct, detailed disclosures are often met with silence or slow action, leaving sensitive data exposed for extended periods.

Technical Deep Dive

▶ Watch: Methodology: Filtering millions of public AMIs (5:20)

The core challenge of this research was not merely identifying public AMIs, but efficiently and cost-effectively accessing their contents at scale. The speakers outlined their journey through various methods, ultimately developing a robust technique.

AMI Collection

The initial step involved compiling a comprehensive list of public AMIs. AWS automatically deprecates images after two years, and customers can deprecate their images at any time. These deprecated images are no longer visible in the AWS web portal. To overcome this, the researchers utilized the AWS Command Line Interface (CLI) and specifically included the --include-deprecated flag with their describe-images calls. This ensured they captured all public AMIs, regardless of their deprecation status. For each AMI, they extracted its JSON definition, which contains vital metadata such as the Image ID, information about the underlying EBS snapshot, the AMI's Description, Title, and crucially, the Owner account ID. This metadata proved useful for filtering and identifying potential targets.

Accessing AMI Contents: The "Secret Searcher" Method

The most significant technical hurdle was accessing the file system contents of a public AMI. The team explored several options:

1. Copying the EBS Snapshot (Rejected): Their first attempt was to copy the AMI's underlying EBS snapshot into their own AWS account and then download it. This failed because, as noted earlier, a public AMI does not imply its associated EBS snapshot is also public by default. This distinction is critical and differentiates their work from prior research focusing solely on public EBS snapshots.
2. Launching EC2, Snapshotting, and Downloading (Rejected): The second approach involved launching an EC2 instance from the target public AMI, creating a new EBS snapshot of its volume, and then downloading that snapshot. This method was quickly deemed impractical due to its prohibitive time and cost. Downloading a 100GB EBS snapshot could take 20-30 minutes, and scaling this to terabytes of data across thousands of AMIs would not only be excessively slow but also likely trigger AWS rate limits or IP blocks.

The successful, innovative method developed by the researchers leveraged the internal AWS network for high-speed data access, which they termed the "Secret Searcher Instance" method:

1. Preparation: The researchers maintained a pre-configured "secret searcher instance" within their own AWS account. This instance was a working EC2 instance to which they could connect and perform analysis.
2. Target Instance Launch: For each target public AMI, they launched a new EC2 instance based on that AMI. A critical requirement was to launch this new instance in the same availability zone as their secret searcher instance to allow for volume transfers.
3. Volume Detachment: Once the target instance was running, it was immediately stopped. Its root EBS volume was then detached.
4. Volume Reattachment: The detached volume (containing the target AMI's filesystem) was then reattached to the pre-existing secret searcher instance. This effectively mounted the target AMI's "hard drive" onto their analysis machine.
5. Analysis: From the secret searcher instance, they could then mount the newly attached volume and access its entire file system for secret scanning. This process occurred entirely within the AWS environment, making it extremely fast and reliable, avoiding the need for costly and slow internet transfers.

A common technical obstacle encountered during this process was an ID collision between the newly attached volume and the root volume of the secret searcher instance. This required using "all kind of weird tools" (as described by the speaker) to resolve the conflict before the volume could be successfully mounted and accessed. After analysis, the temporary EC2 instance created from the target AMI could be deleted to minimize costs.

Automation Architecture

To execute this method at scale, the team designed a robust automation architecture:

• A master EC2 instance served as the control plane, running the main orchestration script.
• In each AWS region, a dedicated secret searcher instance was deployed.
• The master instance would initiate batches of 20 target EC2 instances concurrently, each based on a unique public AMI. This parallelization significantly reduced scanning time.
• For each instance in a batch, the volume detachment and reattachment process to the regional secret searcher instance was automated.
• Once attached, the secret searcher instances would then execute the secret scanning logic.

Secret Searching Strategy

Initially, the team considered using popular secret scanning tools like Truffle Hog. However, they quickly found these tools to be too slow and prone to generating a large number of false positives, which would inflate their AWS bill and analysis time. Embracing the "Zen of Python" principle of "simple is better than complex," they pivoted to a more targeted and "lazy" approach.

Their refined strategy focused on specific types of files and directories known to contain secrets:

• Environment secrets: Files like .env
• Git repositories: .git/config files, which can contain credentials, or other files within .git directories.
• AWS credentials: Files like credentials within .aws directories.
• SSH private keys: Files such as id_rsa, id_dsa, id_ecdsa, id_ed25519.
• Other sensitive files like config, history, and various json and yaml files.

Instead of complex regular expressions across entire filesystems, they opted for the native Unix utility find. This tool allowed for rapid searching of files and directories based on their names and paths. To further optimize performance and reduce noise for their proof-of-concept, they implemented additional filtering:

• Skipped Windows-specific directories like Program Files.
• Ignored files larger than 25 megabytes, assuming that most critical secrets would be in smaller configuration files.

This minimalist yet effective approach enabled them to quickly locate potential secret-containing files, which were then extracted and subjected to validation.

Demo / Proof of Concept

▶ Watch: Technical solution: The 'secret searcher' instance (8:30)

During the talk, Eduard Agavriloae provided a manual demonstration of accessing the contents of a public AMI, illustrating the critical steps of their automated process.

The demo began by showing how to search for public AMIs using the AWS CLI. Users can search by description (e.g., to target specific technologies or companies) or by AWS account ID (useful for bug hunting specific organizations). It was emphasized that specifying the correct AWS region and including the --include-deprecated flag is crucial to find all relevant AMIs.

The first attempt in the demo involved trying to copy the snapshot of a selected public AMI into their own AWS account. As predicted, this resulted in an error message: "Source snapshot is not found." This visually confirmed the earlier point that a public AMI does not equate to a public EBS snapshot, thus validating the necessity of their unique access method.

Next, the manual "secret searcher instance" method was demonstrated step-by-step:

1. An existing "secret searcher instance" was shown running in the demo account.
2. A new EC2 instance was launched based on the target public AMI. A key detail highlighted was ensuring this new instance was placed in the same availability zone as the secret searcher instance to allow for volume mobility.
3. Once the new instance was running, it was stopped.
4. Its root EBS volume was then detached.
5. The detached volume was subsequently reattached to the running secret searcher instance.
6. The presenter then connected to the secret searcher instance (using the web portal for ease of demonstration, but SSH is also an option).
7. The volume was mounted. The presenter acknowledged the "ID collision" issue as a common error that might require "weird tools" to fix, but for the demo, it was handled.
8. Finally, using command-line tools like grep, the presenter searched for AWS credentials (e.g., AWS_ACCESS_KEY_ID) within the mounted volume.
9. The demo concluded with the exciting moment of finding AWS credentials and then validating their functionality. The command to check the validity of the found keys was executed, and to the audience's (and the researchers') "surprise," the keys were indeed working. This live validation underscored the severity of the findings and the practicality of their method.

This clear, step-by-step demonstration provided tangible proof of concept for the feasibility of their approach and the critical security implications of public AMIs.

Defensive Implications

▶ Watch: Demo: Manually accessing AMI contents (9:30)

The research presented in "AWS CloudQuarry" reveals a significant and often overlooked attack surface in AWS environments. For defenders, the implications are clear and demand immediate action:

1. Comprehensive AMI Audit: Organizations must regularly and systematically audit all their AMIs for public exposure. This audit should utilize the AWS CLI with the --include-deprecated flag to ensure no AMIs are missed, even those no longer actively used or visible in the console. Automate this check across all regions and accounts.
2. Pre-Publication Scanning: Implement automated security scanning as a mandatory step in the AMI creation pipeline. Before an AMI is ever made public (or even shared privately), it should be thoroughly scanned for any embedded secrets, credentials, API keys, SSH keys, or other sensitive data. Tools like find (as demonstrated) or more sophisticated secret scanning tools can be integrated into CI/CD workflows.
3. Strict Least Privilege for AMI Contents: If, in rare and unavoidable circumstances, secrets must be embedded in an AMI, they must adhere to the principle of least privilege. These credentials should have the absolute minimum permissions required and be subject to frequent rotation. Ideally, secrets should never be embedded directly into images.
4. Secure Credential Management: Proactively migrate away from embedding secrets directly into AMIs or application code. Leverage AWS-native secure credential management solutions like AWS Secrets Manager or AWS Systems Manager Parameter Store. These services allow applications to retrieve secrets dynamically at runtime, ensuring they are never hardcoded or stored persistently in images.
5. AMI Lifecycle Management: Establish clear policies for AMI deprecation and deletion. AMIs that are no longer needed should be deprecated and eventually de-registered to reduce the attack surface. Regularly review the ownership and necessity of all active and deprecated AMIs.
6. Enhanced Security Awareness and Training: Educate development, operations, and security teams on the risks associated with public AMIs and the dangers of embedding sensitive information. Foster a security-first culture where developers understand the implications of their cloud configurations.
7. Improve Incident Response for Disclosure: The challenges faced by the researchers in responsible disclosure highlight a broader industry problem. Organizations must establish clear, accessible, and responsive channels for receiving and acting on vulnerability reports. A well-defined incident response plan for leaked credentials, including rapid key invalidation and system auditing, is essential. The fact that only 10 out of 70 disclosures led to invalidation after 90 days is a stark reminder of this deficiency.
8. Monitor for Anomalous Activity: Implement robust monitoring and alerting for suspicious activity related to EC2 instances, EBS volumes, and AMI creation/modification. While not directly preventing secret leakage, this can help detect exploitation attempts if credentials are compromised.

By adopting these defensive postures, organizations can significantly reduce their exposure to secret leakage via public AMIs and enhance their overall cloud security posture.

Key Takeaways

• Public AMIs are a Critical Source of Secret Leaks: Despite AMIs being private by default, a significant number are inadvertently made public, often containing live AWS access keys, SSH keys, and other sensitive credentials, posing a severe risk to affected organizations.
• The "Public AMI != Public EBS Snapshot" Distinction is Key: Accessing the contents of public AMIs requires a specific method because the underlying EBS snapshots are not automatically public. The "secret searcher instance" technique, leveraging internal AWS network speeds for volume reattachment, is an effective way to bypass this restriction.
• Simple Tools Can Be Highly Effective: While sophisticated secret scanning tools exist, the research demonstrated that a targeted approach using native Unix utilities like find can be highly efficient and cost-effective for identifying specific types of secrets within a large dataset, especially when combined with intelligent filtering.
• Widespread Lack of Awareness and Poor Disclosure Response: Many organizations are unaware of their public AMI exposures. Furthermore, the low response rate and slow remediation for responsible disclosures highlight a broader industry challenge in vulnerability management and incident response.
• Proactive Auditing and Secure Design are Imperative: Defenders must actively audit their public AMIs across all regions, integrate secret scanning into their AMI creation pipelines, and adopt secure credential management practices (e.g., AWS Secrets Manager) to prevent embedding secrets directly into images.
• Cost-Effective Large-Scale Scanning is Possible: The research successfully scanned 26,000 filtered AMIs across all AWS regions for a mere $450 bill, demonstrating that comprehensive security assessments can be achieved with careful architectural design and automation.

About the Speaker(s)

Eduard Agavriloae was a Senior Penetration Tester at KPMG Romania while conducting this research, which was supported by his firm. He is also a contractor at Syn Cubes, where he focuses on offensive cloud security projects. With a background in cloud security research, he briefly worked at Crowdstrike before seeking new opportunities. His expertise lies in uncovering vulnerabilities within cloud environments and developing innovative methods for security assessment.

Matei Josephs is a Senior Security Researcher at KPMG, where he has been for about a year and a half. His professional experience spans penetration testing, threat hunting, vulnerability management, and some aspects of people management. Matei is also an entrepreneur, having co-founded Hive Hack with his wife, Alexa. Their company is built on the philosophy of collaboration in cybersecurity, drawing inspiration from the collective efforts of bee colonies.