CloudShell Hide-n-Seek: enjoying the sweet persistent sounds of silence!

CloudShell Hide-n-Seek: enjoying the sweet persistent sounds of silence!

Speakers: Jenko Hwong, Principal Security Researcher, Huntress Labs; Chris Ryan, Principal Security Researcher, Huntress Labs

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=wnKLOl5qmu8

Overview

In "CloudShell Hide-n-Seek: enjoying the sweet persistent sounds of silence," Jenko Hwong and Chris Ryan, Principal Security Researchers at Huntress Labs, delve into the often-overlooked security implications of Cloud Shell environments across major cloud providers. The talk, originally conceived from a Defcon Cloud Village Capture The Flag (CTF) challenge, exposes how these interactive, browser-based Linux terminals, intended for administrative convenience, can become potent vectors for persistence and abuse by adversaries.

The core message of the presentation is that Cloud Shells operate outside the traditional API-centric security models of cloud environments, creating significant blind spots for defenders. Hwong and Ryan demonstrate how an attacker can establish persistent access within these "unmanaged compute environments," bypass standard API token revocation mechanisms, and even leverage unique features like AWS's RoleSessionName to create an effectively infinite number of hidden, persistent shells. This research highlights a critical area where cloud security paradigms fall short, offering a full OS-level attack surface that is difficult to detect and remediate using conventional cloud security tools and playbooks.

Background

▶ Watch: Talk Introduction: CloudShell Hide-n-Seek Overview (0:00)

The genesis of this research originated from a CTF challenge Hwong built for Defcon's Cloud Village. The objective was to design a challenge where the flag was intentionally hidden within a Cloud Shell environment, specifically to frustrate participants whose traditional cloud hacking tools and API-based enumeration techniques would prove ineffective. This "screw you" attitude, as Hwong humorously puts it, inadvertently uncovered profound security implications for real-world cloud environments.

Cloud Shells are interactive, browser-based compute environments provided by all major cloud providers (AWS CloudShell, Azure Cloud Shell, Google Cloud Shell). They offer a full Linux terminal, pre-installed with common command-line tools and Cloud CLIs, allowing administrators to quickly execute commands against their cloud resources without needing to set up a local development environment. Critically, these environments are largely unmanaged by the user's organization; they are ephemeral, reset after a period of inactivity (typically 20 minutes to 12 hours), and their underlying infrastructure is abstracted away from direct API access or traditional monitoring. This unique operational model, while convenient for legitimate users, creates a distinct security boundary that existing cloud security practices often fail to address. The challenge lies in the fact that Cloud Shells are not directly API-wrapped in the same way EC2 instances or S3 buckets are, meaning traditional API call logging and security tooling often miss activities occurring within them, making them prime targets for covert operations and persistence.

Key Findings

▶ Watch: CTF 'screw you' design philosophy and roleplay (2:00)

The research by Hwong and Ryan unveils several critical findings regarding the abuse and persistence capabilities within cloud shell environments:

1. Unique Persistence Vector: Cloud Shells provide a full Linux operating system environment, allowing attackers to leverage traditional Linux persistence mechanisms (e.g., modifying .bashrc, installing binaries) within the user's persistent home directory. This persistence is independent of the ephemeral nature of the underlying container, surviving resets and providing a foothold.

1. Session Revocation Bypass: A significant discovery is that standard AWS API token revocation (e.g., using revoke-session) does not terminate an active Cloud Shell session established via a websocket protocol. An attacker with an existing interactive shell can continue to operate within the environment for its full lifespan (up to 12 hours), even after the associated API credentials have been invalidated. This creates a critical window of continued access for adversaries, making rapid incident response significantly more challenging.

1. RoleSessionName Abuse for Infinite Persistence: The most alarming finding is the abuse of the RoleSessionName parameter when assuming an IAM role. If an attacker compromises an identity with sts:AssumeRole permissions, they can specify a unique RoleSessionName string. Each unique RoleSessionName used to assume a role and subsequently access Cloud Shell results in a distinct, separate Cloud Shell environment with its own persistent home directory and container. This means a single compromised role can be leveraged to create an effectively unlimited number of hidden, persistent Cloud Shells across multiple regions, each acting as a covert beacon that is exceedingly difficult for defenders to discover or attribute.

1. OS-Level Access and Tradecraft: Cloud Shells offer a full OS-level access surface. This allows attackers to:

• Execute arbitrary commands.
• Upload/download files (leveraging underlying curl commands).
• Install custom tools (e.g., Netcat, crypto miners).
• Exploit CVEs in pre-installed packages.
• Establish reverse shells to external C2 infrastructure, effectively turning the Cloud Shell into a free, cloud-native compute node for malicious activities like crypto mining or data exfiltration.

1. Direct Websocket Interaction: The speakers demonstrated the ability to interact with Cloud Shell environments directly via its underlying websocket protocol, bypassing the browser interface entirely. This programmatic access allows for automated exploitation, removes the need for a graphical interface, and can potentially evade browser-based logging or security controls.

1. Cross-Cloud Applicability: While the detailed demonstration focused on AWS, Hwong and Ryan confirm that similar unmanaged compute environments and associated vulnerabilities exist in Azure Cloud Shell and Google Cloud Shell. For instance, Google Cloud Shell is automatically provisioned for anyone with a personal Gmail account, representing a broad, low-friction entry point for attackers.

Technical Deep Dive

▶ Watch: Live demo begins: Initial web reconnaissance (3:20)

The technical exposition of "CloudShell Hide-n-Seek" is largely framed through a compressed CTF walkthrough, demonstrating a multi-stage attack that culminates in Cloud Shell compromise and persistence.

The initial phase of the CTF involved gaining a foothold in a simulated cloud environment. This began with a vulnerable web application presenting an entry point. The attacker (Chris Ryan) first identified an IDOR (Insecure Direct Object Reference) vulnerability by manipulating a user parameter in the URL, changing U=agent to U=admin to reveal different content. This was a warm-up, but the real exploit was an SSRF (Server-Side Request Forgery) vulnerability. By injecting a curl command into a search form, the attacker was able to make the web application's backend server query its local IMDS (Instance Metadata Service) endpoint (e.g., http://169.254.169.254/latest/meta-data/iam/security-credentials/web-role). This classic SSRF exploit, prevalent in older AWS IMDSv1 configurations, allowed the attacker to dump temporary API credentials associated with the web-role attached to the compromised EC2 instance.

With these temporary AWS API credentials, the attacker gained programmatic access to the AWS environment. Initial enumeration attempts using aws s3 ls and aws ec2 describe-instances were intentionally made unfruitful by the CTF builder (Jenko Hwong) to redirect the participant. The breakthrough came from enumerating the permissions of the compromised role using aws sts get-caller-identity and aws iam get-authorization-details. This revealed explicit permissions for cloudshell:* actions, serving as a critical clue for the attacker to pivot towards the Cloud Shell environment.

The pivot from API credentials to a browser-based console session is a key technical leakage. Using a script, the temporary AWS API tokens were exchanged for a federated URL, allowing direct login to the AWS console and Cloud Shell without requiring separate browser-based authentication. Once inside Cloud Shell, the attacker found a full Linux environment. The CTF challenge included several specific modifications: sudo access was removed to prevent participants from impacting the challenge for others, and the flag was hidden in the persistent home directory.

To achieve persistence and extract the flag, the attacker delved into Linux tradecraft. Enumerating files in the home directory, they discovered suspicious modifications to .bashrc files. One .bashrc file hinted at a Netcat listener and the creation of temporary files triggering a specific action. The attacker then set up a local Netcat listener on a high port (e.g., 1024, as lower ports require root) and created a temporary file, triggering the pre-configured script in the compromised Cloud Shell to send the flag back to the listener.

The talk then shifts from CTF to real-world implications. Hwong and Ryan demonstrate how an attacker, having gained access to a Cloud Shell, can establish a persistent reverse shell to an external C2 server (e.g., Digital Ocean droplet) using Netcat. This provides full OS-level access, enabling activities like crypto mining or data exfiltration.

A more advanced technique involves bypassing the browser entirely. The speakers showcased a Python script that connects directly to the Cloud Shell via its websocket protocol. This allows for programmatic execution of commands, potentially evading browser-based logging and providing a more stealthy and automated way to interact with the shell. Furthermore, the Cloud Shell's file upload feature, which internally uses a curl command, can be leveraged to upload arbitrary malware.

The most insidious technical finding is the abuse of the RoleSessionName. When assuming an IAM role via sts:AssumeRole, a unique string can be passed as RoleSessionName. If a Cloud Shell is then opened under this assumed role, it creates a separate, distinct Cloud Shell environment with its own persistent home directory, isolated from other sessions of the same underlying role or user. An attacker with AssumeRole permissions can generate an arbitrary number of these unique RoleSessionName values, effectively creating an "infinite" number of persistent Cloud Shell environments across all enabled regions. Each of these environments acts as a covert, unmanaged compute instance, making discovery and remediation a monumental task for defenders.

Demo / Proof of Concept

▶ Watch: IDOR discovery and secret agent clues (4:50)

The entire presentation effectively functions as an extended demonstration, moving from initial compromise to advanced persistence techniques within cloud shell environments.

The initial phase of the demo walked through the CTF challenge, illustrating:

1. Web Application Exploitation: Chris Ryan demonstrated navigating a vulnerable website, identifying an IDOR vulnerability by modifying a URL parameter, and then exploiting an SSRF to query the IMDS.
2. Credential Dumping: The curl command injected via SSRF successfully retrieved temporary AWS API credentials (access key, secret key, session token) for a web-role.
3. API-to-Browser Pivot: Ryan then used these stolen credentials with a custom script to generate a federated URL, enabling direct access to the AWS console and subsequently the Cloud Shell, bypassing traditional login.
4. Cloud Shell Persistence: Within the Cloud Shell, Ryan demonstrated how to locate the hidden flag by investigating modified .bashrc files and setting up a Netcat listener to receive the flag, showcasing a basic persistence mechanism.

Beyond the CTF, the demo transitioned to real-world attack scenarios:

1. Reverse Shell Establishment: Ryan showed how to set up a Netcat reverse shell from a compromised Cloud Shell to an external listener (e.g., on Digital Ocean), providing interactive OS-level access outside the browser.
2. Websocket Protocol Bypass: A Python script was demonstrated connecting directly to the Cloud Shell's websocket protocol, allowing command execution without using the AWS console in a browser. This highlighted the ability to automate Cloud Shell interaction and potentially evade certain logging mechanisms.
3. Revocation Failure: Jenko Hwong, playing the blue teamer, attempted to revoke Chris Ryan's access by performing a revoke-session API call on the compromised role. Crucially, Ryan demonstrated that his active Cloud Shell session, established via the websocket, remained operational, proving that standard API revocation does not immediately terminate these sessions.
4. RoleSessionName Abuse: The concept of using RoleSessionName to spawn multiple, distinct, and persistent Cloud Shell environments was explained and demonstrated, emphasizing how this significantly complicates detection and remediation by allowing an attacker to spread their foothold across numerous isolated environments.

The demonstrations collectively underlined the talk's central thesis: Cloud Shells represent a potent, often overlooked attack surface for persistence, particularly due to their unmanaged nature and the unique ways in which session management deviates from traditional API-centric cloud security.

Defensive Implications

▶ Watch: Source code inspection, identifying form for injection (6:30)

The revelations from "CloudShell Hide-n-Seek" demand a re-evaluation of defensive strategies for cloud environments, particularly concerning unmanaged compute environments like Cloud Shells. Defenders must recognize that these environments operate outside typical API monitoring and traditional cloud security playbooks.

1. Risk Analysis and Inventory: The first step is to perform a comprehensive risk analysis. Defenders should actively enumerate which users and roles have permissions to access Cloud Shells across all regions. This involves querying IAM policies to understand the potential exposure. Hwong demonstrated scripting this enumeration process to identify active Cloud Shell users and their associated regions.

1. Restrictive IAM Policies: Implement granular IAM policies to restrict Cloud Shell access.

• Least Privilege: Grant Cloud Shell access only to necessary users and roles.
• Region Restriction: Limit Cloud Shell access to specific, approved regions, reducing the attack surface.
• MFA Enforcement: Enforce Multi-Factor Authentication (MFA) for any action that initiates a Cloud Shell session.
• Deny All: For organizations with no legitimate use case, the most secure approach is to deny all Cloud Shell access via IAM policies.

1. Leverage VPC-enabled Cloud Shells: Major cloud providers offer VPC-enabled Cloud Shells (e.g., AWS CloudShell in your VPC). This crucial feature places the Cloud Shell environment within the organization's virtual private cloud, allowing network traffic to be controlled by security groups and firewall rules. While this option often comes with the trade-off of no persistence (home directory resets), the enhanced network control is a significant security gain. Defenders should carefully weigh the administrative convenience of persistence against the security benefits of network isolation.

1. Customized Provisioning and Hardening: For organizations that require Cloud Shell functionality, consider implementing a "do-it-yourself" approach to provisioning:

• Pre-install Agents: If feasible, pre-install security agents or monitoring tools within the Cloud Shell's base image or upon session initialization.
• Remove sudo: Replicate the CTF's hardening by removing sudo access, limiting an attacker's ability to modify system files.
• Package Whitelisting: Control which packages and binaries (e.g., Netcat) are available or can be installed.
• Custom CLIs: For specific administrative tasks, consider providing custom CLI tools that interact with API-managed compute environments (e.g., EC2 with SSM) rather than relying on the unmanaged Cloud Shell.

1. Enhanced Detection and Auditing: Since Cloud Shell activity is not fully logged in traditional API logs, defenders need to look for indirect indicators:

• Session Creation Bursts: Monitor for anomalous spikes in Cloud Shell session creation for a single identity.
• Beaconing Behavior: Detect regular, periodic activity from Cloud Shells, indicating an attempt to overcome idle timeouts and maintain persistence, similar to C2 traffic.
• Long-Running Sessions: Identify Cloud Shell sessions that consistently run for the maximum allowed duration (e.g., 12 hours in AWS), which could indicate automated malicious activity.
• Anomalous API Calls: While Cloud Shell commands aren't logged, any API calls made from the Cloud Shell are. Look for unusual API activity originating from Cloud Shell identities.
• RoleSessionName Monitoring: Crucially, monitor cloud logs for unusual or non-standard values in the RoleSessionName field when sts:AssumeRole is invoked. The presence of suspicious strings could indicate an attacker attempting to create hidden, persistent Cloud Shells.

1. "Nuclear Option" for Remediation: While revoke-session doesn't immediately kill active websocket sessions, there is a more drastic measure: the delete-user-action for Cloud Shell. This action, when triggered, will wipe the user's persistent home directory, destroying any attacker-planted persistence mechanisms. This should be combined with API token revocation and other identity-based controls to fully evict an adversary.

Key Takeaways

• Cloud Shells are Unique Attack Surfaces: These browser-based Linux environments offer full OS-level access and operate outside traditional API-centric cloud security models, creating blind spots for defenders.
• Persistence Bypasses Standard Revocation: An active Cloud Shell session established via websocket protocol will persist for its full duration (up to 12 hours) even after the associated API credentials have been revoked, providing a critical window for adversaries.
• RoleSessionName Enables Covert, Infinite Persistence: Attackers can leverage the RoleSessionName parameter with sts:AssumeRole to create an effectively unlimited number of distinct, persistent Cloud Shell environments, each with its own isolated home directory, making detection and remediation extremely challenging.
• Traditional Linux Tradecraft Applies: Once inside a Cloud Shell, attackers can use standard Linux persistence mechanisms (e.g., .bashrc modification, Netcat for reverse shells, crypto mining) and bypass browser-based controls via direct websocket protocol interaction.
• Defensive Strategies Must Adapt: Organizations need specific IAM policies for Cloud Shell, consider VPC-enabled Cloud Shells for network control, and implement enhanced logging/detection for unusual session creation, beaconing behavior, and suspicious RoleSessionName values.
• Cross-Cloud Vulnerability: The issues highlighted are not exclusive to AWS; similar unmanaged compute environments and associated risks exist across Azure and GCP, requiring a multi-cloud defensive approach.

About the Speaker(s)

Jenko Hwong is a Principal Security Researcher at Huntress Labs. In the talk, he describes himself as the "ranter" who gets passionate about topics and built the initial CTF challenge that led to this research. His role involved designing the challenge, embedding vulnerabilities, and later reversing the Cloud Shell protocol for defensive insights.

Chris Ryan is also a Principal Security Researcher at Huntress Labs. He is characterized by Jenko as the "serious one" due to his lack of social media presence. During the presentation, Chris played the role of the attacker (the CTF participant and later the red teamer), demonstrating the exploitation techniques and persistence methods in a live environment.

Both speakers actively engage in "security stuff," contributing to research that uncovers novel attack vectors and helps organizations better defend their cloud environments.