Sshamble Unexpected Exposures in the Secure Shell

Sshamble Unexpected Exposures in the Secure Shell

Speakers: HD Moore, Co-founder & CTO at runZero; Rob King, Security Researcher at runZero

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=G7GwXu684JY

Overview

In "Sshamble Unexpected Exposures in the Secure Shell," HD Moore and Rob King from runZero delve into the intricate and often overlooked security landscape of the Secure Shell (SSH) protocol. Prompted by the high-profile XZ Utils backdoor (CVE-2024-3094), their research began with an attempt to hunt for the backdoor's author, "Jia Tan," across the internet using SSH public key analysis. This initial investigative thrust, however, unexpectedly unearthed a far broader array of critical vulnerabilities, misconfigurations, and dangerous divergences within SSH implementations across various platforms and products.

The talk serves as a comprehensive exposé on the inherent complexities and potential attack surfaces that arise when a protocol as fundamental and widely adopted as SSH is implemented without rigorous adherence to its strict state machine and security best practices. Moore and King present a compelling case for re-evaluating our understanding of SSH security, moving beyond common assumptions to reveal a landscape riddled with "broken SSH" instances, post-authentication pitfalls, environment control flaws, and even state transition vulnerabilities that can lead to "free shells." Their work underscores the critical need for continuous vigilance and specialized tooling to identify and mitigate these pervasive threats, particularly in a world still reeling from sophisticated supply chain attacks like XZ Utils.

Background

▶ Watch: Introduction and XZ Utils backdoor context (CVE-2024-3094) (0:00)

The impetus for the "Sshamble" research originated from the discovery of the XZ Utils backdoor, a sophisticated supply chain attack (CVE-2024-3094) that came to light in early 2024. This multi-year campaign, initiated in 2021 by a likely state-backed actor operating under the persona "Jia Tan," aimed to inject malicious code into the widely used liblzma library, a component of XZ Utils. The backdoor specifically targeted SSHD (the OpenSSH server daemon) on Debian and RHEL-based distributions by exploiting systemd patches, designed to allow remote attackers to gain unauthorized access. Its near-perfect stealth and the fact it was caught at the eleventh hour by Andres Freund, who noticed unusual CPU usage in sshd, highlighted the profound impact such a compromise could have on global infrastructure. The backdoor made it into rolling releases only, narrowly avoiding widespread deployment in stable distributions.

Motivated by the XZ Utils incident, Moore and King posed a provocative question: if Jia Tan targeted SSH, could SSH itself be used to target Jia Tan? This led them to explore the nuances of SSH public key authentication. They observed that SSH public key authentication is a two-stage process where an SSH client can confirm if a public key is valid for a given user before full authentication. This property, documented in OpenSSH source code comments since at least version 9.8p1, notes that a specific message (PK_OK) is sent only if a user is allowed to log in with a provided public key, even if the full authentication handshake isn't completed. This technique, while supported by tools like Metasploit since 2012, remains largely unknown or underutilized. The researchers aimed to leverage this subtle characteristic to scan the internet for systems accepting public keys associated with the "Jia Tan" persona, whose keys were publicly available on GitHub. This specific hunting exercise, while not yielding the expected results for Jia Tan, became the catalyst for discovering a much broader and more concerning array of SSH vulnerabilities and misconfigurations, forming the core findings of their "Sshamble" project.

Key Findings

▶ Watch: Introducing SSHamble, a custom SSH scanner (4:00)

The "Sshamble" research, initially conceived as a hunt for the XZ Utils backdoor author, "Jia Tan," ultimately uncovered a pervasive landscape of unexpected exposures within SSH implementations. The primary findings can be categorized into several critical areas:

Firstly, the researchers discovered a vast number of "broken SSH" systems across the internet. During their scans, approximately 14 million systems reached the SSH authentication state, but only about 110,000 of these resulted in a successful session. The remaining millions were characterized as "tarpits & buggy systems," indicating widespread misconfigurations, non-standard implementations, or deliberate attempts to mislead scanners. This highlighted a significant blind spot in how the security community perceives and scans for SSH services.

Secondly, a concerning pattern of post-session authentication was identified. Various products were found to allow none authentication (i.e., no initial credentials required) and then implement their interactive login within the SSH session itself. This approach is inherently dangerous, as it exposes the extensive post-authentication attack surface of SSH, including features like port forwarding, environment variable control, and session commands, to unauthenticated or minimally authenticated users.

Thirdly, specific environment control vulnerabilities were found in popular Git platforms like Gitlab, Gitea, and Forgejo. These platforms, while supporting SSH and environment variables, handled them incorrectly, leading to potential command injection. The vulnerability revolved around the GIT_PROTOCOL mechanism, where Git only parsed the version parameter. Although generally considered safe, bugs were found, such as one affecting Go versions less than 1.19.3 (CVE-2022-41716), which could be exploited through specially crafted environment variables to achieve arbitrary command execution.

Fourthly, a critical command injection flaw was discovered in Ruckus Wireless Access Points (APs). Specifically, certain Ruckus APs configured with none SSH authentication would drop to an interactive login session where the password input was directly passed into a shell without proper escapes. This allowed for trivial root access and, alarmingly, approximately 900 such vulnerable devices were still exposed on the internet at the time of the talk. Despite the severity, this bug had no assigned CVE and no security mention in the relevant firmware release notes (fixed in v5.2.1 stable & 6.2.1 tech), raising questions about its prolonged existence and disclosure.

Fifthly, the research unveiled new state transition vulnerabilities in SSH. The Secure Shell protocol relies on a strict state engine, where accepted client message types change as the connection progresses. By systematically requesting a session at every possible state transition, the researchers discovered scenarios leading to "free shells," indicating that some SSH implementations were not correctly enforcing state transitions, allowing unauthorized session establishment. This builds upon historical state transition bugs like CVE-2018-10933 in libssh.

Finally, a significant finding was the divergence of OpenSSH implementations across various platforms. While OpenSSH is the de facto standard, different operating systems and vendors maintain their own forks with varying degrees of modification. These divergences range from "light" (Apple macOS, limited to compatibility and Keychain support) to "moderate" (Debian/Ubuntu and Red Hat Linux, with 36+ and ~60 patches respectively, often for systemd integration) to "major" (PKI-X SSH, forked in 2002 for X509 support, common in networking gear and FIPS appliances) and "extreme" (Microsoft Windows, with over 350 files changed, replacing fork with subprocesses, removing chroot support, sending telemetry, and using Lsa* functions for password auth, notably still affected by Terrapin and not by regreSSHion). These divergences create a fragmented and complex security landscape, where vulnerabilities or fixes in one version may not apply or exist in another, increasing the overall attack surface.

Technical Deep Dive

▶ Watch: Hunting Jia Tan and finding unexpected 'broken SSH' (5:00)

The technical depth of the "Sshamble" talk spans several key areas, from custom scanning tools to specific protocol-level vulnerabilities and platform-specific deviations.

The research began with the Jia Tan hunt, a direct response to the XZ Utils backdoor. The methodology involved three steps:

1. Copying Jia Tan's SSH public keys directly from GitHub repositories where the persona had contributed. These keys were then used as probes.
2. Scanning all IPv4 addresses for port 22 using Zmap, a fast network scanner, to identify active SSH services globally.
3. Utilizing SSHamble to scan for key acceptance. This leverages the two-stage nature of SSH public key authentication, where the server indicates if a public key is valid for a user before requiring the private key for full authentication. The goal was to identify systems configured to accept Jia Tan's public keys, potentially indicating a backdoored system or a system administered by the persona.

The custom SSH scanner, Sshamble, was a crucial tool developed for this research. Built in Go using a "mangled" version of the x/crypto/ssh library, Sshamble was designed for flexibility, speed, and deep SSH protocol interaction. It originated as a half-auth public key scanner, focusing on the initial key acceptance phase, but evolved into a versatile SSH research tool capable of probing various aspects of SSH server behavior. The sshamble scan --checks pubkey-hunt --pubkey-hunt-file jia.keys --input-targets ipv4.txt command-line example illustrates its targeted functionality.

Beyond the Jia Tan hunt, the researchers uncovered critical issues with post-session authentication. This occurs when SSH servers, often in embedded devices or custom applications, allow the none authentication method. Instead of requiring traditional password or key-based authentication upfront, they drop the client into an interactive session, where a custom login prompt or authentication mechanism is presented. This practice is extremely dangerous because, even without full authentication, an SSH session has been established. This opens up the extensive post-authentication attack surface of SSH, including channel requests (e.g., exec, shell, subsystem), port forwarding (direct-tcpip, forwarded-tcpip), and environment variable passing. An attacker could potentially request a shell or initiate port forwarding before the custom interactive login even occurs, bypassing the intended security mechanism.

The environment control vulnerabilities were particularly highlighted in Gitlab, Gitea, and Forgejo. These platforms support SSH for Git operations and allow certain environment variables to be passed. The specific vulnerability leveraged the GIT_PROTOCOL variable. While Git is designed to only parse the version parameter within this variable, a bug in Go's net/http package (CVE-2022-41716), affecting Go versions less than 1.19.3, allowed for HTTP header injection due to improper handling of newlines (\x0a). When combined with SSH's ability to pass environment variables, an attacker could craft an environment variable like GIT_PROTOCOL=version=2:\x0aPATH=C:\Users\gitlab\repositories\rob to inject arbitrary shell commands or manipulate the PATH variable, leading to command execution.

The Ruckus Wireless AP command injection demonstrated a classic vulnerability. In certain Ruckus firmware versions, if SSH was configured for none authentication, it would proceed to an interactive login prompt within the SSH session. Crucially, the input provided for the password at this interactive prompt was passed directly to an underlying shell without proper escaping. This allowed attackers to inject shell metacharacters and execute arbitrary commands, leading to trivial root access. The example echo -n "$(echo pa55wOrd 1>&2)" | sha256sum illustrates how an attacker could inject commands. The fix was implemented in firmware versions v5.2.1 (stable) and 6.2.1 (tech), but the lack of a CVE or security advisory in the release notes meant many administrators might have been unaware of the critical flaw.

The concept of state transition vulnerabilities delves into the core of the SSH protocol's design. SSH employs a strict state engine, where the set of accepted client message types changes as the connection progresses through different phases (e.g., transport, user authentication, connection). OpenSSH and Dropbear servers manage this by remapping tables of command handlers based on the current session state, and message IDs are clamped to specific allowed ranges (e.g., SSH2_MSG_TRANSPORT_MIN to MAX, SSH2_MSG_USERAUTH_MIN to MAX). The researchers explored what happens if a client requests a session (e.g., a shell channel) at an "illegal" or unexpected state transition. Building on historical bugs like CVE-2018-10933 (a libssh bug where the server incorrectly trusted a client-sent USERAUTH_SUCCESS message), their new findings demonstrated that some implementations could be tricked into granting "free shells" by simply asking for a session at various state changes, implying a failure to strictly enforce the state machine.

Finally, the talk meticulously detailed OpenSSH divergence by platform. This fragmentation significantly complicates security.

• Apple macOS exhibits "light" changes, primarily for macOS compatibility, Keychain support, PKCS helper integration, and endpoint event logging.
• Debian/Ubuntu Linux and Red Hat Linux show "moderate" divergence, incorporating systemd support and numerous other patches (36+ for Debian/Ubuntu, ~60 for Red Hat), which can introduce platform-specific bugs or alter behavior.
• PKI-X SSH represents a "major" fork, originating in 2002 for X.509 certificate support. Commonly found in networking gear and FIPS-compliant appliances, it generally follows OpenSSH upstream but with significant deviations.
• Microsoft Windows exhibits "extreme" divergence, with over 350 files changed. This version replaces fork with subprocesses, removes chroot support and log sanitization, logs to Windows Events, and notably sends telemetry containing SSH-encrypted values. Password authentication uses Lsa* functions, and it was noted to still be vulnerable to Terrapin (a prefix truncation attack) while not affected by regreSSHion. This extreme divergence means that security assumptions or patches for standard OpenSSH often do not apply to the Windows implementation.

These technical details collectively paint a picture of SSH as a complex, fragmented, and often insecure protocol ecosystem, necessitating a deeper, more nuanced understanding than commonly assumed.

Demo / Proof of Concept

▶ Watch: Dangers of post-session authentication in various products (6:00)

The talk effectively demonstrated several proof-of-concept (PoC) scenarios, illustrating the real-world impact of their findings. While not a live, interactive demo in the traditional sense, the researchers presented concrete examples and confirmed results for each major vulnerability category.

The initial Jia Tan hunt served as the primary demonstration of Sshamble's capabilities. The researchers copied Jia Tan's specific SSH public keys from GitHub. They then used Sshamble, configured with the pubkey-hunt check and the jia.keys file, to scan the internet for hosts that would accept these keys during the half-authentication process. Although they "got results," these were not the expected Jia Tan targets but rather a vast number of "broken SSH" systems, inadvertently demonstrating the widespread prevalence of non-standard SSH behavior and the effectiveness of Sshamble in identifying these anomalies.

The environment control vulnerability in Gitlab, Gitea, and Forgejo was demonstrated through the crafted GIT_PROTOCOL variable. The example GIT_PROTOCOL=version=2:\x0aPATH=C:\Users\gitlab\repositories\rob showcased how a newline injection, combined with the Go < 1.19.3 vulnerability (CVE-2022-41716), could manipulate the PATH environment variable. This effectively demonstrated a path to arbitrary command execution by altering how the Git client invoked subsequent commands.

For the Ruckus Wireless AP command injection, the PoC was conceptualized around the interactive login process. The core idea was that if none authentication was allowed, dropping into an interactive shell, an attacker could simply input a specially crafted string as the "password" that contained shell metacharacters. The example echo -n "$(echo pa55wOrd 1>&2)" | sha256sum clearly illustrates how an injected command (echo pa55wOrd 1>&2) could be executed before the legitimate sha256sum command, leading to command execution and trivial root access. The fact that approximately 900 such devices were still exposed on the internet underscored the severity and ease of exploitation.

Finally, the state transition vulnerabilities were demonstrated by the researchers asking for a session at "every possible state transition." This systematic probing revealed instances where SSH implementations would grant "free shells" even when the connection was not in the expected state for session establishment. While the exact code for this "asking for a session" wasn't shown, the conceptual demonstration highlighted a fundamental flaw in state machine enforcement, allowing an attacker to bypass authentication or privilege checks by manipulating the timing and type of messages sent. This effectively showed that some SSH servers were not robustly validating the connection state before granting privileged access.

These PoCs collectively illustrated that the issues were not theoretical but represented tangible security flaws with direct implications for system compromise, ranging from initial access to arbitrary code execution.

Defensive Implications

▶ Watch: Gitlab, Gitea, Forgejo environment control vulnerabilities (7:00)

The findings from "Sshamble" present a critical wake-up call for defenders, highlighting numerous vectors for SSH-related compromise that extend beyond traditional brute-force or weak credential attacks. Addressing these implications requires a multi-faceted approach:

1. Comprehensive Asset Inventory and SSH Scanning: Defenders must gain a deep understanding of their SSH exposure. This means not just identifying systems running SSH on port 22, but actively scanning with tools like Sshamble (or custom scripts leveraging its principles) to detect non-standard behaviors, post-session authentication setups, and state transition vulnerabilities. Identifying "broken SSH" systems, tarpits, and non-compliant implementations is crucial.

1. Eliminate "None" Authentication and Post-Session Logins: Any SSH server configured to allow none authentication or to implement interactive logins within the SSH session after minimal or no initial authentication should be immediately reconfigured. This practice creates a massive attack surface for post-authentication features like port forwarding, environment variable injection, and shell requests, even for unauthenticated clients. Standard SSH authentication methods (public key, strong passwords with MFA) should be enforced strictly at the protocol level.

1. Patch and Update Regularly, with Awareness of Divergence: While general advice, the talk emphasizes its importance due to OpenSSH fragmentation. Defenders must not only patch OpenSSH but also understand the specific version and patches applied to their OS (e.g., Debian, RHEL, macOS, Windows) and proprietary devices (e.g., Ruckus APs, PKI-X SSH). Updates for one platform may not address vulnerabilities in another. For devices like Ruckus APs, actively monitoring vendor security advisories (even when CVEs are not assigned) and immediately applying firmware updates (e.g., Ruckus v5.2.1 or 6.2.1) is vital to fix known command injection flaws.

1. Secure Environment Variable Handling: For applications that integrate with SSH and process environment variables (like Gitlab, Gitea, Forgejo), ensure that the underlying software (e.g., Go language runtime) is patched to the latest versions (e.g., Go >= 1.19.3 to mitigate CVE-2022-41716). Developers of custom SSH applications must implement robust input validation and sanitization for all environment variables, preventing injection attacks.

1. Strict SSH Configuration and Hardening:

• Disable unnecessary features: Turn off features like AllowTcpForwarding, AllowStreamLocalForwarding, PermitUserEnvironment, X11Forwarding, and AgentForwarding if not explicitly required.
• Restrict SSH access: Limit SSH access to specific users, IP addresses, and enforce PermitRootLogin no.
• Monitor SSH logs: Implement robust logging and monitoring for SSH connections, failed authentication attempts, and unusual activity. Look for attempts to request shells or channels at unexpected states.

1. Review Custom SSH Implementations: Organizations using custom SSH servers or embedded devices with SSH functionality must conduct thorough security audits. These custom implementations are often the source of state transition vulnerabilities, post-session authentication flaws, and unique command injection points. Ensure they strictly adhere to the SSH state machine and validate all client messages.

1. Supply Chain Vigilance: The XZ Utils backdoor underscores the need for continuous vigilance in the software supply chain. While not a direct defensive action, it serves as a reminder to vet upstream dependencies and monitor for unusual activity, even in fundamental utilities.

By taking these steps, defenders can significantly reduce their exposure to the unexpected and often subtle vulnerabilities lurking within the complex world of Secure Shell implementations.

Key Takeaways

• The XZ Utils backdoor (CVE-2024-3094) highlighted the profound impact of supply chain attacks targeting core infrastructure like SSH, prompting deeper research into the protocol's security.
• SSH public key authentication has a little-known two-stage process where servers indicate key validity before full authentication, a property that can be leveraged for reconnaissance and vulnerability scanning.
• The Sshamble tool, built in Go, evolved from a half-auth public key scanner into a powerful SSH research tool capable of identifying non-standard and vulnerable SSH implementations across the internet.
• A significant amount of "broken SSH" exists online, with millions of systems reaching the authentication state but exhibiting buggy behavior, tarpits, or non-standard protocols.
• Post-session authentication (allowing none auth then interactive login) is dangerous, exposing the full SSH post-authentication attack surface (e.g., port forwarding, environment control) to unauthenticated users.
• Vulnerabilities like environment control flaws (e.g., in Gitlab/Gitea/Forgejo via GIT_PROTOCOL and CVE-2022-41716) and command injection (e.g., Ruckus Wireless APs) demonstrate how seemingly minor input handling errors can lead to trivial root access.
• State transition vulnerabilities can lead to "free shells" by exploiting implementations that fail to strictly enforce the SSH state machine, allowing session requests at unexpected points in the protocol handshake.
• OpenSSH implementations diverge significantly across platforms (macOS, Linux distros, Windows, PKI-X SSH), creating a fragmented security landscape where patches and vulnerabilities are not universally applicable.

About the Speaker(s)

HD Moore is a co-founder and CTO at runZero, a company specializing in asset inventory and network visibility. He is widely recognized for his extensive contributions to the cybersecurity community, most notably as the creator of the Metasploit Framework. His work consistently focuses on identifying and understanding systemic vulnerabilities in widely used technologies.

Rob King is a security researcher at runZero. His research, including the work presented in "Sshamble," focuses on uncovering novel vulnerabilities and developing tools to assess the security posture of network devices and protocols. His contributions to the talk highlight deep technical expertise in analyzing and exploiting complex systems like SSH.