Secure Shells in Shambles

Secure Shells in Shambles

Speakers: HD, Rob

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=N-gZavGUKWs

Overview

This talk, "Secure Shells in Shambles," delves into the complex and often fragmented ecosystem of Secure Shell (SSH), a protocol critical for internet administration. Presented by HD and Rob, the session highlights the evolution of SSH, its ubiquitous presence, and a range of vulnerabilities—both historical and newly discovered—that challenge its perceived security. The speakers emphasize that while OpenSSH and Dropbear dominate the landscape, a significant "long tail" of custom and embedded SSH implementations, particularly in networking, storage, OT/ICS, and development tools, presents a fertile ground for novel attacks.

The presentation aims to demystify SSH by breaking down its protocol phases and contrasting its security model with that of TLS. It exposes critical issues like inadequate host key management, the challenges posed by SSH's extensive post-authentication attack surface, and the implications of recent high-profile vulnerabilities such as the Terrapin attack and the XZ Utils backdoor. A key contribution of the talk is the introduction of Shamble, a new research tool designed to identify and exploit vulnerabilities across the diverse SSH environment, offering a practical approach for security assessments and further research.

Background

▶ Watch: Talk overview: SSH ecosystem, attacks, and defense (0:00)

SSH, originally created by Tatu Ylonen in 1995 as SSHv1, has undergone significant evolution and fragmentation over nearly three decades. Its lineage includes prominent forks like OpenSSH and Dropbear, which together account for approximately 90% of internet-facing SSH daemons. However, the remaining 10% comprises a "long tail" of highly specialized and often proprietary implementations. These include SSH servers embedded in network devices (Cisco, NetScreen), storage solutions, HP ILO, Lancom, Zyxel, and various OT/ICS equipment (Siemens, Netpower, Mocana). Crucially, many of these custom implementations are not based on OpenSSH or have diverged significantly, leading to unique vulnerabilities and inconsistent security postures.

SSH stands as the second most common administration protocol on the internet, surpassed only by HTTP. It is more prevalent than RDP or SMB, serving as the backbone for remote administration across cloud providers, major operating systems, embedded devices, servers, and mobile platforms. The protocol operates in two distinct phases: an initial cleartext phase for version exchange and key exchange init, followed by an encrypted phase post-Diffie-Hellman, encompassing authentication and channel communication. SSH banners, such as SSH-2.0-OpenSSH-9.8p1, provide valuable pre-authentication information about the protocol version, product name, and package details, alongside advertised ciphers and MACs.

Unlike TLS, which typically relies on a centralized Certificate Authority (CA), SSH commonly uses host keys for authentication, a decentralized model that introduces significant management challenges. While SSH certificates can be implemented, they are not widely adopted. Another critical distinction highlighted is SSH's cipher negotiation mechanism: during key exchange, the first cipher sent by the client that is supported by the server is used, rather than the cryptographically strongest or preferred option. This behavior was central to the effectiveness of the Terrapin attack. Compliance standards, while robust for TLS, often provide only superficial guidance for SSH, focusing on cryptographic algorithms but neglecting the vital aspect of host key management. Over the years, SSH has also seen the introduction of new extensions for protocols, key exchange mechanisms, and signing algorithms, alongside enhanced authentication methods like Multi-Factor Authentication (MFA) via challenge-response, keyboard-interactive prompts, and hardware keys such as YubiKeys. These advancements, while improving security, also expand the protocol's complexity and potential attack surface.

Key Findings

▶ Watch: SSH's ubiquity and criticality as an admin protocol (2:00)

The talk uncovers several critical issues and vulnerabilities spanning the SSH ecosystem, emphasizing that SSH's ubiquity and complexity have created a challenging security landscape.

1. Fragmentation and the "Long Tail" Vulnerabilities: A significant finding is the security risk posed by the approximately 10% of internet-facing SSH servers that are not OpenSSH or Dropbear based. These custom implementations, found in firewalls, network devices, storage, and OT/ICS equipment, often lag in security updates and may contain unique, unpatched vulnerabilities. Even source code forges like Git and Gerrit utilize embedded SSH servers through libraries, extending this vulnerable "long tail."

1. Windows OpenSSH CRLF Bug: A specific, concerning vulnerability was identified in the Windows version of OpenSSH. Due to an attempt to make a critical cryptographic function CRLF compatible, two lines of code were added that effectively backdoored it. This introduced a timing-unsafe operation, a one-byte out-of-bounds memory read, and an unequal byte match in a function used for every encrypted byte buffer comparison. While difficult to exploit, it demonstrated a significant lapse in security, which Microsoft reportedly declined to fix.

1. Terrapin Attack (CVE-2023-48795): The presentation reiterates the significance of the Terrapin attack, a man-in-the-middle vulnerability that allows attackers to downgrade SSH connection security by manipulating sequence numbers. This attack specifically leverages the fact that SSH uses the first mutually supported cipher during key exchange, not necessarily the most secure one, enabling prefix truncation attacks on the secure channel.

1. XZ Utils Backdoor (CVE-2024-3094): The talk references the high-profile XZ Utils backdoor as a stark reminder of supply chain risks impacting OpenSSH. This multi-year campaign aimed to inject malicious code into OpenSSH via systemd patches in specific Linux distributions (RHEL, Debian rolling releases), allowing for remote code execution under specific conditions. Its discovery by Andres Freund highlighted the importance of vigilance against subtle performance anomalies.

1. Extensive Post-Authentication Attack Surface: Despite significant efforts to secure the pre-authentication phase of SSH, the protocol's feature richness—including interactive shells, command execution, file transfers, TCP/Unix/X11/agent forwarding—creates a substantial post-authentication attack surface. Once authenticated, an attacker has numerous vectors for further exploitation, especially against restricted shells or misconfigured forwarding.

1. Poor Host Key Management: A persistent and widespread issue is the inadequate management of SSH host keys. These unique key pairs identify machines but are frequently duplicated across VM images or rarely rotated, leading to thousands of machines sharing the same host key. This practice undermines trust and simplifies impersonation attacks, further complicated by the lack of a central CA for SSH in most deployments.

1. Public Keys as Identities: The discussion points out that SSH public keys, while mathematically designed to be public, increasingly serve as public identities. Scraped from platforms like GitHub or Launchpad, these keys can be used to identify individuals, potentially exposing more information than users realize.

Technical Deep Dive

▶ Watch: Understanding SSH protocol phases and banner (2:30)

The SSH protocol is foundational for secure remote administration, yet its internal mechanisms and deployment complexities reveal numerous attack vectors. The protocol begins with a cleartext phase involving a simple version exchange where both client and server announce their supported SSH protocol version (e.g., SSH-2.0). This is followed by the key exchange init (KEXINIT), where both parties advertise their preferred cryptographic algorithms for key exchange, encryption, and message authentication codes (MACs).

A critical aspect of SSH's security, and a point of vulnerability, lies in its key exchange (KEX) process, typically involving Diffie-Hellman. Following a successful key exchange, all subsequent communication occurs over an encrypted channel. The session ID, derived during the key exchange, is crucial; it's incorporated into the signature blob for public key authentication, binding the authentication to the specific session and host, thus preventing simple relay attacks.

SSH is designed as a multiplexer, allowing multiple logical "channels" (e.g., interactive shells, command execution, file transfers, various forms of port forwarding like TCP, Unix, X11, and agent forwarding) to operate concurrently over a single underlying TCP connection. This multiplexing, while efficient, significantly expands the post-authentication attack surface.

The Terrapin attack (CVE-2023-48795) exposed a weakness in how SSH negotiates ciphers. Unlike a "best-fit" approach, SSH uses the first cipher proposed by the client that is supported by the server. This allows a man-in-the-middle attacker to potentially downgrade the connection to a less secure cipher if the client offers a weak cipher first, even if stronger ones are available. By manipulating sequence numbers, Terrapin can effectively truncate messages, bypassing integrity checks in certain configurations.

A notable vulnerability discussed was the Windows OpenSSH CRLF bug. In an attempt to make the memcmp function – a critical function for comparing encrypted byte buffers and verifying signatures – compatible with CRLF line endings, Microsoft introduced a flaw. This change involved adding two lines of code that rendered the comparison timing-unsafe, allowed a one-byte out-of-bounds memory read, and resulted in an unequal byte match. This bug meant that an encrypted blob containing specific \r\n sequences could cause the function to read outside its intended memory range, potentially bypassing cryptographic checks by consuming two bytes of entropy. While difficult to reliably exploit due to the need for specific buffer conditions, it represented a significant security regression in a core component. Microsoft's reported decision not to fix it further underscores the challenge of securing the diverse SSH ecosystem.

SSH has evolved to include various extensions, enhancing its capabilities and security. These include support for new key exchange mechanisms, signing algorithms, and post-quantum cryptography. Authentication has seen significant improvements with Multi-Factor Authentication (MFA) options like challenge-response and keyboard-interactive methods, as well as integration with hardware security keys (e.g., YubiKeys) that require user presence or PIN verification. The adoption of SSH certificates for both user and host authentication offers a powerful, centralized alternative to managing individual host keys, enabling more robust key rotation and trust management. However, these are still underutilized. Pre-authentication banners have become more common, often disclosing significant system information before any authentication occurs. Furthermore, specific protocol extensions, such as [email protected], are advertised and negotiated during the initial protocol phase, providing further fingerprinting opportunities.

Defenders can leverage various techniques to identify SSH servers. TCP stack fingerprinting can often reveal the underlying operating system. The advertised key exchange algorithms, ciphers, and MACs provide cryptographic profiles. The content of pre-authentication banners and the specific wording of enhanced authentication questions (e.g., for MFA prompts) can all be used to build a detailed fingerprint of the SSH server and its configuration.

Demo / Proof of Concept

▶ Watch: SSH compliance gaps, especially host key management (4:30)

The talk introduces Shamble, a newly released research and assessment tool designed to explore the vast and fragmented SSH ecosystem for vulnerabilities. Available live at shamble.com and on GitHub, Shamble is presented as both a practical assessment utility and an extensible research platform.

Shamble's core functionality is built upon a forked Go crypto SSH library. This strategic fork allows the tool to perform "terrible things" to SSH connections while maintaining ease of patching and future maintenance. The tool is capable of executing a range of attacks, including:

• Pre-authentication state transition bugs: These target vulnerabilities in the initial cleartext and key exchange phases of the SSH protocol, before a user has even attempted to authenticate.
• Post-authentication attacks: Shamble includes baked-in exploits for known post-authentication vulnerabilities, such as those found in Gogs (a self-hosted Git service), which often runs embedded SSH servers.
• "Larval Session Environment" Exploitation: A unique feature of Shamble is its ability to open an SSH session without immediately requesting a pseudo-terminal (PTY) or spawning a shell. By setting session.interact to nothing, Shamble enters what the speakers call a "larval session environment." In this state, the tool can interact with the SSH server prior to a shell being granted, allowing for exploration and exploitation of restricted shells or other pre-shell command execution opportunities. An example cited is authenticating to a NetScreen box as "nobody" and sending commands without receiving direct output, indicating a pathway to attack restricted environments.

The tool aims to simplify the process of testing SSH environments, which can often be cumbersome with existing libraries like Paramiko. Shamble provides an interactive shell by default but offers the flexibility to operate in the larval session environment, making it a valuable asset for security researchers and penetration testers looking to uncover hidden vulnerabilities in diverse SSH implementations, particularly those in the "long tail" of custom devices.

Defensive Implications

▶ Watch: Evolution of SSH: new features and authentication methods (4:50)

Securing SSH requires a multi-faceted approach that goes beyond basic password hygiene, addressing both fundamental protocol weaknesses and common misconfigurations. The speakers offer several actionable recommendations for defenders:

1. Prioritize Public Key Authentication: Eliminate password-based authentication (auth none with left beef) wherever possible. Public key authentication is significantly more secure. Implement robust key management practices, including regular key rotation and secure storage of private keys.

1. Strictly Restrict SSH Agent Forwarding: While convenient, SSH agent forwarding can be a significant security risk if not properly managed. Defenders should restrict the destinations to which an agent can forward keys. Furthermore, when generating public keys, specify options that disable capabilities like X11 forwarding or agent forwarding for that particular key, limiting its potential misuse.

1. Utilize Forced Commands: For specific SSH keys or users, implement forced commands. This feature restricts the user to executing only a predefined command upon authentication, preventing arbitrary shell access. This is particularly useful for automation scripts or limited administrative tasks.

1. Embrace SSH Certificates for Host and Public Keys: Transitioning to SSH certificates for both server host keys and user public keys significantly enhances security and manageability. A central SSH CA can sign host keys, simplifying distribution and rotation, and providing a more robust trust model than relying on individual known_hosts files. Similarly, user public keys signed by a CA streamline user authentication management, especially in large environments. The speakers highlight this as an "amazingly powerful" and "hidden secret" feature of SSH.

1. Be Mindful of SSH as a Transport Layer: SSH is frequently used as a versatile transport layer, almost like a lightweight VPN, for tunneling various types of traffic. While beneficial, this usage expands the attack surface. Defenders must apply the same scrutiny to SSH tunnels and forwarded ports as they would to any other network service, ensuring proper access controls and monitoring.

1. Improve Host Key Management: While not a direct action item, the pervasive issue of duplicated and unrotated host keys is a critical area for improvement. Organizations should ensure that each SSH server has a unique host key and that these keys are rotated regularly, breaking dependency on burned-in VM images.

1. Monitor for Anomalies: The XZ Utils backdoor highlighted the importance of monitoring system behavior for subtle anomalies, such as unexpected CPU usage by SSH processes. Implementing robust logging and monitoring solutions can aid in early detection of sophisticated supply chain attacks.

By implementing these advanced defensive measures, organizations can significantly improve their SSH security posture, mitigating risks from both long-standing vulnerabilities and newly discovered attack vectors.

Key Takeaways

• SSH Ecosystem Fragmentation is a Major Risk: Beyond OpenSSH and Dropbear, a vast "long tail" of custom and embedded SSH implementations in network devices, OT/ICS, and development tools presents unique, often unpatched vulnerabilities due to inconsistent updates and proprietary codebases.
• Pre-Auth Security Isn't Enough: Despite significant efforts to secure the pre-authentication phase, SSH's rich feature set (channels, forwarding, PTYs) creates a large post-authentication attack surface that is often overlooked.
• Recent High-Profile Vulnerabilities Underscore Systemic Issues: The Terrapin attack (cipher negotiation flaw), the Windows OpenSSH CRLF bug (critical crypto function compromise), and the XZ Utils backdoor (supply chain attack) demonstrate that even core SSH components are susceptible to complex and impactful vulnerabilities.
• Host Key Management is Critically Flawed: Widespread duplication and infrequent rotation of SSH host keys undermine trust and simplify impersonation, highlighting a persistent gap in security practices that is exacerbated by the lack of centralized certificate authorities.
• Advanced SSH Features are Underutilized: Centralized SSH certificates for both hosts and users, MFA with hardware keys, and forced commands offer powerful security enhancements that are often neglected but can significantly improve SSH security posture.
• Shamble Provides a New Tool for Assessment: The release of Shamble offers researchers and defenders a specialized tool to probe the SSH ecosystem, uncover pre-authentication state transition bugs, and exploit post-authentication vulnerabilities, including a unique "larval session environment" for attacking restricted shells.

About the Speaker(s)

The talk was delivered by two speakers, referred to as HD and Rob throughout the presentation. While specific titles and companies are not detailed in the transcript or metadata, their in-depth knowledge of SSH's historical evolution, cryptographic intricacies, and practical exploitation techniques indicates their expertise in network security and vulnerability research. They demonstrated a comprehensive understanding of SSH's protocol mechanics, its diverse implementations, and the current threat landscape, culminating in the release of their research tool, Shamble.