Are Your Backups Still Immutable, Even Though You Can't Access Them?

Are Your Backups Still Immutable, Even Though You Can't Access Them?

Speakers: Ryan Kane, Pen Tester, Red Teamer, Northwestern Mutual; Roshank, Pen Tester, Red Teamer, Purple Teamer, Security Researcher, Northwestern Mutual

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=1o8nZGQ5idc

Overview

In an era where ransomware attacks are not just encrypting data but actively seeking to cripple recovery efforts, the integrity and accessibility of backup systems have become paramount. This talk, "Are Your Backups Still Immutable, Even Though You Can't Access Them?", delivered by Ryan Kane and Roshank of Northwestern Mutual at Black Hat USA, dives deep into the security posture of modern immutable backup solutions. The speakers present a critical perspective on the common misconception that data immutability alone guarantees ransomware resilience, highlighting that even if data cannot be altered, its unavailability can be just as devastating.

The core of their research explores the attack surfaces beyond direct data manipulation, focusing instead on the underlying infrastructure that hosts and manages these critical backup repositories. By examining both physical and cloud-based solutions, Kane and Roshank demonstrate how sophisticated threat actors might bypass immutability guarantees by disrupting access to the backups themselves. Their findings provide invaluable insights for organizations grappling with ransomware threats, urging a shift in defensive strategies to encompass the entire backup ecosystem, not just the data's immutability flag.

This article will dissect the speakers' methodology, technical findings, and practical recommendations, emphasizing the need for comprehensive security testing and robust access controls for backup infrastructure. The work underscores that true resilience against ransomware requires a holistic approach, extending beyond data protection to ensure operational continuity and swift recovery capabilities.

Background

▶ Watch: Ransomware scenario and the myth of immutable backups (0:00)

The increasing sophistication of ransomware groups has dramatically shifted the focus of cyber defense. Modern ransomware campaigns, as highlighted by intelligence reports from entities like CrowdStrike (mentioning groups like Alpha Spider using Alpha V ransomware), no longer simply encrypt production data. Instead, they actively target and destroy backups, understanding that eliminating an organization's ability to restore forces a ransom payment. This tactic underscores the critical importance of secure, resilient backup solutions.

To counter this threat, the industry has widely adopted Write Once, Read Many (WORM) devices and technologies. WORM systems are designed to allow data to be written once and then locked, preventing any subsequent modification or deletion for a defined period. This "immutability flag" ensures a clean, untampered dataset remains available for recovery operations, whether from a ransomware attack or a general disaster. These systems operate through mechanisms known as retention lock or vault lock, which define the duration for which data remains immutable. Some solutions, for instance, allow data to be locked for up to 70 years, providing extreme long-term protection.

Retention locks typically operate in two modes:

1. Governance Mode: This is the less strict of the two. While data is immutable, the immutability flag can be undone before the expiration period if certain conditions are met, often requiring specific administrative privileges or processes.
2. Compliance Mode: This is the much stricter, and current industry-standard, mode. Once data is set with retention lock in compliance mode, it is absolutely locked until the defined time period expires. Neither root nor administrative privileges can bypass this lock, short of physically wiping the entire device and starting from scratch. This mode is designed to meet stringent regulatory requirements for data retention and integrity.

Organizations, including Northwestern Mutual, rely heavily on these WORM solutions for ransomware resilience and general disaster recovery. The expectation is that these solutions will always provide timely access to backups for restoration. However, the speakers' research challenges this assumption by exploring vulnerabilities in the infrastructure surrounding these inherently immutable data stores.

Key Findings

▶ Watch: Explaining WORM, retention lock, and compliance modes (1:50)

The central revelation of Kane and Roshank's research is a paradigm shift in how organizations should approach the security of their immutable backups. Their initial hypothesis, like many in the industry, was to attempt to directly attack the immutability mechanisms of these systems. However, they quickly discovered that these solutions are exceptionally robust in their primary function: preventing data modification or deletion once locked in compliance mode. As the speakers noted, "attacking the immutability itself was pretty pointless. That's what these solutions do, right? You had one job. They do it really well."

This realization led to their key finding: while the data itself might be truly immutable, the infrastructure hosting, managing, and providing access to that data is often not. The more productive and successful attack vector lies in targeting the servers and management interfaces that control these backup systems. If an attacker can disrupt access to the immutable backups, render the backup servers inoperable, or compromise the administrative controls, the practical outcome for an organization is the same as if the data itself had been destroyed – the inability to restore.

Their research specifically focused on three different vendor solutions, representing both physical and cloud deployments:

1. Dell EMC Data Domain Operating System (DDOS): A physical backup appliance, tested with version 7.7 and retention lock in compliance mode.
2. IBM DSA 1000: Another physical backup appliance.
3. AWS Backup: A cloud-based backup service.

The successful exploitation paths centered on gaining control over the management plane or the operating system of the backup appliance, rather than attempting to modify the WORM-protected data volumes directly. This highlights a critical blind spot in many organizations' backup security strategies, which often over-rely on the immutability feature without adequately securing the surrounding operational layers.

Technical Deep Dive

▶ Watch: Attacking infrastructure hosting backups, not immutability itself (3:38)

The technical deep dive into the Dell EMC Data Domain Operating System (DDOS) version 7.7, configured with retention lock compliance mode, formed a significant portion of the speakers' research. The choice of DDOS was strategic, given its widespread enterprise adoption and the stringent nature of its compliance mode, which makes data virtually indestructible once locked.

Initial research into DDOS revealed a scarcity of public security information. Beyond a single known CVE for a local privilege escalation, there was little pre-existing work. This necessitated a from-scratch approach to understanding the system's architecture and potential vulnerabilities. The primary interface for interacting with DDOS is the DD shell, a specialized, locked-down Unix-like environment. While the system also features a web interface, the research primarily focused on the DD shell due to its direct access to system commands.

The DD shell is designed to be highly restrictive, offering hundreds of commands but severely limiting their impact to within the "jail" session. Attempts to escape the shell or perform destructive actions like overwriting data or unmounting file systems were consistently met with failure. Even sophisticated attacks, such as attempting to manipulate time synchronization with a rogue NTP server (which could potentially alter retention periods if not properly handled by the system), proved ineffective against Dell's robust design. The speakers concluded that "Dell had thought of everything we thought we could do" regarding direct attacks on data or core system functions.

However, the team identified a critical path for deeper system access: the system engineer (SE) mode and the underlying bash shell. Accessing the bash shell is a multi-step, highly privileged operation, typically intended for Dell support personnel or high-level administrators for system maintenance. The process involves:

1. Executing the system bash enter command from the DD shell.
2. Providing an initial set of administrative credentials, often sourced from vaulted Active Directory (AD) accounts provided by the backup team.
3. Being prompted for a second set of credentials for a designated security role user. This enforces a crucial separation of duties, requiring agreement from two distinct individuals to proceed.
4. Crucially, obtaining a temporary bash key from Dell support. This key is time-limited, typically lasting only a few hours, and is required for full bash shell access.

The speakers likened this multi-factor, multi-person authorization process to the "WarGames" scenario, where two individuals must simultaneously turn their keys to launch nuclear missiles. This design is intended to prevent a single compromised administrator from gaining unfettered control. While the speakers were initially granted admin-level vaulted AD accounts, they had to independently gain access to the local accounts with admin rights and navigate this complex bash shell access procedure.

Once inside the bash shell, an attacker with sufficient privileges and the temporary key would have significantly expanded capabilities compared to the locked-down DD shell. While the research did not explicitly detail successful destructive actions from within the bash shell against the immutable data, gaining this level of control over the underlying operating system of the backup appliance presents a severe risk. It could potentially allow an attacker to:

• Disable network interfaces, rendering the device unreachable.
• Manipulate system services, causing operational outages.
• Interfere with the backup software agents running on the appliance.
• Create new administrative accounts or modify existing ones.
• Potentially brick the device or render it unbootable, even if the data itself remains immutable.

The inability to access or manage the device effectively makes the immutable data as good as lost for operational recovery purposes. This technical deep dive underscores that the attack surface extends far beyond the data itself, encompassing the entire management and operational infrastructure of these critical systems.

Demo / Proof of Concept

▶ Watch: Dell EMC DDOS case study: initial research findings (4:45)

The talk included a conceptual demonstration of how to gain privileged access to the Dell EMC Data Domain Operating System (DDOS) through the bash shell, which serves as a critical proof of concept for their findings. While the transcript doesn't detail a live, interactive demo, the speakers meticulously walked through the steps required, implying a visual representation was part of their presentation.

The demonstration focused on the multi-layered security controls designed to protect the DDOS bash shell and how an attacker, despite these controls, could potentially gain access if certain conditions are met. The process, as described, is as follows:

1. Initial Access via DD Shell: An attacker first needs to gain administrative access to the DD shell, which is the primary command-line interface for the Data Domain appliance. The speakers noted they were provided with admin-level vaulted AD accounts by the backup team for their testing.
2. Executing system bash enter: From within the DD shell, the command system bash enter is executed. This initiates the process of transitioning from the restricted DD shell to the more powerful underlying bash environment.
3. First Credential Prompt (Admin): The system prompts for the first set of administrative credentials. This would typically be the credentials of a system administrator.
4. Second Credential Prompt (Security Role): Following successful authentication with the first set of credentials, the system then prompts for a second set of credentials, specifically for a user with a designated security role. This implements a mandatory separation of duties, often referred to as a "two-person rule," meaning two distinct administrative roles must authorize the bash shell access. The speakers emphasized that, ideally, these should be different people or different vaulted credentials.
5. Bash Key Requirement: Even with both sets of credentials provided, full bash shell access is not immediately granted. A temporary bash key is required. This key is typically provided by Dell support during a support call and is time-limited, valid only for a few hours. This introduces an external dependency and a limited window for highly privileged operations, aiming to prevent persistent unauthorized access.

The "War Games" analogy was used to illustrate this multi-factor, multi-person authorization: "both users have to turn their key at the same time to launch the nukes, they both have to agree to it." The implication is that if a security user does not agree, the system administrator would ideally be blocked. However, the speakers' research suggests that if an attacker manages to compromise both sets of credentials and obtain a valid bash key (perhaps through social engineering, insider threat, or compromising a Dell support channel), they could bypass these controls.

While the immutable data itself remains protected, gaining bash shell access allows an attacker to control the operating system of the backup appliance. This could lead to a denial-of-service scenario by shutting down the device, disabling networking, or corrupting its configuration, effectively making the immutable backups inaccessible for restoration. The proof of concept demonstrates that even with robust immutability, the operational availability of backups hinges on the security of the underlying management infrastructure and the strict enforcement of access controls.

Defensive Implications

▶ Watch: Gaining bash shell access in Dell EMC DDOS (6:05)

The research presented by Ryan Kane and Roshank provides critical insights for organizations seeking to strengthen their ransomware resilience, particularly concerning immutable backup solutions. The primary defensive implication is a shift in focus from solely trusting data immutability to comprehensively securing the entire backup ecosystem.

Here are key defensive implications and recommended actions:

1. Secure the Management Plane: The most significant takeaway is that the management interfaces and underlying operating systems of backup appliances (both physical and cloud-based) are prime targets. Organizations must apply the highest levels of security to these systems:

• Strong Authentication and Authorization: Implement multi-factor authentication (MFA) for all administrative access to backup systems. Enforce robust password policies.
• Principle of Least Privilege: Grant administrators only the minimum necessary permissions required for their roles. Avoid using highly privileged accounts for routine tasks.
• Network Segmentation: Isolate backup management interfaces on dedicated, highly restricted network segments, separate from the production network and other administrative networks. Implement strict firewall rules to limit access to only authorized sources and protocols.
• Vulnerability Management: Regularly scan and patch backup appliance operating systems and management software. While direct immutability attacks may fail, a local privilege escalation CVE (like the one mentioned for DDOS) could be the entry point to gain the necessary access for infrastructure disruption.

1. Strict Separation of Duties for Critical Operations: The "two-person rule" observed in the DDOS bash shell access is a valuable control. Organizations should:

• Implement Dual Authorization: For any highly sensitive operations on backup systems (e.g., accessing the bash shell, modifying core configurations, initiating disaster recovery procedures, or changing retention policies in governance mode), require approval from two distinct individuals or roles. This significantly complicates an attacker's ability to act unilaterally.
• Segregate Credentials: Ensure that credentials for different administrative roles (e.g., system admin vs. security admin) are distinct and managed separately, ideally in different privileged access management (PAM) vaults.

1. Manage Vendor Support Access and Keys: The reliance on vendor-provided temporary keys (like the DDOS bash key) introduces a unique risk.

• Strict Protocol for Key Usage: Establish clear, documented protocols for when and how vendor support keys are obtained and used. This should include mandatory supervision by internal security personnel.
• Audit and Monitor: Thoroughly log and audit all uses of vendor support keys and the actions performed during their validity period.
• Minimize Usage: Only use these keys when absolutely necessary and for the shortest possible duration.

1. Include Backup Infrastructure in Red Teaming and Penetration Testing: The speakers explicitly recommend including backup environments in the testing landscape.

• Simulate Ransomware Scenarios: Conduct exercises that mimic ransomware attacks, not just encrypting data but also attempting to disable or corrupt backup systems. This includes testing the ability to disrupt access to immutable backups.
• Test Access Controls and Privilege Escalation: Specifically target the management interfaces, network access, and user accounts associated with backup solutions to identify potential weaknesses.
• Verify Recovery Procedures: Beyond technical testing, regularly test the entire recovery process, from accessing backups to restoring critical systems, to ensure operational readiness.

1. Monitor for Anomalous Activity: Implement robust logging and monitoring for all backup systems. Look for:

• Unusual login attempts or access patterns to management interfaces.
• Attempts to modify retention policies or delete snapshots (especially in governance mode).
• Changes to network configurations or system services on backup appliances.
• Access from unauthorized IP addresses or during off-hours.

By adopting these defensive strategies, organizations can move beyond a superficial reliance on "immutability" and build a truly resilient backup infrastructure that can withstand sophisticated ransomware attacks aimed at crippling recovery capabilities.

Key Takeaways

• Immutability Alone is Insufficient: While WORM technologies effectively prevent data modification, they do not guarantee accessibility or operational continuity in the face of sophisticated attacks targeting the underlying infrastructure.
• Target the Infrastructure, Not Just the Data: Ransomware groups are shifting tactics to disable access to backups by compromising the management plane, operating systems, or network connectivity of backup solutions, even if the data itself is immutable.
• Prioritize Backup Infrastructure Security: Treat backup servers, appliances, and cloud backup accounts with the highest level of security, equivalent to or even exceeding that of production systems.
• Enforce Strict Access Controls: Implement multi-factor authentication, principle of least privilege, network segmentation, and robust separation of duties (e.g., two-person rule for critical operations) for all backup management interfaces.
• Manage Vendor Access Risk: Be acutely aware of the risks associated with vendor support access and temporary keys (like the Dell EMC bash key). Establish strict protocols for their usage, monitoring, and auditing.
• Integrate Backup Systems into Security Testing: Regularly conduct red teaming and penetration testing exercises against backup environments to identify vulnerabilities in access controls, management interfaces, and operational resilience.

About the Speaker(s)

Ryan Kane is a seasoned cybersecurity professional with approximately nine years of experience at Northwestern Mutual. His expertise lies in pen testing and red teaming, where he actively works to identify and exploit vulnerabilities within an organization's systems and infrastructure. Additionally, Ryan contributes to the broader cybersecurity community as a volunteer at CypherCon in Milwaukee, Wisconsin.

Roshank works alongside Ryan at Northwestern Mutual, specializing in a diverse range of offensive security practices including pen testing, red teaming, and purple teaming. He also dedicates time to security research, focusing on uncovering new attack vectors and vulnerabilities. This presentation at Black Hat USA marked Roshank's debut as both a speaker and an attendee at the prestigious conference.