Detecting persistent threats on Draytek devices

Detecting persistent threats on Draytek devices

Speakers: Octavio Gianatiempo, Gastón Aznarez

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=BiBMsw0N_mQ

Overview

In an era where edge devices serve as critical gateways to corporate and home networks, their security has become paramount. The talk "Detecting persistent threats on Draytek devices" by Octavio Gianatiempo and Gastón Aznarez from the Faraday research team sheds light on the often-overlooked vulnerabilities and sophisticated persistence mechanisms present in Draytek SOHO (Small Office/Home Office) routers. These devices, frequently found at the network perimeter, are increasingly targeted by advanced threat actors, making a deep understanding of their security posture essential.

The speakers detail their journey from responding to a client compromise involving outdated Draytek equipment to reverse engineering proprietary firmware. Their research not only uncovers several critical vulnerabilities but also reveals a unique persistence mechanism involving Dynamic Loadable Modules (DLM). This mechanism allows malicious code to survive firmware updates and even factory resets, posing a significant challenge for detection and remediation. The talk culminates in the release of Drytech Arsenal, an open-source tool designed to aid researchers and defenders in analyzing Draytek firmware.

This presentation is highly significant for network administrators, security researchers, and anyone concerned with the security of edge devices. It underscores the critical need for proactive security measures beyond traditional patching, highlighting how sophisticated attackers can maintain a foothold in seemingly secure environments. By providing both a detailed analysis of the threats and practical tools for investigation, Gianatiempo and Aznarez empower the community to better defend against these advanced persistent threats.

Background

▶ Watch: Introduction, speakers, and talk agenda (0:00)

The motivation for this research stemmed from a real-world incident: a client's network compromise where outdated Draytek equipment, specifically SOHO models running a MIPS32 RTOS, was identified at the network's edge. This immediately raised questions about whether known vulnerabilities, previously exploited on high-end Draytek routers running Linux, were also exploitable on these less sophisticated SOHO devices, and if these routers served as the initial entry point for attackers.

The threat landscape for edge devices has seen a dramatic increase in sophistication. The speakers cited several prominent examples of malware specifically targeting routers and similar hardware:

• Zuo Rat: A sophisticated malware designed to compromise SOHO routers, demonstrating the capability of attackers to target these low-resource devices.
• Hiatus Rat: Another notable malware that specifically targeted Draytek's higher-end router models, proving that Draytek devices are already on the radar of advanced persistent threat (APT) groups.
• Benign Certain: An NSA tool, leaked by the Shadow Brokers group, which targeted Cisco routers. This example highlights the state-sponsored interest in compromising critical network infrastructure, including edge devices.

A critical aspect of this investigation was understanding the sheer scale of the potential attack surface. Shodan scans revealed over 500,000 Draytek devices exposed online, with a significant concentration in the UK. Furthermore, a plethora of existing vulnerabilities were known, ranging from basic information disclosure flaws to authenticated remote code execution (RCE), indicating a ripe environment for attackers.

The initial hurdle for the research team was the closed-source nature of Draytek's firmware. Traditional firmware analysis tools like Binwalk failed to extract meaningful components from the SOHO firmware images. Specialized tools such as DryTools were found to be severely outdated, with the last commit being 12 years prior, rendering them ineffective for contemporary firmware versions. The team also evaluated an approach described by Philip Loret at Hexacon 2022, which involved emulating Draytek's bootloader to extract the RTOS kernel. While innovative, this method was highly manual, required address tweaking for each firmware version, and crucially, could not extract other executable components or file systems beyond the bootloader and kernel. These limitations underscored the need for a more comprehensive and automated solution for firmware analysis.

Key Findings

▶ Watch: Threats targeting edge devices and attack surface (2:00)

The research yielded several significant discoveries that collectively paint a concerning picture for the security of Draytek SOHO devices and offer crucial insights for defenders.

Firstly, the team successfully reverse-engineered the proprietary firmware format for Draytek's RTOS-based routers. This was a foundational step, revealing a complex structure comprising binary and web sections, each with specific headers and compressed components. Crucially, this process uncovered the existence of Dynamic Loadable Modules (DLM), stored within a compressed and encrypted file system, which would later be identified as a primary vector for persistence.

Building upon this understanding, Gianatiempo and Aznarez developed and open-sourced Drytech Arsenal, a collection of Python-based tools. This innovative solution overcomes the limitations of previous methods by providing comprehensive support for extracting all firmware versions from various Draytek models. It can dissect the firmware into its constituent parts, including the bootloader, RTOS kernel, web file systems, and critically, the DLM code, enabling unprecedented access for security analysis.

The investigation uncovered multiple critical vulnerabilities:

• Insecure Password Storage: Credentials and passwords were found to be stored in plaintext in memory. This flaw means that any attacker gaining even limited access to the device's memory could easily extract sensitive login information.
• Non-Constant Time Comparison: The authentication process used a comparison function that exhibited non-constant time execution. This vulnerability opens the door to timing attacks, where an attacker can infer parts of a password by measuring the time taken for credential comparisons, progressively guessing characters until the full password is revealed.
• Predictable Second-Factor Authentication (2FA): Draytek implemented a second-step authentication mechanism, which, while well-intentioned, was found to be predictable. The pseudo-random number generation for 2FA codes was seeded using the time elapsed since boot. An attacker who can determine the router's uptime could accurately predict the 2FA codes, effectively bypassing this security layer.

Perhaps the most alarming finding was the inherent persistence mechanism offered by the Dynamic Loadable Modules (DLM). These modules are stored in a separate, encrypted file system that resides in the flash memory of the device. This design means that malicious DLMs, once injected, can survive across firmware updates and even factory resets, making them an ideal mechanism for advanced persistent threats. The researchers also identified a critical architectural weakness: a lack of memory protection between loaded modules and the RTOS kernel. All code runs within the same memory space, allowing a compromised module to modify the kernel's code and vice versa, escalating the impact of any exploit.

Technical Deep Dive

▶ Watch: Challenges with closed-source firmware and existing tools (3:00)

The technical core of this research revolves around the meticulous reverse engineering of Draytek's RTOS firmware format and the subsequent development of Drytech Arsenal. Understanding the firmware's internal structure was paramount to identifying vulnerabilities and potential persistence vectors.

Draytek firmware images for SOHO models are structured into two primary sections: the binary section and the web section. Both begin with a header containing crucial metadata, including the size of the section and a pointer to the subsequent section. The initial header of the binary section further includes information about the firmware version and other device-specific metadata.

The web section encapsulates a compressed file system, which contains all the static files served by the router's web user interface. This section is less critical for executable code analysis but is important for understanding the router's administrative interface.

The binary section is where the core executable components reside:

1. Bootloader: This is the first component, consisting of raw MIPS32 assembly code. It is responsible for initial hardware setup and loading the RTOS kernel. Prior attempts at firmware extraction, such as Philip Loret's method, focused on emulating this bootloader to obtain the kernel.
2. Compressed RTOS Kernel: Following the bootloader, the main operating system kernel for the RTOS is found in a compressed format.
3. Dynamic Loadable Modules (DLM) Code: This is arguably the most critical and interesting part of the binary section. It is a compressed and encrypted file system containing dynamic modules. These modules are loaded by the RTOS to extend functionality, but their design also presents a significant security risk.

The challenges with existing tools for firmware extraction were substantial. Binwalk, a widely used tool for firmware analysis, failed to produce any results for new Draytek firmware versions, indicating a proprietary or non-standard format. DryTools, while specialized, was severely outdated, last updated 12 years ago, and thus incompatible with modern firmware. Philip Loret's method at Hexacon 2022, which involved emulating the bootloader, could extract the RTOS kernel but was a manual process requiring address tweaking for each firmware version and, crucially, could not extract the DLM code or other file systems.

To overcome these limitations, the Faraday research team developed Drytech Arsenal. This Python-based, open-source collection of tools was specifically designed to:

• Support all new firmware versions.
• Extract firmware from different Draytek models.
• Extract all file systems, including the web UI's static files.
• Extract all executable components, notably the bootloader, RTOS kernel, and the entire DLM code section.

The tool's release aims to democratize access to Draytek firmware analysis, allowing other researchers and defenders to inspect these devices more effectively.

The DLM mechanism is central to understanding the persistence threat. These modules are stored in a distinct, encrypted file system within the router's flash memory. When the router boots or during certain update processes, these modules are loaded. A key vulnerability identified is the lack of memory protection between these loaded modules and the RTOS kernel. This means that if an attacker compromises the router and injects a malicious DLM (e.g., a modified SSH module), this module can operate with the same privileges as the kernel, modify kernel code, and maintain persistence. Critically, because DLMs reside in a separate file system in flash, they are not overwritten during standard firmware updates and persist even after a factory reset. This makes traditional remediation methods ineffective against such a persistent threat.

The discovered vulnerabilities add layers to the attack surface:

• Insecure Password Storage: The storage of administrative credentials in plaintext in memory means that any exploit providing memory access (e.g., a memory leak or a buffer overflow) could directly expose login details.
• Non-Constant Time Credential Comparison: The function responsible for verifying user credentials does not execute in a fixed amount of time. Instead, the execution time varies based on the number of correct characters in the input password. This allows an attacker to perform a timing attack, guessing one character at a time, measuring the response time, and progressively reconstructing the entire password.
• Predictable Second-Factor Authentication: The 2FA mechanism relies on pseudo-random numbers generated using rand(). However, the seed for this function is derived from the time elapsed since boot. If an attacker can determine the router's uptime, they can predict the sequence of 2FA codes, effectively bypassing this security control. The speakers demonstrated this by triggering a crash and reboot, then predicting the code for login.

Demo / Proof of Concept

▶ Watch: Our solution: Drytech Arsenal for firmware extraction (4:45)

The speakers presented a compelling demonstration showcasing the practical implications of their findings, particularly the predictable second-factor authentication and the potential for persistent compromise. The demo setup involved serial communication with a Draytek router, a script to trigger the vulnerability, and the router's web UI.

The demonstration began by showing the router undergoing a series of reboots, triggered by an exploit. This action was crucial because the predictability of the 2FA codes depended on knowing the router's uptime. By initiating the reboots and monitoring the serial output, the researchers could accurately determine the time elapsed since the last boot.

With the uptime established, the team then ran a script designed to predict the 2FA code that the router would generate. This prediction was based on their understanding of how the router's pseudo-random number generator was seeded (i.e., using the time since boot). Once the code was predicted, they navigated to the router's web UI and successfully logged in using the predicted 2FA code, effectively bypassing the intended security measure.

While the demo primarily focused on the predictable 2FA, the speakers used it to illustrate a broader and more dangerous scenario: the persistence of malicious modules. They emphasized that if an attacker were to craft a modified SSH module (a type of DLM) and upload it to the router, that module would persist even through firmware updates and factory resets. The lack of memory protection means such a module could deeply embed itself and maintain control without detection. The demo thus served as a gateway to understanding the profound implications of the DLM persistence mechanism, demonstrating how an attacker could gain initial access and then establish a durable foothold.

Defensive Implications

▶ Watch: Detailed analysis of Draytek firmware format (5:15)

The findings presented by Octavio Gianatiempo and Gastón Aznarez carry significant implications for the defense of Draytek edge devices and, by extension, other similar embedded systems. The primary takeaway for defenders is that traditional security measures, such as simply updating firmware or performing factory resets, are insufficient against the sophisticated persistence mechanisms discovered.

The most critical defensive implication stems from the Dynamic Loadable Modules (DLM) and their ability to persist across firmware updates and factory resets. Once a malicious DLM is injected, it becomes a permanent resident in the router's flash memory. This necessitates a paradigm shift in how these devices are secured and remediated. Defenders cannot assume that a fresh firmware installation or a configuration reset will cleanse a compromised device.

The speakers proposed several mitigation strategies:

1. Removal of the DLM Feature: The most straightforward, albeit potentially disruptive, solution would be for Draytek to remove the dynamic module loading feature entirely. While this might be suitable for some users, others might rely on the extended functionalities provided by DLMs, making this a non-universal solution.
2. Hooking Module Loading Calls: A more nuanced approach involves intercepting and validating calls to module loading functions during both the boot process and firmware updates. This would allow the router to check the integrity or authenticity of any module before it is loaded into memory.
3. In-Memory Integrity Checking: The most robust proposed solution is to implement a mechanism for checking the in-memory integrity of loaded modules. This would involve cryptographic verification of modules as they are loaded and potentially periodically thereafter. Such a system would need to:

• Fetch signatures from a trusted server: As modules are updated, their legitimate signatures would also need to be updated and retrieved from a secure source.
• Check certificates: The integrity checking process must incorporate robust certificate validation to ensure that the signatures are genuine and originate from Draytek. This prevents attackers from signing their malicious modules with self-generated or compromised keys.

Beyond these specific mitigations for DLMs, the discovered vulnerabilities also highlight broader defensive needs:

• Patch Management: While firmware updates alone don't remove persistent DLM threats, they are still crucial for patching other vulnerabilities like information disclosure, RCE, insecure password storage, and predictable 2FA. Organizations should ensure their Draytek devices are running the latest firmware versions.
• Strong Authentication Practices: The plaintext password storage and predictable 2FA underscore the need for extremely strong, unique passwords for router administration. Where possible, administrators should leverage VPNs or other secure out-of-band management solutions to access router interfaces, reducing direct exposure to the internet.
• Network Segmentation and Monitoring: Edge devices are high-value targets. Organizations should implement robust network segmentation to limit the blast radius if a router is compromised. Continuous monitoring of network traffic originating from or passing through edge devices can help detect anomalous behavior indicative of compromise, even if the persistence mechanism is not immediately detectable.
• Firmware Analysis: The release of Drytech Arsenal is a critical resource for defenders. Security teams can use this tool to analyze firmware images, understand the components, and potentially identify indicators of compromise or anomalous modules if they suspect a device has been tampered with.

In conclusion, defending against threats on Draytek SOHO devices requires a multi-faceted approach that goes beyond conventional patching. The focus must shift towards deep integrity verification of loaded components and a recognition of the sophisticated persistence capabilities that attackers can leverage.

Key Takeaways

• Edge devices are critical, vulnerable targets: Draytek SOHO routers, like many edge devices, present a large attack surface (>500,000 exposed devices) and are increasingly targeted by sophisticated malware and state-sponsored tools.
• Proprietary firmware requires deep analysis: The closed-source nature of Draytek's RTOS firmware and the inadequacy of existing tools necessitated extensive reverse engineering to uncover critical security flaws.
• Drytech Arsenal is a vital open-source tool: The development and release of Drytech Arsenal (Python-based) provide researchers and defenders with an essential capability to extract and analyze all components of Draytek firmware, including the crucial Dynamic Loadable Modules (DLM).
• Dynamic Loadable Modules (DLM) enable potent persistence: Draytek's DLM mechanism, stored in an encrypted, separate flash file system, allows malicious code to persist across firmware updates and factory resets, making detection and remediation exceptionally challenging.
• Multiple critical vulnerabilities were discovered: The research identified insecure plaintext password storage, timing attack vulnerabilities due to non-constant time credential comparison, and a predictable second-factor authentication mechanism based on router uptime.
• Defenders need advanced integrity checks: Effective defense against persistent threats on these devices requires implementing in-memory integrity checking for loaded modules, fetching and validating signatures from trusted sources, and a re-evaluation of traditional remediation strategies.

About the Speaker(s)

Octavio Gianatiempo and Gastón Aznarez are members of the Faraday research team. Both speakers possess a strong foundation in computer science and extensive experience working with firmware. They describe themselves as individuals who enjoy playing CTFs (Capture The Flag competitions), breaking things, and conducting in-depth security research. Their combined expertise was instrumental in dissecting the complex Draytek firmware and uncovering the vulnerabilities and persistence mechanisms discussed in their DEF CON 32 talk.