APT Down and the mystery of the burning data centers

APT Down and the mystery of the burning data centers

Speakers: Christopher Kunz, Sylvester

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=0dMzlw-qSZI

Overview

This talk, "APT Down and the mystery of the burning data centers," delves into the intricate and alarming details of a significant cybersecurity leak concerning South Korean government and corporate infrastructure, allegedly perpetrated by a state-sponsored threat actor. Presented by journalists Christopher Kunz and Sylvester, who acted as proxies for the original, anonymous researchers from "Frack the Hacker," the presentation meticulously unpacks the contents of a dumped workstation and virtual machines belonging to a highly active, yet surprisingly sloppy, adversary dubbed "Kim." The talk not only exposes the sophisticated targeting and vast compromise of sensitive data but also reveals a startling series of physical infrastructure failures and bureaucratic missteps within South Korea, culminating in a catastrophic data center fire on the very day a parliamentary inspection was scheduled.

The presentation highlights the real-world consequences of inadequate cybersecurity practices, poor operational security, and a critical lack of resilience in national digital infrastructure. It serves as a stark warning about the potential for advanced persistent threats (APTs) to exploit not only digital vulnerabilities but also systemic organizational weaknesses. By tracing the adversary's digital footprint and correlating it with public events, Kunz and Sylvester paint a compelling picture of a nation grappling with persistent cyber espionage and the profound challenges of securing its most vital digital assets against a backdrop of geopolitical tensions.

Background

▶ Watch: Researchers discover and dump threat actor's workstation (2:50)

The genesis of this story lies in a significant leak discovered in June 2025 (as stated by the speakers, though context suggests 2023), when security researchers, later known as "Frack the Hacker," stumbled upon and subsequently dumped the workstation and two virtual machines (VMs) of a threat actor they designated "Kim." This unprecedented access allowed the researchers to "shoulder-surf" the adversary, observing his activities in real-time. The original authors of the discovery, unable or unwilling to present their findings publicly, granted Christopher Kunz and Sylvester, both journalists specializing in IT security, permission to act as proxies and disseminate the story further, augmented by their own independent research.

The dumped data comprised a Deepin Linux VM, which also contained the mounted homedir of its Windows host, and a separate Ubuntu VM hosted on a VPS in Singapore. Initial analysis revealed a chaotic and disorganized directory structure, characterized by generic folder names like "new folder" and "1111," often containing critical malware alongside mundane files. Kim's heavy reliance on US services, particularly GitHub (with over 400 bookmarked repositories), and his increasing frustration with auto-translation tools for Korean documents, hinted at his operational challenges.

The attribution of "Kim" became a central, albeit speculative, point of discussion. While the original Frack authors suggested an affiliation with Kimuki, a North Korean-associated APT group, citing Kim's observance of North Korean working hours and alignment with North Korean targets, other evidence pointed towards a Chinese origin. This included the observance of Chinese holidays, use of Chinese fonts, comments written in Chinese within code, and specific targeting of Taiwanese institutions. The speakers' educated guess leaned towards Kim being a Chinese actor, possibly collaborating with North Korean APTs, highlighting the blurred lines in state-sponsored cyber operations. This problem exists due to the persistent geopolitical tensions in the region, driving continuous cyber espionage and offensive operations against critical infrastructure.

Key Findings

▶ Watch: Overview: the dump's contents and availability for research (4:20)

The comprehensive analysis of "Kim's" dumped systems revealed a treasure trove of compromised data, sophisticated tooling, and evidence of widespread infiltration into critical South Korean infrastructure.

Kim's Toolkit and Operational Practices:

• Malware Variety: The dump contained a diverse array of malware, including a remote kernel backdoor named "Tomcat," a Cobalt Strike command and control (C2) beacon, a web shell client for ante systems (x1_admin.py), an anti-exploit, the Spawn Chimera backdoor, various loaders and packers like Titan loader, and reverse shells.
• Linux Rootkit: A previously unknown variant of a bespoke Linux rootkit was identified, indicating specialized capabilities.
• Payload Embedding: Kim attempted to embed a payload into the OneDrive standalone updater using sick flip to maintain a valid digital signature, demonstrating a level of sophistication in evasion techniques.
• Toy Box Modifications: Kim was actively modifying Toy Box, an all-in-one Linux command-line tool, specifically attempting to integrate "Pony," a generic payload, to spawn a shell. This work, potentially targeting Android systems, was found in a third-party fork, not the official repository.
• Sloppy OpSec: Despite the high-value targets, Kim exhibited poor operational security, mixing personal activities (e.g., "beerfinder" app, AC fun for anime/comics, YouTube tutorials on battery management systems) with work on the same workstation. His browser and bash histories, filled with translation attempts, further illustrated his struggles and lack of native Korean language skills.

Vast Compromise of South Korean Systems:

• Government Network Access: Log files indicated Kim had access to non-public South Korean government infrastructure, specifically the Onara system (Onara 9.sus.gcloud.go.kr), a core government computer network. This access was later confirmed by the South Korean secret service.
• Telecommunications Infiltration: Certificates and private keys for a domain belonging to South Korea Telecom's remote control service were found, suggesting access not only to the telco's network but also to its customers' systems. Information about U+ (part of LG Corporation), another mobile operator, included approximately 8,500 internal IP addresses, dozens of AWS relational database system endpoints, usernames, hostnames, and hashed passwords (SHA1 and MD5).
• Sensitive Government Data: The dump contained source code for an email system of the South Korean Ministry of Foreign Affairs, authentication systems, and information about the South Korean government PKI (GPKI). Thousands of private keys belonging to government employees and university professors (expired between 2021 and 2023) were found, serving as further proof of past intrusions.
• Classified Intelligence: Disturbingly, Google Translate history fragments contained snippets from interrogations of North Korean defectors, highly sensitive and classified information that should not have been accessible to a threat actor.
• Targeted Reconnaissance: Evidence of various phishing campaigns against the South Korean Defense Counter Intelligence Command and reconnaissance efforts targeting Taiwanese institutions was also present.

Catastrophic Infrastructure Failure:

• Data Center Fire: The most dramatic outcome was the fire at a data center run by the National Information Resources Service (NIRS) near Daejeon on September 26th, precisely the day a parliamentary on-site inspection was scheduled. This fire, attributed to lithium-ion batteries, destroyed almost 100 servers and took over 700 e-government services offline, including tax systems, health insurance, and citizen portals.
• Negligence and Non-Compliance: The NIRS data center was found to be in violation of post-2022/2023 safety recommendations (issued after a Cacao/Nava data center fire) regarding the spacing and compartmentalization of lithium-ion battery racks. Workers were reportedly relocating overage (11-year-old) LG batteries without proper discharge, leading to ignition by a power drill.
• Lack of Redundancy and Backups: A staggering 92.8% (speakers corrected from 70.2% to 7.2%, then 92.8% implied by context of "7.2% failover") of systems lacked failover capabilities. The G Drive, a 900 terabyte government data exchange system used by over 100,000 employees, had no backups, allegedly because it was "too big." Many existing backups were co-located with the primary servers, rendering them useless in the fire.
• Human Cost: The immense pressure to restore services led to the tragic suicide of a project manager for the recovery efforts, highlighting the severe psychological toll of such incidents.

Technical Deep Dive

▶ Watch: Threat actor's messy desktop: backdoors, Cobalt Strike, weird naming (4:50)

The technical analysis of "Kim's" operations reveals a blend of readily available tools, bespoke malware, and a surprising lack of sophistication in certain areas. The core of the compromise stemmed from the dumped Deepin Linux VM (running on a Windows host) and a YUbuntu VM on a Singaporean VPS.

Kim's toolkit included common offensive security staples. For instance, the presence of a Cobalt Strike C2 beacon in a directory named "1111" signifies reliance on a widely used commercial penetration testing tool often adopted by APTs for post-exploitation activities. The x1_admin.py file, identified as a client for a web shell for "ante systems," points to typical web-based access and command execution methods. The Tomcat remote kernel backdoor, found within a zip file on the desktop, suggests a deeper level of persistence and control over compromised Linux systems. The attempt to embed a payload into the OneDrive standalone updater using sick flip for signature bypass demonstrates an understanding of defensive evasion, aiming to leverage a legitimate, signed executable to deliver malicious code without invalidating its digital signature, making detection more challenging.

A particularly insightful discovery was Kim's active development work on Toy Box, an all-in-one Linux command-line utility. Found in his home/downloads folder, Kim was patching and commenting within the source code of a third-party fork (similar to one found in the Open Harmony repository), rather than the official version. His primary goal was to integrate "Pony," described as a generic payload, with the aim of forcing Toy Box to default-spawn a shell. This indicates an attempt to create a custom, persistent access mechanism, possibly for Android devices given Toy Box's typical target environment. The speakers showed a snippet of Kim's C code, illustrating commented-out attempts to achieve this, suggesting ongoing development and iteration.

The compromise of the South Korean government PKI (GPKI) was evidenced by thousands of certificates and their corresponding private keys. While the certificates themselves might be public, the private keys are highly sensitive. The fact that these keys had expiration dates between 2021 and 2023 strongly indicates that the intrusion into the GPKI infrastructure occurred around or before this timeframe, allowing the adversary to exfiltrate these critical cryptographic assets. Such keys could be used for impersonation, signing malicious code, or decrypting sensitive communications.

The data center fire, while a physical event, had deep technical implications. The root cause was identified as overage (11-year-old) lithium-ion batteries (likely manufactured by LG, a major South Korean producer) that were being relocated. Lithium-ion battery fires are notoriously difficult to extinguish because they undergo thermal runaway, producing their own oxygen, rendering traditional oxygen-depriving fire suppression systems ineffective. Effective suppression typically requires large volumes of water, which fire brigades were hesitant to use due to the risk of damaging adjacent servers. This highlights a critical design flaw where battery racks were co-located with server racks, contrary to post-2022/2023 industry recommendations that mandate at least 1 meter of separation and physical compartmentalization between battery and server infrastructure. The NIRS data center failed to adhere to these crucial safety guidelines, which were established after previous incidents like the Cacao/Nava data center fire.

Demo / Proof of Concept

▶ Watch: Humorous detail: threat actor's frustration with auto-translation (7:20)

While the talk did not feature a live demonstration of hacking tools or exploits, it powerfully presented compelling evidence and visual proof points derived directly from the leak and subsequent events. The closest equivalent to a "demo" of the threat actor's activities was a screenshot from the original researchers, showcasing Kim's Deepin Linux desktop while he was actively working. This "shoulder-surfing" image vividly illustrated his disorganized file management, with numerous GitHub tabs open, and generic folder names like "new folder" containing critical malware. This visual artifact served as direct proof of the unprecedented access the researchers gained into the adversary's operational environment.

For the catastrophic physical consequences, the presentation included a dramatic video clip of the NIRS data center fire in South Korea. The speakers provided a frame grab of the exact second the first sparks ignited, showing a worker crouching near a rack. This footage, publicly available on YouTube, visually demonstrated the rapid escalation of a lithium-ion battery fire, from initial ignition to the room being engulfed in smoke within seconds, underscoring the destructive power of such incidents. The images of the charred and water-soaked battery packs, removed from the data center and dumped into pools of water, further underscored the scale of the damage and the challenging nature of containing these specific types of fires. These visual elements, combined with specific details about Kim's tools and the compromised data, served as the "proof of concept" for the talk's core narrative.

Defensive Implications

▶ Watch: A treasure trove: vast data for workshops and deep analysis (8:00)

The detailed revelations from the "APT Down" leak and the subsequent data center fire offer crucial insights for defenders, emphasizing the need for a holistic approach to security that encompasses both cyber and physical domains, alongside robust organizational resilience.

1. Reassess Threat Actor Sophistication: The concept of "Advanced Persistent Trash" (APT) highlights that even state-sponsored actors can be sloppy and rely on basic operational security failures. Defenders should prioritize fundamental security hygiene, such as strong Identity and Access Management (IAM), multi-factor authentication, and role-based access control, rather than solely focusing on advanced, zero-day threats. The Onara system's vulnerability, where Kim could simply log in with compromised credentials and access anything, underscores this.
2. Enforce Strict Operational Security (OpSec): The adversary's mixing of personal and work activities on the same machine, and using public translation services for classified data, represents a significant OpSec failure. Organizations must implement strict policies against such practices, provide isolated environments for sensitive work, and educate employees on the dangers of personal device use and unapproved tools for official business.
3. Prioritize Supply Chain and Third-Party Risk Management: The compromise of South Korea Telecom's remote control service, granting access to customer networks, and the reliance on LG-manufactured batteries (which became a fire hazard) illustrate the cascading risks from third parties. Thorough vetting of suppliers, contractual security requirements, and continuous monitoring of third-party access are essential.
4. Implement Robust Backup and Disaster Recovery (DR) Strategies: The failure of the NIRS data center highlighted a catastrophic lack of resilience. Organizations must strictly adhere to the 3-2-1 backup rule (3 copies of data, on 2 different media, with 1 copy off-site). The fact that the 900 TB G Drive had no backups because it was "too big," and other backups were co-located, is an unacceptable risk. Regular testing of DR plans, including failover mechanisms, is critical.
5. Address Physical Infrastructure Safety: The lithium-ion battery fires are a recurring, known risk, especially in South Korea. Data center operators must strictly adhere to and regularly update physical safety guidelines, such as maintaining at least 1 meter distance between battery racks and ensuring their physical separation from server racks. Regular maintenance, including timely replacement of overage equipment (like the 11-year-old batteries), is paramount. Fire suppression systems must be appropriate for the specific hazards (e.g., water for lithium-ion fires, despite potential equipment damage).
6. Improve Incident Response and Forensics Readiness: The destruction of affected servers by LG and Korea Telecom before thorough forensic analysis could be completed severely hampered investigation efforts. Organizations must have clear protocols for preserving evidence post-incident, even if it means delaying service restoration for a short period.
7. Foster a Culture of Security and Accountability: The political aftermath and the tragic suicide of a project manager underscore the immense pressure and lack of accountability that can plague organizations following major incidents. Leadership must foster a culture where security is prioritized, vulnerabilities are addressed proactively, and employees are supported, not scapegoated, during crisis. Adhering to industry best practices should be a requirement, not an option, especially for critical government services.

Key Takeaways

• APT actors can be surprisingly sloppy: Despite accessing highly sensitive government systems, the threat actor "Kim" exhibited poor operational security, disorganized file management, and struggled with basic tasks like language translation. Defenders should not underestimate the impact of fundamental security hygiene.
• Critical infrastructure is vulnerable to both cyber and physical failures: The compromise of numerous South Korean government and corporate networks, coupled with the catastrophic data center fire, demonstrates that security must address both digital threats and physical infrastructure resilience.
• Lithium-ion battery fires are a known, severe risk: The NIRS data center fire, and previous incidents, highlight the unique challenges of extinguishing these fires and the critical need for strict adherence to safety guidelines, including physical separation and proper maintenance of battery systems.
• Robust backup and disaster recovery strategies are non-negotiable: The widespread data loss and prolonged service outages due to inadequate failovers, co-located backups, and a "too big to backup" mentality underscore the absolute necessity of adhering to principles like the 3-2-1 rule for all critical data.
• Organizational negligence has severe consequences: The failure to implement known safety recommendations, replace overage equipment, and maintain basic cybersecurity practices led directly to national service disruption, massive data loss, and even tragic human cost.
• Geopolitical tensions drive persistent, multi-faceted threats: The likely collaboration between Chinese and North Korean actors, targeting South Korean and Taiwanese institutions, illustrates the complex and persistent nature of state-sponsored cyber espionage in the region.

About the Speaker(s)

Christopher Kunz and Sylvester are journalists who specialize in reporting on IT security. They are known for writing, talking, and occasionally appearing on video to discuss various topics within the cybersecurity domain. For this specific talk, they served as proxies, presenting the findings of "Frack the Hacker" and other anonymous researchers who discovered the "APT Down" leak, augmenting the original research with their own investigations. They explicitly stated they were neither the source nor the original reporters of the initial leak.