Surfacing a Hydra: Unveiling a Multi-Headed Chinese State-Sponsored Campaign

Surfacing a Hydra: Unveiling a Multi-Headed Chinese State-Sponsored Campaign Against a Foreign Government

Speakers: Morgan Demboski, Threat Intel Analyst, Sophos MDR Operations; Mark Parsons, Threat Hunter, Sophos MDR Operations

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=C_csnhjFZ4o

Overview

This presentation, "Surfacing a Hydra: Unveiling a Multi-Headed Chinese State-Sponsored Campaign Against a Foreign Government," delivered by Sophos MDR Operations team members Morgan Demboski and Mark Parsons, sheds light on Operation Crimson Palace, a sophisticated and long-running Chinese Advanced Persistent Threat (APT) cyber espionage campaign. The talk details a complex attack against a government organization in a geopolitically sensitive Southeast Asian country, revealing the coordinated efforts of multiple state-sponsored groups, referred to as Cluster Alpha, Cluster Bravo, and Cluster Charlie. Sophos's investigation uncovered two distinct stages of the campaign, with the second stage representing a significant escalation following defensive actions.

The speakers, a threat intelligence analyst and a threat hunter respectively, provide an insider's view into the challenging realities of investigating a persistent nation-state adversary. They highlight the difficulties encountered, such as operating within a "mixed estate" environment with limited visibility, inheriting an existing long-term breach, and facing contractual restrictions on full remediation. Despite these constraints, Sophos successfully monitored and impeded the threat actors' operations, demonstrating the effectiveness of flexible analysis methods and proactive threat hunting in mitigating sophisticated espionage campaigns.

The significance of Operation Crimson Palace lies not only in its technical sophistication and the coordination of multiple Chinese APT groups but also in the valuable lessons it offers to the wider cybersecurity community. By sharing their investigative journey, Demboski and Parsons aim to enhance understanding of the evolving Chinese APT threat landscape and equip defenders with practical strategies for overcoming similar analytical and operational hurdles. The talk underscores the critical importance of continuous monitoring, adaptive response, and a deep understanding of adversary tactics, techniques, and procedures (TTPs) when confronting well-resourced state-sponsored threats.

Background

▶ Watch: Introduction to 'Surfacing a Hydra' talk (0:00)

The campaign, dubbed Operation Crimson Palace, primarily targeted a government organization situated in a Southeast Asian country. This region is characterized by significant geopolitical tensions with China, particularly concerning territorial and resource disputes in the South China Sea. Consequently, the victim organization, and indeed the entire region, holds immense intelligence value for the People's Republic of China (PRC), making it a prime target for state-sponsored espionage. The initial phase of the campaign focused on this specific government entity, but by January 2024, Sophos observed an expansion of activity to other critical organizations within the same geographical region, indicating a broader strategic objective.

Sophos's investigation into Operation Crimson Palace was fraught with several immediate and significant challenges. Firstly, the target organization operated what Sophos characterized as a mixed estate environment. This meant that Sophos's Managed Detection and Response (MDR) and Extended Detection and Response (XDR) services covered only a subset of the network, resulting in a crucial lack of full visibility across the entire infrastructure. This partial visibility inherently limited the scope of their monitoring and analysis, making comprehensive understanding of the adversary's movements considerably more difficult.

Secondly, the customer was onboarded as an MDR client with an existing long-term breach. This meant that the organization was already compromised prior to Sophos's engagement, preventing the security team from determining the initial mode of access used by the threat actors. Further analysis revealed related malicious activity dating back to early 2022, strongly suggesting that the Chinese APT groups had maintained longstanding access on at least some unmanaged assets within the network for an extended period before their discovery. This pre-existing compromise added layers of complexity to the investigation, requiring Sophos to untangle a deeply entrenched and mature adversary presence.

Finally, Sophos faced contractual limitations regarding response actions. Their MDR agreement with the client did not grant them authorization to take full remediation steps, such as directly removing malware or isolating compromised systems. This placed the Sophos team in a difficult position, unable to fully eradicate the threat actors from the network. However, rather than succumbing to inaction, this constraint prompted a strategic shift: the team focused on continuous monitoring of the threat actors' activity, aiming to impede their tooling, disrupt their command and control (C2) infrastructure, and ultimately make their operations significantly harder, thereby maximizing their defensive impact within the given limitations.

Key Findings

▶ Watch: Immediate challenges in initial investigation (3:20)

Operation Crimson Palace was unveiled as a sophisticated, multi-stage Chinese APT cyber espionage campaign characterized by an unprecedented level of coordination among multiple state-sponsored groups. Sophos identified three distinct clusters of activity, designated Cluster Alpha, Cluster Bravo, and Cluster Charlie, all working in concert to achieve long-term access and intelligence gathering objectives within the targeted foreign government agency. This multi-headed approach, reminiscent of the mythical Hydra, was a significant finding, as threat actors typically operate with a more linear progression. The observed non-linear movement, with groups "moving back and forth on one another" and deploying diverse C2 channels and mechanisms, strongly suggested the involvement of multiple, distinct entities rather than a single actor.

The campaign unfolded in two primary stages. The first stage, which Sophos initially published research on in June 2023, spanned from at least March to August of 2023. During this period, the identified clusters focused on maintaining persistent access to the primary government organization. The second stage represented previously unpublished research, commencing at the end of September 2023. This escalation occurred directly following Sophos's actions to block the actors' C2 implants, indicating a reactive and adaptive adversary. The second stage was marked by a complete change in the actors' pace and tactics, alongside a significant expansion of their activities to include other critical organizations within the same Southeast Asian region, demonstrating a broader scope and renewed offensive.

The initial discovery of Operation Crimson Palace was the result of a targeted threat hunt employing a thesis-driven hunting methodology. Sophos threat hunters developed a specific thesis centered on PowerShell being used as a command and control channel, particularly leveraging the PowerShell class TCP client. This hunt led to the identification of several critical indicators of compromise (IOCs) that collectively pointed to the sophisticated nature of the intrusion:

1. VMnet executable: An unusual execution of a legitimate VMware executable from a "Microsoft Vault directory." This suspicious file path, combined with its network activity, immediately raised flags.
2. TCP client communication to a .info domain: The VMnet executable was observed using the TCP client mechanism to communicate with a .info top-level domain (TLD). The use of such a rare TLD for C2 communication is highly atypical and indicative of malicious intent.
3. Sophos UD (unsigned binary): An unsigned binary masquerading as Sophos software was discovered. Sophos explicitly stated that they do not use unsigned binaries, confirming this as a clear attempt at evasion and masquerade by the threat actors.
4. SSLWND64 with Phantom Net hash match: Perhaps the most definitive finding was an executable named SSLWND64 that maintained an exact hash match with Phantom Net, a known piece of nation-state malware. This direct link to established state-sponsored tooling provided strong attribution signals and underscored the severity of the compromise.

These findings, particularly the presence of nation-state malware and the coordinated, non-linear behavior across the network, validated the hypothesis of multiple, sophisticated state-sponsored actors at play, prompting a deeper dive into the environment that ultimately uncovered the full scope of Operation Crimson Palace.

Technical Deep Dive

▶ Watch: MDR strategy when full remediation isn't possible (4:30)

Sophos's success in uncovering Operation Crimson Palace stemmed from a rigorous thesis-driven hunting methodology. This approach begins with the creation of a precise statement defining what "evil looks like" within a network. This hypothesis guides the subsequent querying of security logs to identify matching patterns of behavior. Crucially, the methodology emphasizes the importance of contextual analysis around identified events to definitively determine their malicious nature, rather than relying solely on signature-based detections. This proactive stance is essential for detecting advanced threats that have successfully evaded initial defenses.

In the context of Operation Crimson Palace, the specific hunt thesis focused on the use of PowerShell as a command and control (C2) channel, specifically targeting instances where the PowerShell class TCP client was being invoked. PowerShell, a powerful scripting language, is often abused by adversaries for its native capabilities, and its TCPClient class provides a direct mechanism for establishing network connections, making it an attractive choice for C2 communications that might bypass traditional proxy or application-layer filtering.

The initial indicators that triggered deeper investigation were a confluence of anomalous activities:

1. VMnet executable from a Microsoft Vault directory: Threat hunters observed a VMnet executable, a legitimate component associated with VMware virtualization software, executing from a highly unusual location – a "Microsoft Vault directory." This misplacement of a known executable immediately raised suspicion, as legitimate software typically resides in its designated installation paths. The execution from an unexpected location is a classic technique for masquerading malicious activity.
2. TCP client communication to a .info domain: The VMnet process was then observed leveraging the TCP client mechanism to establish outbound communication to a .info domain. The .info top-level domain (TLD) is relatively uncommon for legitimate enterprise C2 or standard network traffic, making its presence a strong indicator of potential malicious activity. This combination of a misplaced executable and communication to an unusual TLD was a significant red flag.
3. Sophos UD (unsigned binary) masquerade: Another critical piece of evidence was the discovery of an unsigned binary purporting to be "Sophos UD." Sophos confirmed that their legitimate software does not utilize unsigned binaries. This represented a direct attempt by the threat actors to masquerade their malicious payloads as trusted security software, aiming to blend in with legitimate system processes and evade detection. The lack of a digital signature is a strong indicator of tampering or malicious intent when found in the context of an established vendor's software.
4. SSLWND64 and Phantom Net: The most compelling evidence for nation-state involvement was the identification of an executable named SSLWND64. This binary maintained an exact hash match with Phantom Net, a well-documented piece of nation-state malware. Phantom Net is known for its use in cyber espionage campaigns, and its presence unequivocally tied Operation Crimson Palace to a sophisticated, state-sponsored adversary. This finding shifted the investigation from a generic breach to a targeted campaign by a highly capable actor.

The sheer volume and diversity of these attacks across the network led Sophos to describe the situation as a "critical high yikes situation." The threat actors exhibited behavior that deviated significantly from typical linear progression through an environment. Instead, they were observed "moving back and forth on one another," simultaneously deploying different C2 channels and multiple distinct mechanisms. This highly unusual, non-linear, and seemingly redundant activity led Mark Parsons to hypothesize that not one, but "two or three" distinct actor groups might be operating concurrently within the compromised network.

To prove this theory and effectively uncover the different threat clusters, Sophos meticulously collected and analyzed a wide array of data:

• Authentication information: This included logs detailing where compromised accounts were authenticating to and from, helping to map lateral movement and identify the scope of credential compromise.
• File creation data: By tracking when and where malware created new files or components, the team could establish connections between different malicious artifacts and understand their deployment chains.
• Network telemetry: This crucial data provided insights into external communications, revealing the C2 domains and IP infrastructure that the deployed malware communicated with.

From this aggregated data, Sophos constructed attack flows, which served as the cornerstones for defining the distinct clusters. Each attack flow was meticulously detailed, encompassing the attack vector, the machine the actor authenticated from, the account used to authenticate, the target machine compromised, the malware deployed, the C2 domain it communicated with, and finally, the IP infrastructure hosting that domain. While Cluster Alpha was the first to be identified through the initial hunt, a chronological reconstruction of events based on these attack flows revealed that Cluster Bravo was, in fact, the earliest active group in the campaign, underscoring the importance of flexible thinking and re-evaluating initial assumptions in complex investigations.

Demo / Proof of Concept

▶ Watch: Sophos's thesis-driven threat hunting methodology (6:00)

The provided transcript focuses entirely on the investigative process, findings, and challenges encountered during the Operation Crimson Palace campaign. There is no mention of any live demonstration or proof-of-concept presented during the talk.

Defensive Implications

▶ Watch: Key indicators from the PowerShell TCP client hunt (6:40)

The detailed investigation into Operation Crimson Palace offers several critical defensive implications for organizations facing sophisticated state-sponsored threats:

1. Prioritize Full Network Visibility: The "mixed estate" challenge faced by Sophos highlights the paramount importance of comprehensive monitoring across an entire enterprise environment. Gaps in visibility, whether due to unmanaged assets, legacy systems, or incomplete sensor deployment, create blind spots that sophisticated adversaries will inevitably exploit for long-term persistence and data exfiltration. Organizations must strive for unified visibility solutions that cover all endpoints, network segments, and cloud resources.
2. Embrace Proactive Threat Hunting: Relying solely on automated alerts and signature-based detections is insufficient against adaptive APTs. Operation Crimson Palace was uncovered through a thesis-driven threat hunt, demonstrating that proactive, hypothesis-led investigations are crucial for identifying evasive threats. Security teams should invest in building threat hunting capabilities, encouraging analysts to formulate hypotheses about "what evil looks like" and systematically search for anomalous behaviors.
3. Scrutinize Legitimate Tools and Rare TLDs: Adversaries frequently abuse legitimate system tools like PowerShell and VMware executables (e.g., VMnet) to blend into normal network traffic. Defenders must develop robust baselines for expected behavior of these tools and actively look for anomalies, such as executions from unusual directories or communications to rare TLDs (like .info domains) via native system processes or unusual network protocols (e.g., PowerShell TCP client).
4. Verify Binary Authenticity: The discovery of an unsigned binary masquerading as "Sophos UD" underscores the need for stringent verification of software integrity. Organizations should implement policies and technical controls to detect and flag unsigned binaries or those with invalid digital signatures, especially when they purport to be from trusted vendors. This also extends to verifying hashes of critical system files against known good values.
5. Be Prepared for Long-Term Breaches: The fact that the victim was onboarded with an "existing long-term breach" dating back to early 2022 is a stark reminder that adversaries can maintain access for extended periods without detection. Organizations must assume compromise and focus on continuous monitoring, historical log analysis, and incident response readiness to uncover deeply entrenched threats, even if initial access cannot be determined.
6. Develop Adaptive Response Strategies: Sophos's inability to perform full remediation due to contractual limitations forced them to adapt by focusing on monitoring, impeding tooling, and disrupting C2 infrastructure. This demonstrates that even with constrained response capabilities, defenders can still significantly impact adversary operations. Organizations should develop tiered response strategies that account for various levels of authority and technical capabilities, prioritizing intelligence gathering and disruption when full eradication is not immediately possible.
7. Recognize Multi-Actor Coordination: The "critical high yikes situation" and the non-linear, overlapping activities of Cluster Alpha, Bravo, and Charlie highlight that sophisticated campaigns can involve multiple, coordinated threat groups. Defenders should be alert to atypical actor behaviors – such as multiple C2 channels on the same host, redundant persistence mechanisms, or seemingly contradictory actions – as these can be indicators of a multi-headed adversary.
8. Foster Flexible Analytical Thinking: The speakers emphasized the importance of flexibility in analysis, as evidenced by the discovery that Cluster Bravo was chronologically active before Cluster Alpha, despite Alpha being the initial detection. Security analysts must be willing to challenge initial assumptions, re-evaluate findings, and adjust their investigative approach as new data emerges to accurately map complex campaigns.

Key Takeaways

• Multi-Headed APTs: Chinese state-sponsored campaigns can involve the coordinated efforts of multiple distinct APT groups (e.g., Cluster Alpha, Bravo, Charlie) within a single operation, making them highly resilient and complex to unravel.
• Proactive Threat Hunting is Essential: Thesis-driven threat hunting, focusing on specific anomalous behaviors like PowerShell TCP client C2, is critical for detecting sophisticated, evasive threats that bypass traditional defenses.
• Adaptive Defense Under Constraints: Even when full remediation is not possible, continuous monitoring, intelligence gathering, and impeding adversary tooling (e.g., blocking C2 implants) can significantly disrupt sophisticated operations.
• Unusual Behavior Indicates Complexity: Non-linear movement, deployment of diverse C2 channels, and masquerading as legitimate software (e.g., unsigned Sophos UD, VMnet in unusual paths) are strong indicators of advanced and potentially multi-actor campaigns.
• Phantom Net Attribution: The discovery of SSLWND64 with an exact hash match to Phantom Net malware provides strong evidence of nation-state involvement and underscores the severity of the compromise.
• Full Visibility and Historical Data are Paramount: Operating in mixed estates and inheriting long-term breaches highlight the need for comprehensive network visibility and access to historical data to fully understand the scope and duration of an intrusion.

About the Speaker(s)

Morgan Demboski is a Threat Intelligence Analyst on Sophos's Managed Detection and Response (MDR) Operations team. In her role, Morgan focuses on analyzing and enriching the vast amounts of data collected by Sophos's operations and hunt teams to identify and understand emerging threats. Outside of her work protecting over 25,000 organizations, Morgan resides in Washington, DC, where she pursues her passions as an up-and-coming pottery artist and an emerging beach volleyball superstar.

Mark Parsons serves as a Threat Hunter for the Sophos MDR Operations team. His primary responsibility involves actively seeking out and discovering "new and unfound evil" within customer environments, a task he approaches with dedication and expertise. While Mark humbly acknowledges he should "in no way be an artist or let near any kind of doodling," he is a passionate snowboarder and a loving dog dad when not engaged in the critical work of uncovering advanced cyber threats.