A Tale of Two Leaks: How Hackers Breached the Great Firewall of China

A Tale of Two Leaks: How Hackers Breached the Great Firewall of China

Speakers: Jade Sheffey, PhD Student, University of Massachusetts Amherst

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=ubLr1MWiqwA

Overview

In this compelling talk at 39C3, Jade Sheffey, a PhD student at the University of Massachusetts Amherst, unveiled two significant breaches into the operational mechanisms of China's formidable Great Firewall (GFW). The presentation, titled "A Tale of Two Leaks," detailed how researchers uncovered critical vulnerabilities that not only allowed for the exfiltration of sensitive user data but also provided an unprecedented glimpse into the GFW's internal architecture and patching processes. This work stands as a testament to the ongoing efforts to understand, measure, and circumvent internet censorship, highlighting the inherent insecurity even in state-sponsored blocking systems.

The first leak, dubbed WallBleed, exposed a memory dumping vulnerability within the GFW's DNS injectors, allowing researchers to collect billions of packets containing private user information and internal network traffic. The second leak involved the unexpected public release of 600 GB of internal documents, source code, and binaries from GeGe Networks, a Chinese cybersecurity company founded by Fang Bing, often referred to as the "father of the Great Firewall." Together, these leaks offer a unique dual perspective: one from external, active measurement and exploitation, and the other from an internal, accidental disclosure. This research is crucial for understanding the evolving landscape of internet censorship, its privacy implications for users both inside and outside China, and the potential for targeted attacks against such systems.

The talk underscores the importance of academic research in challenging state-level censorship and protecting digital rights. By meticulously analyzing the GFW's behavior and internal components, Sheffey and her collaborators have not only demonstrated the fragility of these systems but also provided actionable intelligence for circumvention strategies and a stark warning about the global reach of China's censorship technology.

Background

▶ Watch: Introduction to talk and GFW DNS injector (0:00)

The Great Firewall of China is one of the most sophisticated and pervasive internet censorship systems globally, designed to block access to a vast array of websites and online services deemed undesirable by the Chinese government. Its mechanisms are diverse, ranging from HTTP/TLS resets, which terminate connections attempting to reach blocked sites, to DNS injection, which is the primary focus of this research. When a user in China attempts to access a blocked website, the GFW's DNS injectors intercept the DNS request and return fake, bogus IP addresses. These IPs are typically blackholed at a national level, preventing the user from establishing a connection, and often cause TLS handshakes to fail due to certificate mismatches.

A key revelation in the talk is that the GFW does not operate with a single, monolithic DNS injector but rather employs at least three distinct injectors, each with unique characteristics and fingerprints. While the real DNS request eventually reaches its intended resolver, the GFW's injectors simultaneously send back multiple, often conflicting, injected responses. The research primarily focused on injector 3. An intriguing observation was that some of these bogus IPs, while mostly blackholed, would respond to a TCP handshake and nothing else, with a "small percentage" even leading to adult websites for requests targeting blocked content. Furthermore, the GFW's DNS injection system is bidirectional, meaning that researchers outside China (e.g., in America or Germany) can send DNS lookups for blocked Chinese domains to Chinese IPs and receive injected responses, making experimentation and measurement significantly easier. Prior academic work, notably the WallBleed paper, laid the groundwork for understanding and exploiting these DNS injection mechanisms.

Key Findings

▶ Watch: Introducing 'WallBleed' and its academic origins (3:20)

The research presented by Jade Sheffey uncovered several critical findings stemming from both the WallBleed vulnerability and the GeGe Networks leak.

1. WallBleed: A DNS Injector Memory Leak:

The primary discovery was WallBleed, a memory dumping vulnerability in the GFW's DNS injectors. By crafting malformed DNS queries, specifically by manipulating the length field within the DNS packet's name structure (e.g., changing 8 Facebook 3 com to 8 Facebook 0x20 com), researchers could trick the injector into dumping portions of its memory. This leaked memory contained actual network traffic, including UPN (Universal Plug and Play) traffic, which should never be exposed on the public internet.

2. Longitudinal Data Collection and Privacy Leaks:

Through a two-year longitudinal experiment, researchers collected an astounding 5.1 billion WallBleed responses. This massive dataset revealed real internet traffic from users in China, including highly sensitive information. Analysis showed leaks of cookies, mail content, and even passwords, exposing significant privacy risks for Chinese citizens. The leaked data also contained internal GFW management traffic, indicating that the same memory buffer was used for both user traffic and internal operations.

3. Targeted Surveillance and Global Reach:

The bidirectionality of the GFW's DNS injector allowed for a form of "roughly targeted surveillance." By sending queries to specific Chinese IPs, researchers could trigger injectors near those geographical locations. More alarmingly, a global probe experiment from UC Boulder demonstrated that any internet traffic routing through China, regardless of its origin or destination, was susceptible to WallBleed, making it a privacy threat to users worldwide.

4. GFW Patching and Side-Channel Analysis:

The GFW actively attempted to patch WallBleed. Researchers observed two main patching events: around September-November 2023 and March 2024. The original domain 3.tt was initially removed from the blocklist, then the first patch caused all WallBleed-responding IPs to stop. Researchers adapted, creating "WallBleed V2" by reintroducing Qtype/Qclass and moving the label, which was also patched within weeks. By analyzing patterns in leaked x86-64 stack pointers and code pointers, researchers could infer when the GFW was patching its systems, effectively using the vulnerability as a side channel to monitor GFW development.

5. GeGe Networks Leak: Internal GFW Insights:

The public leak of 600 GB of documents, source code, and binaries from GeGe Networks provided an unprecedented internal view of GFW components. GeGe Networks, founded by the "father of the GFW," is a major player in China's cybersecurity landscape. Researchers identified the Misa platform's DNS parser as a key component, strongly suspecting it to be part of the GFW's Deep Packet Inspection (DPI) system. While git comments related to "memory leak" were found, they likely referred to standard memory management issues rather than WallBleed directly, suggesting WallBleed might be a separate vulnerability in a different component or a later iteration.

6. Export of Censorship Technology:

Analysis of the GeGe Networks leak revealed that the company actively exports its censorship software to other countries. Identified customers include Kazakhstan, Pakistan, Myanmar, and Ethiopia, with a mysterious "A21" customer still under investigation. This finding highlights China's role in proliferating internet censorship technologies globally, extending the reach of its surveillance capabilities beyond its borders.

Technical Deep Dive

▶ Watch: Optimizing WallBleed for maximum memory extraction (5:40)

The technical foundation of the WallBleed vulnerability lies in the manipulation of the DNS packet structure. A standard DNS query includes a transaction ID, flags, number of questions, and crucially, the name length and string segments for the domain being queried. For instance, facebook.com might be represented as 8 Facebook 3 com 0 (where 8 and 3 are length bytes, and 0 is the terminator).

The WallBleed exploit leverages a flaw in how the GFW's DNS injector processes these length fields. Instead of a valid length, researchers found that if they changed the length byte to a non-standard value, specifically 0x20 (a space character in ASCII), the injector would begin dumping adjacent memory. The initial exploit used a domain like 8 Facebook 0x20 com. To maximize the amount of leaked memory, researchers refined the technique:

1. Shortened Domain: They discovered that a very short domain, like 3.tt (later 4.tt after the original was unblocked), was sufficient to trigger the injector, leaving more space for the exploit payload.
2. Payload Optimization: They found they could remove the Qtype (Query Type) and Qclass (Query Class) fields at the end of the DNS query without breaking the exploit.
3. Maximum Length Byte: By setting the length byte to 0xFF, they could consistently exfiltrate up to 124 bytes of memory with each WallBleed query.

The leaked memory contained various types of network traffic. Early analysis revealed SSDP (Simple Service Discovery Protocol) and UPN (Universal Plug and Play) packets, which are typically confined to local networks and should not appear on the public internet. Further deep dives into the leaked data identified raw IPv4 packet headers, often starting with the signature 0x45 0x00. Filtering for these headers allowed researchers to reconstruct full network packets, not just application-layer data. Within these packets, a significant portion was TCP and UDP traffic, much of which involved private IP addresses, suggesting internal GFW management or highly localized network activity. The presence of these private IPs, alongside public IP traffic, led to the suspicion that the GFW's internal management traffic shared the same vulnerable memory buffers as user-facing DNS injection.

The GeGe Networks leak provided additional technical insights. Researchers identified the Misa platform as the likely DPI engine, with its DNS parser being a key component. Analysis of git comments within the leaked source code showed entries like "memory leak" and indicated patches that involved moving parsing logic earlier in a function to avoid allocating a session object for the DNS session. This suggests the GFW developers were aware of and addressing memory management issues, although it's still debated if these specific patches directly addressed WallBleed or other memory leaks. An interesting detail from the source code was the use of a custom memory allocator named dictator_malloc, which appeared designed for single applications on dedicated server appliances, possibly at the kernel level. This custom allocator could itself be an interesting target for future exploitation.

The use of x86-64 stack pointers and code pointers within the leaked memory provided a unique side channel. By observing changes in the frequency and patterns of these pointers (e.g., S for stack, C for code, and specific values like 1248), researchers could infer when the GFW's underlying software was being updated or patched. This technique allowed them to correlate observed changes in WallBleed responses with specific GFW patching timelines, offering a rare window into the GFW's operational maintenance.

Demo / Proof of Concept

▶ Watch: GFW's patching attempts and researchers' adaptations (6:10)

While the talk did not feature a live, interactive demo, the researchers presented compelling evidence of their findings through two major proof-of-concept (PoC) experiments: a longitudinal data collection and a tagged probe experiment, along with a global reach assessment.

The primary PoC was the longitudinal experiment, which demonstrated the persistent and widespread nature of the WallBleed vulnerability. Over a period of two years, researchers from UMass Amherst continuously sent specially crafted WallBleed DNS queries. These probes were directed from UMass to a Tencent cloud VPN, which they controlled, ensuring ethical research practices by not targeting random users. The queries were sent over UDP on varied ports to port 53 at a rate of 100 packets per second. This sustained effort resulted in the collection of 5.1 billion WallBleed responses, providing an enormous dataset of leaked memory. The sheer volume and consistency of these responses unequivocally proved the vulnerability's existence and its potential for extensive data exfiltration. The analysis of this massive dataset allowed them to identify real user traffic, including sensitive data like cookies and passwords, and internal network traffic such as UPN and SSDP.

To further validate that the leaked data represented actual internet traffic and not just internal GFW noise, the team conducted a tagged probe experiment. In this PoC, researchers embedded a unique, identifiable string, "GFW bleed," along with specific tags, into their own probe traffic. This allowed them to distinguish their experimental traffic from the vast ocean of general internet data. They sent these tagged queries from a single source IP to port 53 at a rate of approximately 30 packets per second, simultaneously sending WallBleed packets. The expectation was to see WallBleed responses containing their own "GFW bleed" tagged traffic, which they indeed observed at a somewhat low rate due to the GFW's high traffic volume. Crucially, the percentage of their own tagged traffic observed in the WallBleed responses correlated with when people were awake in China, providing strong evidence that the leaked data was representative of real, active internet usage.

Finally, to assess the global impact of WallBleed, a team from UC Boulder conducted a global probe experiment. They sent WallBleed-triggering queries to various arbitrary IP addresses around the world, using the Geolite2 database for geographical location. This experiment conclusively showed that traffic routed through China, even if originating and terminating outside the country, was susceptible to the WallBleed vulnerability. This demonstrated that the GFW's memory leak was not confined to Chinese users but posed a privacy risk to anyone whose internet traffic traversed the GFW.

Defensive Implications

▶ Watch: Potential for targeted surveillance using GFW injectors (8:00)

The findings from both WallBleed and the GeGe Networks leak carry significant defensive implications for various stakeholders, from individual internet users to network operators and human rights organizations.

For Users in Censored Regions:

• Awareness of Privacy Risks: Users in China and other countries employing GFW-like censorship systems must understand that their internet traffic, even when seemingly innocuous, could be exposed due to vulnerabilities like WallBleed. This includes sensitive data like passwords, cookies, and browsing history.
• Enhanced Circumvention: The detailed understanding of DNS injection mechanisms and their vulnerabilities can inform the development of more robust circumvention tools. Users should prioritize circumvention methods that encrypt DNS traffic (e.g., DNS-over-HTTPS, DNS-over-TLS) and route all traffic through trusted, secure VPNs or proxies that do not traverse GFW-controlled infrastructure.
• Secure Practices: While not a complete solution, users should adopt strong password practices, enable multi-factor authentication, and avoid sending sensitive information over unencrypted HTTP connections, as much of the leaked data was observed from such traffic.

For Global Internet Users and Organizations:

• Traffic Routing Awareness: The global reach of WallBleed means that users and organizations outside China whose traffic routes through Chinese networks are also at risk. Network administrators should be aware of their traffic's geographical routing and consider alternative paths or secure tunnels to avoid passing through potentially compromised censorship infrastructure.
• Advocacy and Policy: The export of censorship technology by companies like GeGe Networks to other countries (Kazakhstan, Pakistan, Myanmar, Ethiopia) highlights a growing global threat. International bodies, human rights advocates, and policymakers should push for greater transparency and accountability regarding the sale and deployment of such technologies, recognizing their potential for human rights abuses and privacy violations.

For Censorship System Operators (e.g., GFW):

• Improved Security Posture: Ironically, the leaks expose critical security flaws that GFW operators should address. Memory safety vulnerabilities like WallBleed are fundamental security issues. Implementing secure coding practices, conducting thorough security audits, and employing memory-safe languages or techniques are crucial to prevent similar leaks.
• Segmentation of Traffic: The observation that internal management traffic shared memory buffers with user traffic is a severe security misconfiguration. Operators should strictly segment internal management networks and processes from public-facing services to prevent cross-contamination and the exposure of sensitive operational data.
• Transparency (Idealistic): While unlikely, greater transparency about the GFW's operations and vulnerabilities could foster a more secure internet ecosystem.

For Security Researchers and Academics:

• Continued Measurement and Analysis: The success of WallBleed and the insights from the GeGe Networks leak underscore the importance of ongoing, active measurement and analysis of censorship systems. This research provides a roadmap for identifying new vulnerabilities and understanding the evolving tactics of state-sponsored censorship.
• Ethical Considerations: Researchers must continue to prioritize ethical considerations, as demonstrated by Sheffey's team using manual inspections for initial analysis but relying on regular expressions for further, large-scale data processing to avoid targeting individuals.

In summary, the defensive implications call for a multi-faceted approach: individual users adopting stricter security hygiene, network operators re-evaluating traffic routing, and the global community engaging in advocacy and research to counter the spread and impact of state-sponsored censorship technologies.

Key Takeaways

• The Great Firewall's DNS injectors harbor memory safety vulnerabilities, specifically WallBleed, which allows for the exfiltration of sensitive user data.
• Through malformed DNS queries, researchers could dump up to 124 bytes of memory per query, collecting billions of packets over two years.
• Leaked data included passwords, cookies, email content, UPN/SSDP traffic, and internal GFW management data, posing significant privacy risks.
• The WallBleed vulnerability affected not only users within China but also global internet traffic that routed through the GFW.
• The GFW actively attempted to patch WallBleed, but researchers were able to adapt their exploit and use changes in leaked stack/code pointers as a side channel to monitor patching efforts.
• The GeGe Networks leak provided unprecedented internal visibility into GFW components, revealing the Misa platform's DNS parser and confirming China's export of censorship software to countries like Kazakhstan, Pakistan, and Myanmar.

About the Speaker(s)

Jade Sheffey is a PhD student at the University of Massachusetts Amherst, where she is part of the Spin lab. Her advisor is Amir Monsider. Sheffey's research primarily focuses on internet censorship measurement and internet censorship circumvention. She made contributions to the WallBleed academic work, although she credits the lead authors for spearheading the project. Her expertise lies in dissecting the mechanisms of state-level censorship systems like the Great Firewall to understand their operation, identify vulnerabilities, and develop strategies to bypass them. She actively encourages collaboration with other researchers in the field.