Flipping Bits: Your Credentials Are Certainly Mine

Flipping Bits: Your Credentials Are Certainly Mine

Speakers: Stök, Johoi

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=7lUPTxNNxM0

Overview

In "Flipping Bits: Your Credentials Are Certainly Mine," security researchers Stök and Johoi delve into the intriguing and often misunderstood phenomenon of bit squatting. This talk exposes how subtle, single-bit changes in domain names, whether caused by environmental factors or other system anomalies, can redirect sensitive traffic and lead to the leakage of valuable credentials and data. The speakers present a compelling blend of historical context, practical experimentation, and a critical re-evaluation of long-held assumptions regarding the root causes of such "bit flips."

The presentation meticulously unpacks the mechanics of how a single bit alteration can transform a legitimate domain like googleapis.com into a subtly different coogleapis.com, and subsequently, how attackers can leverage these "flipped" domains to passively collect sensitive information. Stök and Johoi not only demonstrate the feasibility of bit squatting but also challenge the prevailing theory that cosmic rays are the primary driver of such events, presenting their own experimental findings from a four-month observation period during a significant solar storm. This research underscores the pervasive nature of hardware and software vulnerabilities that can manifest as bit flips, posing a significant, often overlooked, threat to digital security.

The talk is crucial for security professionals, developers, and system administrators who need to understand the nuanced ways in which data integrity can be compromised at a fundamental level. By dissecting the technical infrastructure required to exploit bit flips and offering insights into their true origins, Stök and Johoi provide actionable intelligence for bolstering defenses against this sophisticated form of attack. Their work highlights the importance of comprehensive security strategies that consider not just logical vulnerabilities but also the physical and environmental factors that can influence data processing.

Background

▶ Watch: Introduction to bit flips: typo squatting for computers (1:20)

The concept of "bit squatting" emerges as a digital analogue to the more widely recognized "typo squatting." Typo squatting exploits human error, where users inadvertently type a slightly incorrect domain (e.g., visma.com vs. vismq.com), leading them to a malicious site. Bit squatting, as coined by Artem Dinaburg at Defcon 19, operates on a similar principle but targets machine-level errors. Instead of a human typo, a single bit flip—a change from a 1 to a 0 or vice-versa within a computer's memory or transmission—can alter a domain name. For example, wordpress.com could become gordpress.com with just one bit flip. Dinaburg's original research primarily focused on these bit flips occurring at the DNS level.

The prevailing theory for these machine-induced bit flips has long pointed to environmental factors, specifically cosmic rays causing DRAM errors. Cosmic rays, high-energy particles from space, can strike memory cells, altering their charge state and flipping a bit. This phenomenon has been documented in various contexts, leading to the assumption that such errors could be a significant source of bit flips in DNS lookups or other data transmissions. The speakers were inspired by an anecdote from a colleague, Dylan, who registered coogleapis.com (a one-bit flip away from googleapis.com) and began receiving a "shitload of loot," including API tokens and JavaScript access, which he attributed to cosmic rays.

To understand how bit squatting works, it's essential to grasp the standard DNS HTTP request flow. When a client application (e.g., a web browser) wants to access a domain, it first queries its local DNS server. This server, if it doesn't have the record cached, contacts root DNS servers, which direct it to the authoritative name servers for the top-level domain, and eventually to the name server for the specific domain. This name server then provides the IP address of the target web server. The client then initiates an HTTP (or HTTPS) connection to this IP address.

The critical vulnerability in this process arises if a bit flip occurs during the DNS resolution phase. If wordpress.com flips to gordpress.com at any point in the resolution chain, the client will receive the IP address of gordpress.com instead of the intended server. Crucially, the client application itself is often unaware that a bit flip has occurred; it simply proceeds to connect to the resolved IP address. This redirection allows an attacker who owns the bit-flipped domain to intercept requests intended for the legitimate service, potentially capturing sensitive data.

However, the advent of SSL/TLS (Secure Sockets Layer/Transport Layer Security) introduced a significant hurdle for bit squatters. When a client connects to an HTTPS endpoint, it expects the server to present a certificate that matches the domain name it intended to connect to. If a bit flip occurs (e.g., wordpress.com becomes gordpress.com), and the client connects to gordpress.com over HTTPS, the server at gordpress.com must present a valid certificate for gordpress.com. If it presents a wordpress.com certificate, the client will terminate the connection due to a certificate mismatch, rendering the bit squatting attempt ineffective for secure traffic. This necessitates that bit squatters acquire valid SSL/TLS certificates for their flipped domains, often using services like Let's Encrypt, or employ wildcard certificates to cover multiple potential flips.

Key Findings

▶ Watch: Real-world success: Dylan's coogleapis.com and collected loot (2:15)

The primary and most significant finding of Stök and Johoi's research is the disproving of cosmic rays as the primary, or even a significant, cause of observable bit flips leading to credential leakage in their specific experimental setup. Despite deploying an extensive monitoring infrastructure and running it for four months and 23 days, including a period coinciding with the strongest solar storm in over 20 years (in May, hammering geomagnetic things and breaking stuff on Earth), they observed no anomalies directly attributable to cosmic ray-induced bit flips. This directly challenges the long-held assumption prevalent in many security discussions and research papers that cosmic rays are a substantial driver of such errors in practical, internet-facing systems.

Despite the debunking of the cosmic ray hypothesis, the research reaffirms that bit squatting is a highly effective method for passively collecting sensitive information. Dylan's initial experience with coogleapis.com receiving "shitload of loot" (API tokens, JavaScript access, and other data) was not an isolated incident. The implication is that while cosmic rays might not be the direct cause, other forms of hardware errors, software bugs, or environmental factors (like heat or voltage fluctuations, as hinted at by the speakers) are still causing these bit flips, or more generally, misconfigurations and errors that lead to traffic redirection.

Another key finding, illustrated by Stök's personal experience with the l0.fi domain vs. lo.fi (a bank's domain), is the persistent role of human error in generating traffic to typo-squatted or bit-squatted domains. While the talk focuses on machine-induced bit flips, the speaker's revelation that he was receiving customer data due to users mistyping a bank's domain underscores that the "loot" captured by such domains isn't always from a sophisticated bit flip attack. It can also be a result of simple human mistakes, which attackers can equally exploit. This highlights a broader vulnerability where subtle domain variations, regardless of their origin, can lead to data exposure.

Finally, the research implicitly demonstrates the complexity and resourcefulness required to successfully implement a bit squatting operation in the modern HTTPS-dominated internet. The initial "easy steps" described by Dylan (buy flip domains, catch requests, log) are insufficient. The need to acquire and manage SSL/TLS certificates (especially wildcard certificates or repeated Let's Encrypt renewals every 90 days), set up robust DNS and web server infrastructure (Bind9, Nginx, Flask), and manage log files, all contribute to a non-trivial operational overhead. This finding provides a realistic assessment of the attacker's perspective and the necessary prerequisites for such an attack.

Technical Deep Dive

▶ Watch: Understanding the normal DNS resolution and OAuth flow (3:30)

The core mechanism exploited in bit squatting revolves around DNS resolution and the subsequent HTTP/HTTPS connection. When a single bit flips within a domain name (e.g., google.com to coogle.com or wordpress.com to gordpress.com), it effectively creates a new, distinct domain. If an attacker registers this bit-flipped domain, they can then control its DNS records.

The standard DNS resolution process involves:

1. A client application (e.g., browser) initiating a request for a domain.
2. The client querying its configured DNS resolver.
3. The resolver traversing the DNS hierarchy (root servers, TLD name servers, authoritative name servers) to find the IP address associated with the domain.
4. The resolver returning the IP address to the client.
5. The client then making an HTTP/HTTPS connection to that IP address.

If a bit flip occurs at any point during steps 1-3, the client will receive the IP address of the bit-flipped domain. The client, unaware of the original intent, will proceed to connect to the attacker's server. This is where the technical challenge for the attacker arises, especially with the widespread adoption of HTTPS.

For HTTPS traffic, the client performs a certificate validation step. The server must present an SSL/TLS certificate that is valid for the domain name the client thought it was connecting to. If the client intended to connect to original.com but due to a bit flip, resolved to flipped.com and connected to the attacker's server, the attacker's server must present a certificate for flipped.com. If it presents a certificate for original.com (or no valid certificate at all), the client will issue a certificate error and terminate the connection, preventing data leakage.

To overcome this, the speakers outlined the necessary infrastructure for a successful bit squatting operation:

1. Domain Acquisition: Identifying and registering relevant one-bit flip domains. This involves systematically calculating all possible single-bit flips for high-value target domains (e.g., popular SaaS providers, cloud services, package repositories). Their experiment involved 25 such domains.
2. DNS Catch-all Server: A Bind9 server configured to act as an authoritative DNS server for the bit-flipped domains. This server must be set up to "catch all" requests for any subdomain of the bit-flipped domain and direct them to the attacker's web server.
3. Reverse Proxy: An Nginx server acting as a reverse proxy in front of the application layer. Nginx can handle incoming HTTP/HTTPS requests, direct them to the appropriate backend, and importantly, terminate SSL/TLS connections.
4. Dynamic Application Layer: A backend application, such as one built with Flask (a Python web framework), to process incoming requests. This application would log all incoming request details, including headers, body, query parameters, and any embedded credentials (e.g., API keys, session tokens, JavaScript access tokens).
5. Certificate Management: This is a critical component.

• Pre-staging Specific SANs: For specific subdomains, the attacker would need to obtain individual certificates listing each specific subdomain in the Subject Alternative Name (SAN) field. This is cumbersome due to the sheer number of possible subdomains and the 90-day renewal cycle of services like Let's Encrypt.
• Wildcard Certificates: The more practical approach is to acquire wildcard SSL/TLS certificates (e.g., .flipped.com). A wildcard certificate can secure any first-level subdomain (e.g., www.flipped.com, api.flipped.com, auth.flipped.com). However, it typically does not cover second-level subdomains (e.g., auth.dev-api.flipped.com), which would still require a specific certificate or another wildcard (.dev-api.flipped.com). This presents a trade-off between coverage and complexity. The speakers humorously mentioned the extreme "vanity route" of buying a .KCM TLD, which costs $185,000 and has a seven-year application window, guaranteeing full control over all subdomains.

1. Logging Infrastructure: Robust logging is essential to store the intercepted data securely and efficiently, often in "multiple log files."

The speakers emphasized that their initial "shady kind of half-ass setup" for monitoring over two years was not sufficient for scientific rigor, prompting them to build a more robust system for their experiment. This sophisticated setup was crucial for both capturing the "loot" and for definitively testing the cosmic ray hypothesis.

Demo / Proof of Concept

▶ Watch: How a bit flip hijacks DNS resolution (pre-SSL) (4:20)

The "demo" in this talk was not a live, interactive demonstration of an attack, but rather a detailed exposition of the experimental setup and methodology used by Stök and Johoi to investigate bit squatting and its causes. Their proof of concept was the very infrastructure they built and operated, and the results derived from its operation.

Their setup involved:

1. Target Selection: They identified 25 high-traffic domains associated with popular SaaS providers, cloud providers, and package repositories. These services were chosen because they are commonly used and generate a significant volume of API calls and other network traffic, increasing the likelihood of observing bit flips or misconfigurations.
2. Flip Domain Acquisition: For each target domain, they calculated and registered corresponding one-bit flip domains. For instance, if example.com was a target, they would register examp1e.com (if 'l' and '1' share a bit difference) or other variants.
3. Infrastructure Deployment: As detailed in the technical deep dive, they deployed a comprehensive system including:

• A Bind9 server configured to be authoritative for their flip domains and to catch all DNS queries.
• An Nginx proxy to handle incoming HTTP/HTTPS traffic.
• A backend application (likely Flask) to process requests and log data.
• Let's Encrypt certificates for their flip domains, requiring regular 90-day renewals, or potentially wildcard certificates where applicable.

1. Monitoring Period: The system was run continuously for four months and 23 days.
2. Environmental Correlation: Crucially, their monitoring period coincided with a significant solar storm in May, described as the strongest in over 20 years. This event provided an ideal "natural experiment" to test the cosmic ray hypothesis, as increased solar activity typically correlates with higher cosmic ray flux on Earth, potentially leading to more bit flip events.

The outcome of this proof of concept was particularly striking:

• Despite the long monitoring period and the occurrence of a major solar storm, the researchers found no anomalies in their logs that could be definitively attributed to cosmic ray-induced bit flips. They stated, "there was nothing. There were no anomalies at all. We couldn't match anything on this. This is the perfect fucking perfect storm." This strongly suggests that cosmic rays are not the primary driver of the observed bit squatting traffic.
• However, they did implicitly confirm that traffic does arrive at bit-flipped domains, even if not from cosmic rays. Stök's personal anecdote about his l0.fi domain receiving traffic intended for a bank's lo.fi domain served as a powerful illustration. In this case, the "loot" (customer data) was a result of typos from customers, not bit flips. This highlights that regardless of the underlying cause (hardware error, software bug, or human typo), registering subtly altered domains can lead to significant data exposure.

The "demo" thus served to both validate the concept of bit squatting as an effective data collection method and to critically reassess the scientific understanding of its primary triggers, shifting focus away from cosmic rays towards other, potentially more mundane but equally impactful, sources of error.

Defensive Implications

▶ Watch: Overcoming the SSL/TLS certificate challenge for bit squatting (5:00)

The research presented by Stök and Johoi, while debunking the cosmic ray hypothesis, solidifies the threat posed by bit squatting. Defenders must recognize that even if cosmic rays aren't the primary cause, hardware faults, software bugs, or other environmental factors (such as heat or voltage fluctuations) can still lead to single-bit errors that redirect traffic. Therefore, robust defensive strategies are essential:

1. Proactive Monitoring for DNS Anomalies: Organizations should implement advanced DNS monitoring systems capable of detecting queries to domains that are very similar to their legitimate ones, especially those that are a single bit-flip away. This requires generating a list of potential bit-flipped variants for critical domains and setting up alerts if queries or connections are observed towards these domains.
2. Strengthened Cryptographic Integrity Beyond TLS: While SSL/TLS provides transport security, it doesn't prevent a client from connecting to a bit-flipped domain if that domain presents a valid certificate. Developers should consider implementing additional integrity checks for critical assets. For example, using Subresource Integrity (SRI) for JavaScript files and other resources can ensure that even if a bit flip redirects to a malicious server, the client will refuse to execute scripts that don't match expected cryptographic hashes.
3. Robust Error Handling and Input Validation: Applications should be designed to handle unexpected or malformed inputs gracefully. While this usually refers to user input, it can extend to how applications process network responses or DNS resolutions. Explicitly validating domain names or IP addresses against a known whitelist, where practical, can mitigate risks.
4. DNSSEC Adoption: While not a silver bullet, DNSSEC (Domain Name System Security Extensions) adds cryptographic signatures to DNS data, helping to ensure the authenticity and integrity of DNS responses. This can prevent an attacker from injecting false DNS records for a bit-flipped domain if the original domain's zone is DNSSEC-signed and the client's resolver validates DNSSEC. However, it doesn't protect against a bit flip occurring within the client's memory or during the initial query to a non-DNSSEC-validating resolver.
5. Awareness of Hardware and Software Reliability: The talk implicitly underscores the importance of hardware reliability and robust software development practices. Organizations should invest in high-quality hardware with error-correcting code (ECC) memory where data integrity is paramount. Regular software updates and patching can address bugs that might contribute to data corruption or misdirection.
6. Internal Network Segmentation and Least Privilege: Even if a bit flip occurs, limiting the scope of what an intercepted request can achieve is crucial. Network segmentation can prevent lateral movement, and adhering to the principle of least privilege for API tokens and credentials reduces the impact of their leakage.
7. User Education and Awareness: The l0.fi example vividly demonstrates that human typos remain a significant vector for data leakage. Educating users, particularly employees, about vigilance when entering URLs and scrutinizing domain names can reduce the risk of accidental redirection.
8. Automated Certificate Monitoring: For organizations that manage numerous domains and subdomains, automated tools for monitoring valid certificates and detecting rogue certificates issued for similar-looking domains can be beneficial. Certificate Transparency logs can be a valuable resource for this.

By combining these technical and operational measures, defenders can build a more resilient infrastructure against the subtle yet potent threat of bit squatting, regardless of the precise origin of the bit flips.

Key Takeaways

• Bit Squatting is a Viable Threat: Single-bit flips in domain names can effectively redirect traffic to attacker-controlled domains, leading to the leakage of sensitive data like API tokens and JavaScript access credentials.
• Cosmic Ray Hypothesis Debunked (in this context): Extensive experimentation during a major solar storm found no evidence that cosmic rays are a primary driver of observable bit flips causing credential leakage in practical internet-facing systems.
• Sophisticated Infrastructure Required: Successfully exploiting bit squatting on the modern HTTPS-dominated internet demands a complex setup, including custom DNS servers (Bind9), reverse proxies (Nginx), dynamic application backends (Flask), and robust SSL/TLS certificate management (e.g., wildcard certificates or Let's Encrypt renewals).
• Human Error Remains a Major Factor: Even when investigating machine-induced bit flips, the talk highlighted that simple human typos in domain entry (like l0.fi vs. lo.fi) can lead to significant data exposure, emphasizing the need for comprehensive defensive strategies.
• Proactive Defense is Crucial: Organizations must implement proactive measures such as monitoring for bit-flipped domain queries, employing robust input validation, leveraging DNSSEC, and considering additional cryptographic integrity checks (e.g., SRI) to protect against this subtle attack vector.
• Beyond Bit Flips, Data Integrity Matters: The research underscores a broader message about the fragility of data integrity in digital systems, urging defenders to consider a wider range of physical and environmental factors, alongside software bugs and human error, that can compromise data at a fundamental level.

About the Speaker(s)

The talk was delivered by Stök and Johoi. While specific titles and companies were not provided in the metadata, the transcript reveals insights into their expertise and interests. Stök demonstrated a deep understanding of DNS, HTTP/HTTPS protocols, and server infrastructure, having personally set up and managed the experimental environment for bit squatting. He also shared a personal interest in self-hosting for privacy and curiosity, a practice he has engaged in for over 25 years. Johoi contributed to the presentation, reinforcing the technical aspects and the collaborative nature of their research. Both speakers exhibited a confident, analytical approach to security research, combining practical implementation with a critical examination of established theories.