Inside a large self-hosted VRP

Inside a large self-hosted VRP

Speakers: Sam Erb, Security Engineer, Google

Conference: Bug Bounty Village

YouTube: https://www.youtube.com/watch?v=t0-oXbczPLs

Overview

Sam Erb, a Security Engineer at Google who helps manage the expansive Google and Alphabet Vulnerability Reward Program (VRP), delivered an insightful talk at Bug Bounty Village detailing the unique operational philosophies of a large, self-hosted VRP. Beyond offering a rare glimpse into the internal workings of one of the world's most significant bug bounty initiatives, Erb unveiled a previously undisclosed and novel Cross-Site Scripting (XSS) vulnerability. This vulnerability leveraged a sophisticated understanding of GZIP compression algorithms to inject arbitrary strings into text/html responses, even when the server did not explicitly control the Huffman encoding tables.

The presentation served a dual purpose: to educate the security community on the nuances of Google's VRP, including its distinct triage processes, reward structures, and disclosure policies, and to demonstrate an advanced exploitation technique that pushes the boundaries of web security research. Erb's talk highlighted how Google's security engineers actively work to escalate the impact of reported bugs, ensuring researchers are fairly compensated for the maximum potential severity of their discoveries. This commitment to impact escalation was exemplified by the detailed breakdown of the GZIP XSS, a complex bug that required deep technical expertise to understand and exploit.

This article delves into the intricacies of Google's VRP, exploring its operational mechanics and philosophical underpinnings. Crucially, it provides a comprehensive technical deep dive into the GZIP XSS vulnerability, explaining the underlying compression algorithms and the ingenious methods used to bypass their protective mechanisms. The discussion also covers the defensive implications of such an attack, offering actionable advice for organizations to mitigate similar risks in their own applications.

Background

▶ Watch: Introduction and Google VRP overview (0:00)

Google's Vulnerability Reward Program stands as one of the largest and most established in the industry, having paid out nearly $12 million last year across all Alphabet VRPs and a lifetime total of $65 million. Unlike many programs that rely on platforms like HackerOne or Bugcrowd, Google's VRP is predominantly self-hosted, managing everything from report intake and triage to vulnerability reproduction, escalation, and reward processing internally. While payment processing has recently partnered with Bugcrowd, the core security engineering functions remain in-house. This internal operation allows for unique processes and philosophies that differentiate it from standard VRP models.

The VRP team, comprised of security engineers on rotation, handles approximately 250 reports per week. Of these, about 15% to 25% are escalated for further investigation, and roughly 50% of those escalated reports ultimately receive a reward. This high volume necessitates efficient, yet thorough, processes. Sam Erb highlighted several unique aspects of Google's VRP:

1. Vocabulary Discrepancy: The term "triage" within Google's VRP primarily signifies an initial spam check and a decision for further follow-up, not necessarily a confirmed, reproducible bug ready for reward, which can differ from expectations set by other platforms. Erb suggests an industry-wide standardization of vocabulary to manage researcher expectations better.
2. Impact Escalation: Google's security engineers actively work to escalate the maximum impact of a reported vulnerability. If a researcher reports a bug with seemingly low impact, and Google's internal teams can demonstrate a higher severity, the researcher is rewarded based on that maximum impact. This win-win approach helps prove the bug's true severity to product teams and ensures researchers are fairly compensated. This practice extends to live hacking events, known as bug spot events, where engineers collaborate directly with researchers to maximize impact.
3. Reward Shopping: Given the vast scope of Google and Alphabet's various VRPs (Google, Android, Chrome, Cloud, Open Source), overlapping reports are common. In such cases, Google's VRPs will "reward shop," submitting the bug to all relevant programs and rewarding the highest possible payout. This acts as a simple "max function" for researcher rewards.
4. No Reputation-Based Triage: Google's VRP strives to judge each report solely on its merits, rather than on the researcher's past reputation. While top researchers are recognized internally, their submissions are evaluated objectively.
5. Absence of CVSS for Rewards: Unlike many programs, Google's internal reward process does not utilize CVSS (Common Vulnerability Scoring System) scores. Instead, reports are initially sorted by AI and then manually reviewed, with severity assessed on a P0 to P2 scale for product filing. Rewards are primarily based on user impact, with higher payouts for vulnerabilities affecting a larger number of Google users. CVSS scores are generated only when filing a CVE (Common Vulnerabilities and Exposures), a process separate from the internal reward determination.
6. Disclosure Encouragement: Google actively encourages public disclosure of vulnerabilities, aligning with the principles of Project Zero. They work with researchers to ensure proper disclosure, provided it adheres to their responsible disclosure policies.
7. Incident Rewards: Minor issues, often referred to as "incidents" (e.g., inadvertently uploaded files to GitHub), receive a standard reward of $500. While exceptions exist, this policy can sometimes surprise external researchers expecting higher payouts for what they perceive as data leaks.

The GZIP XSS vulnerability discussed in the talk emerged from this unique VRP environment. A top researcher, "Tom," reported something unusual on a google.com subdomain. The bug involved a Content-Type: text/html response where the input was attacker-controlled plaintext, and the content was gzipped before being returned. The core question was whether an attacker could inject an arbitrary string, specifically an XSS payload, into the gzipped output under these specific and seemingly restrictive conditions. This challenge led to a deep dive into the mechanics of GZIP compression, revealing a novel method for exploitation.

Key Findings

▶ Watch: Introducing a unique bug for impact escalation (3:20)

The talk presented two primary key findings that offer significant insights into both advanced web exploitation and the operational philosophy of a leading VRP:

1. Novel GZIP XSS Vulnerability: The most significant technical finding was the discovery and successful exploitation of a Cross-Site Scripting (XSS) vulnerability leveraging the GZIP compression algorithm. This technique demonstrated the ability to inject arbitrary strings, such as <script> tags, into the decompressed output of a gzipped text/html response. Crucially, this was achieved under conditions where the attacker did not control the Huffman encoding table, a scenario previously thought to be much harder to exploit compared to cases where the table could be manipulated directly. This finding highlights a sophisticated understanding of compression internals and their security implications.
2. Unique Operational Aspects of Google's Self-Hosted VRP: The talk provided an unparalleled look into the distinct practices and principles governing Google's VRP. Key operational findings include the VRP's proactive approach to impact escalation, where security engineers maximize the severity and corresponding reward for researchers; the practice of reward shopping across various internal VRPs to ensure the highest payout; the eschewal of CVSS for internal reward calculations in favor of direct user impact; and a strong encouragement for public disclosure. These insights reveal a VRP designed to be highly researcher-friendly while maintaining rigorous internal security standards.

Together, these findings underscore the complexity of modern web security, where vulnerabilities can arise from the unexpected interplay of common protocols and application logic, and demonstrate how a well-structured VRP can foster both groundbreaking research and robust defense.

Technical Deep Dive

▶ Watch: Technical details of the mysterious GZIP bug (4:00)

The GZIP XSS vulnerability presented by Sam Erb is a testament to the depth of understanding required to uncover subtle flaws in widely used protocols. The core challenge was to inject an arbitrary string, specifically an XSS payload like <script>, into the decompressed output of a gzipped text/html response, where the input was attacker-controlled plaintext, and crucially, the Huffman encoding table was not directly controllable by the attacker.

To understand the exploit, one must first grasp the fundamentals of GZIP compression. GZIP is a file format that uses the Deflate algorithm for compression. Deflate itself is a combination of two older algorithms:

1. LZ77 (Lempel-Ziv 1977): This algorithm identifies repeated byte sequences within the input data. Instead of storing the repeated sequence, it replaces it with a "pointer" or "tag" that indicates a previous occurrence of the same sequence. This tag consists of two parts: a distance (how far back in the already-processed data the sequence occurred) and a length (the length of the repeated sequence). For efficiency, the LZ77 match logic typically only kicks in for sequences that are at least three characters long, as shorter matches are often less efficient than simply encoding the characters directly.
2. Huffman Coding: This is a variable-length encoding scheme. It assigns shorter bit sequences (codes) to characters or symbols that appear more frequently in the input data and longer codes to less frequent ones. This results in an overall reduction in the total number of bits required to represent the data. The Huffman coding table is dynamically built based on the frequency of characters in the input and is often included as part of the GZIP header, or defined by a "fixed" table. As data is processed, the Huffman table can be adjusted to optimize compression.

The vulnerability arose from a specific configuration: a google.com subdomain was returning a Content-Type: text/html response, but the content itself was attacker-controlled plaintext that was then gzipped. When a browser receives a text/html response, it attempts to render it, even if it contains unprintable characters or binary data from the gzip stream. The goal was to manipulate the gzipped output such that, upon decompression, it would contain a valid HTML <script> tag.

The primary difficulty lay in the dynamic nature of Huffman coding and the LZ77 algorithm. If the attacker simply tried to inject <script> directly, the compression algorithm would likely compress it in unpredictable ways, either by finding LZ77 matches or by generating a Huffman table that renders the output unreadable or incorrect.

Sam Erb and the team devised a clever strategy to overcome this:

1. Controlling LZ77 Matches: The first step was to prevent the LZ77 algorithm from creating unwanted matches that would disrupt the payload. By understanding that LZ77 typically avoids matches for strings shorter than three characters, the researchers constructed the initial input string using a pattern of two-character strings separated by unique delimiters. For example, AB_CD_EF_GH_.... This pattern ensures that no two-character sequences repeat in a way that LZ77 would compress them into a tag, thus giving the attacker more granular control over the raw character stream.
2. Building a Predictable Huffman Table: The next challenge was to control the Huffman coding table. Since the table is built dynamically based on character frequencies, simply appending a payload like <script> would alter the frequencies of characters and thus change the entire table, making it impossible to predict the bit values for the desired payload. The trick here was to "brute force" the Huffman table construction. By randomizing the length and characters used in the delimiters of the initial input string (e.g., AB_CDE_FGHI_...), the researchers could generate a new Huffman table with each attempt.
3. Stabilizing the Huffman Table for Payload Injection: This was perhaps the most ingenious part. When the attacker wanted to append specific characters (e.g., the Huffman-encoded representation of hi, which might be JtWa in a given table), simply adding them to the end would again change the character frequencies and thus the Huffman table. To counteract this, the researchers employed a "deletion" trick. When appending the desired characters (e.g., J, t, W, a), they would simultaneously "delete" a corresponding number of previous occurrences of those exact characters from the beginning of the input string. This ensures that the total count of each character in the entire input remains constant, thereby stabilizing the dynamically generated Huffman coding table. While this is a somewhat naive approach and might not work for extremely long or complex payloads, it proved effective for typical XSS payloads.
4. Brute-Forcing Byte Alignment and Endianness: Even with a stable Huffman table and controlled LZ77, the final challenge was ensuring the injected bit values aligned correctly on byte boundaries and that the endianness (byte order) was correct, as GZIP is an older algorithm. Since there's a 1 in 8 chance of achieving correct byte alignment for a given input, the solution involved brute-forcing the input string with a sufficiently large Huffman table until the desired alignment was achieved. Careful attention to endianness was also critical.

The culmination of these techniques allowed the researchers to construct an input string that, when gzipped, would decompress to include an arbitrary string. Sam Erb demonstrated this with a Golang example, noting that different implementations (e.g., Python vs. Golang) can generate slightly different gzip files due to variations in their compression libraries. The Golang code successfully printed a <script> tag into the output, and crucially, this was validated to work in a web browser, confirming the XSS on google.com. This vulnerability was escalated through Google's internal processes and ultimately fixed, highlighting the effectiveness of their VRP in identifying and remediating highly technical and novel security flaws.

Demo / Proof of Concept

▶ Watch: Detailed explanation of GZIP and Deflate compression (4:50)

The demonstration of the GZIP XSS vulnerability was highly practical and served as a clear proof of concept for the sophisticated attack technique. Sam Erb presented a live example using Golang code. He explained that this particular implementation of GZIP generation in Golang produced slightly different results compared to, for instance, Python, underscoring the subtle complexities that can arise when dealing with low-level protocol implementations.

The Go example was designed to encapsulate the four key steps outlined in the technical deep dive:

1. Avoiding LZ77 matches for short strings.
2. Dynamically constructing a Huffman table.
3. Stabilizing the Huffman table by strategically "deleting" prior character occurrences when appending the desired payload.
4. Brute-forcing byte alignment to ensure the injected bits formed valid characters upon decompression.

When the Golang program was executed, it generated a specific GZIP stream. The crucial outcome was that when this stream was decompressed, it contained an injected HTML <script> tag. Erb further validated this by demonstrating that the generated gzipped content, when served with a Content-Type: text/html header, successfully executed the embedded script in a web browser. This confirmed the Cross-Site Scripting (XSS) vulnerability on a google.com subdomain. The successful demonstration not only validated the theoretical underpinnings of the attack but also illustrated its practical exploitability, leading to its classification as an XSS and subsequent remediation through Google's internal processes.

Defensive Implications

▶ Watch: Visualizing Huffman coding and compression rates (6:00)

The GZIP XSS vulnerability highlights several critical defensive implications for organizations and developers, particularly concerning how they handle user-controlled input, content types, and compression:

1. Beware of Content-Type: text/html with Compressed User Input: The most direct implication is to avoid serving attacker-controlled input within a Content-Type: text/html response if that input is also being compressed (especially with GZIP). Even if the input is intended to be "plaintext," the combination with HTML content type and dynamic compression can create an avenue for arbitrary code injection. Developers should be extremely cautious about this specific configuration.
2. Rethink "Plaintext" Assumptions: The attack demonstrates that even if a server believes it's handling "plaintext," the underlying mechanisms of compression, combined with how browsers interpret text/html responses, can turn seemingly innocuous data into executable code. Robust input validation and output encoding are paramount, even for data that appears to be non-HTML.
3. Comprehensive Output Encoding: All user-controlled data that is reflected in a response, regardless of whether it's compressed or not, must be properly output encoded for its specific context (e.g., HTML entity encoding for HTML contexts, JavaScript encoding for script contexts). Relying on the compression algorithm to obscure or prevent malicious payloads is a dangerous assumption.
4. Review Compression Implementations: Security teams should audit their applications for scenarios where user-controlled input directly influences content that is subsequently gzipped and served as text/html. This includes content management systems, user profile pages, or any service that might dynamically generate HTML pages incorporating user data before compression.
5. Understand Low-Level Protocol Interactions: This vulnerability underscores the importance of understanding the intricate details of widely used protocols like GZIP. Security professionals and developers should not treat such protocols as black boxes but rather understand their inner workings, including aspects like LZ77 and Huffman coding, to anticipate potential exploitation vectors.
6. Browser Interpretation of Malformed Data: Browsers are often designed to be fault-tolerant, attempting to render even malformed or unexpected content. This fault tolerance, while generally user-friendly, can create security vulnerabilities when combined with attacker-controlled input and specific content types. Defenders should not rely on a browser's inability to render "unprintable characters" as a security boundary.
7. Threat Model Compression: Include compression mechanisms in threat modeling exercises. Consider how an attacker might manipulate input to influence the compressed output and how this might interact with other layers of the application stack, particularly content type headers.

By addressing these defensive considerations, organizations can better protect themselves against sophisticated attacks that exploit the often-overlooked interplay between compression algorithms and web application logic.

Key Takeaways

• A novel GZIP XSS vulnerability allows arbitrary string injection into text/html responses, even when Huffman encoding tables are not directly controlled, by manipulating LZ77 matches, brute-forcing table construction, and stabilizing character frequencies.
• Google's VRP operates with unique philosophies, including proactively escalating bug impact for higher researcher rewards, reward shopping across internal programs, and prioritizing user impact over CVSS scores for internal payouts.
• Understanding the low-level mechanics of compression algorithms like LZ77 and Huffman coding is crucial for uncovering and mitigating advanced web vulnerabilities.
• Organizations must be acutely aware of the security implications when serving attacker-controlled input that is both gzipped and delivered with a Content-Type: text/html header, and enforce robust output encoding.
• Google actively encourages disclosure of vulnerabilities, working with researchers to ensure proper publication, and offers a standard $500 reward for "incidents" like inadvertently shared internal files.
• The industry would benefit from a standardized vocabulary for bug bounty programs to better manage researcher expectations regarding triage and vulnerability lifecycle stages.

About the Speaker(s)

Sam Erb is a Security Engineer at Google, where he plays a pivotal role in running the extensive Google and Alphabet Vulnerability Reward Program (VRP). With a wealth of experience in the security domain, Sam has personally triaged thousands of bug reports submitted by researchers worldwide and has also contributed to the community by reporting several bugs himself and winning various contests in the past. Having joined Google approximately three years prior to this talk, he brings an insider's perspective on the unique challenges and operational intricacies of managing one of the largest self-hosted VRPs.