Testing Trust Relationships: Breaking Network Boundaries

Testing Trust Relationships: Breaking Network Boundaries

Speakers: Michael Gianarakis (Chief Product Officer, Searchlight)

Conference: Bug Bounty Village

YouTube: https://www.youtube.com/watch?v=1lSFP-CABCM

Overview

In his talk at Bug Bounty Village, Michael Gianarakis, Chief Product Officer at Searchlight and co-founder of Assetnote, presented a critical analysis of modern network access controls, particularly IP whitelisting, in the context of cloud and zero-trust architectures. The presentation, co-authored with Jordan, an engineer at Searchlight, highlighted how shifting architectural trends have introduced new complexities and vulnerabilities that often go unnoticed by traditional security scanning methods. Gianarakis argued that the prevalent reliance on broad or poorly configured IP whitelists by organizations, often advised by SaaS vendors, creates significant security gaps that attackers can exploit to bypass network boundaries and access internal resources.

The core of the talk revolved around demonstrating how security doesn't exist in a vacuum, but rather reflects current technological trends. The move towards SaaS providers and zero-trust, no-VPN based access networks means more applications are externally exposed, albeit behind various controls like mutual TLS, IP whitelisting, or SSO. However, the implementation of these controls frequently suffers from fundamental flaws, making them less effective than intended. Gianarakis showcased real-world examples from bug bounty programs where these misconfigurations led to critical vulnerabilities, including Remote Code Execution (RCE) on sensitive systems, simply by changing the origin of network requests.

To address this pervasive issue, Gianarakis introduced Newtower, a new open-source tool designed to help security researchers and organizations identify these network boundary flaws. Newtower enables users to perform comparative scans from different geographic locations and cloud providers, effectively highlighting discrepancies in application responses that indicate vulnerable whitelisting logic. The talk underscored the importance of treating security scanning as a "science experiment," comparing a control (residential IP) against various experimental conditions (cloud provider IPs) to empirically uncover hidden access paths.

Background

▶ Watch: The problem: Flawed whitelisting in modern cloud (2:00)

The landscape of network access control has undergone a significant transformation with the widespread adoption of modern cloud architectures and the growing reliance on SaaS providers. This shift has moved away from traditional internal-only networks towards environments where applications are designed for external interaction, often necessitating connectivity with third-party vendors. While this paradigm offers flexibility and scalability, it introduces substantial complexity for engineers tasked with deploying and architecting robust network controls. The move towards zero-trust models, which inherently distrust all network connections, paradoxically often still relies on flawed foundational mechanisms like IP whitelisting for access gating.

Despite the conceptual shift, the practical application of network controls frequently falls short. Many SaaS providers and vendors, in an attempt to simplify integration, advise customers to implement broad or fundamentally flawed IP-based whitelisting rules. This advice often involves whitelisting entire cloud regions, such as GCP US Central 1, US East 1, or AWS US East 1, as seen in GitLab's recommendations for shared runners, or the entirety of Azure for Microsoft Power BI/Power Platform. While some vendors, like GitHub, Cloudflare, Atlassian, IAM (for Adobe AM), Okta, and Datadog, provide more specific IP ranges or utilize mutual TLS, their documentation can still lead to misconfigurations, such as users forgetting to enable SSL verification for Cloudflare Workers. This broad whitelisting approach is a direct descendant of practices from two decades ago, yet it's now applied in a far more interconnected and complex environment, making its shortcomings more pronounced.

The problem is exacerbated by the fact that engineers, seeking to facilitate necessary communication between their applications and vendor services (for deployments, config management, API access, uptime monitoring, etc.), often follow these instructions without thoroughly understanding the security implications. This creates a scenario where an application, intended to be internally accessible or restricted to specific trusted sources, becomes exposed to a vast attack surface within a whitelisted cloud region. Furthermore, some whitelisting relies on geolocation, a rarer but equally problematic mechanism, where applications are only accessible from specific geographic regions, such as Citrix self-hosted hosts restricted to a company's own region. This combination of broad IP ranges and location-based restrictions forms the bedrock of the vulnerabilities explored in the talk, effectively creating hidden access paths to sensitive internal resources for anyone originating from a whitelisted cloud provider or region.

Key Findings

▶ Watch: Better vendor advice: GitHub, Cloudflare, IAM (3:10)

The central finding of Michael Gianarakis's presentation is that IP whitelisting, as commonly implemented in modern cloud and zero-trust environments, is a fundamentally broken security concept. This flaw stems from the practice of whitelisting broad IP ranges or entire cloud regions, inadvertently granting access to an attacker who can originate requests from within that same whitelisted space. The talk provided compelling real-world examples from bug bounty engagements, illustrating how this vulnerability translates into critical security risks.

Gianarakis showcased several instances where internal resources, typically protected by restrictive access policies, became accessible when requests originated from a US East 1 AWS IP address, compared to an "access denied" or "not found" response from a residential IP. Examples included:

• A T-Mobile host requiring a TLS certificate, which denied access from a residential IP but granted it from US East 1.
• An internal poll application that returned "not found" from a residential IP but presented a login page when accessed from US East 1.
• An Adobe application that returned "forbidden" from a residential IP but became accessible from US East 1.

These weren't just theoretical bypasses; Gianarakis explicitly stated that "pretty much all of them led to critical vulnerabilities, specifically RCE on really sensitive systems." This underscores the severe impact of such network boundary flaws.

Beyond ingress, the talk also highlighted the implications for egress traffic. Similarly broad egress IP allowed lists are often configured, meaning if an attacker gains a foothold in an application (e.g., via a compute feature like Fastly at the edge, GitHub Actions, Bitbucket pipelines, or Cloudflare Workers), they can leverage the application's whitelisted egress to communicate with other internal services. By identifying the target's cloud region and spinning up infrastructure as close as possible, attackers can effectively proxy through the compromised application to access internal resources, rendering the whitelists "effectively useless" from an egress perspective. This means even SaaS platforms with "good hygiene" in publishing explicit IP addresses can be bypassed if an attacker controls meaningful egress traffic from the platform itself.

To empirically prove the hypothesis that IP whitelisting is broken, a large-scale experiment was conducted. Scanning over 18 million hosts within AWS US East 1 from both inside and outside the region, the researchers identified approximately 7,000 instances of differential behavior. This included cases where a 200 OK status code was returned from US East 1, while any other status code (e.g., 403, 404) was returned from outside the region. This analysis revealed 122 unique organizations on US East 1 allowing access only from within that region. Notably, these organizations included high-profile entities such as Adobe, General Electric, and the United States Department of Health and Human Services, demonstrating the widespread nature and critical implications of this vulnerability across diverse sectors.

Technical Deep Dive

▶ Watch: How to exploit trust relationships for access (5:00)

The technical approach to identifying these network boundary flaws is rooted in a "science experiment" methodology. The core idea is to establish a control (a scan from a typical residential IP address) and compare its results against experiments (scans originating from various cloud providers and regions, such as AWS or Azure). Any significant difference in responses between the control and experimental scans reveals a potential network boundary flaw.

For a bare-bones approach, one could manually use a tool like HTTPX to make requests from a residential machine and then from a cloud instance, subsequently diffing the HTTP responses. However, to scale this analysis and provide more robust detection, the researchers developed a sophisticated methodology and a dedicated tool.

The empirical analysis to prove the widespread nature of flawed IP whitelisting involved a large-scale scanning operation focused on AWS US East 1, chosen for its popularity and high likelihood of being whitelisted. The process comprised several key steps:

1. Port Scanning: Over 18 million hosts within AWS US East 1 were scanned for port 443 (HTTPS) from outside US East 1 using masscan. This returned approximately 2.5 million hosts with port 443 open. Masscan was configured with a rate of 100,000 packets per second and completed the scan in about 4 minutes per instance.
2. HTTP Request Issuance: For all identified hosts, Zgrab2 was used to issue HTTP requests from both outside AWS and inside AWS US East 1. Zgrab2 was set with a concurrency of 500 and took approximately 55 minutes (almost an hour) to complete its banner grabbing and HTTP requests. The results were piped through JQ for precise output extraction.
3. Data Ingestion and Analysis: Around 2 million records (approximately 5 GB of data) were imported into ClickHouse, a high-performance column-oriented database. This ingestion was remarkably fast, taking only 6 minutes. SQL queries were then used for analysis.

Infrastructure and Configuration Details:

The scanning infrastructure was based on Ubuntu 24.04 LTS. Critical to the consistency of masscan results were specific IP tables and security group rules to prevent packet hijacking by the kernel or blocking by AWS. A recommended source port range for masscan was 61000 to 61255, with corresponding IP tables rules to allow traffic from this range:

These rules ensure that masscan's user-space network stack can handle packets without kernel interference, crucial for accurate large-scale scanning.

Query Logic for Anomaly Detection:

The primary SQL query aimed to identify instances where:

• A 200 OK status code was received when the request originated from AWS US East 1.
• Any other status code (e.g., 401, 403, 404, 500) was received when the request did not originate from US East 1.

This logic allowed the aggregation of raw data from two tables (inside AWS vs. outside AWS) to pinpoint differential behavior indicative of whitelisting. Python was then used to parse these aggregated results and generate visualizations.

The new tool, Newtower, automates this "diffing" logic. It incorporates a sophisticated Levenshtein distance algorithm to compare HTTP responses, minimizing false positives by identifying meaningful content differences rather than just status code changes. This algorithm, while not explicitly detailed by Gianarakis (who credited engineer Jordan), is crucial for accurately detecting subtle variations in responses that signify a network boundary bypass. Newtower supports testing from various cloud environments and connectors, including GitHub Actions, GitLab CI pipelines, Bitbucket pipelines, AWS API Gateway, EC2 instances, and direct SSH connections to EC2 for diffing. The web version of Newtower further enhances this by leveraging Cloudflare Durable Objects to test connectivity from specific Cloudflare data centers across different regions (US West, US East, Western Europe, Eastern Europe, Asia, Australia/New Zealand), providing a global perspective on potential access discrepancies without requiring user-side configuration.

Demo / Proof of Concept

▶ Watch: Real-world ingress exploitation examples (T-Mobile) (6:00)

The talk featured a demonstration of Newtower, the open-source tool developed to automate the detection of these network boundary flaws. Newtower is designed to be highly versatile, capable of spinning up compute resources in various cloud environments and integrating into existing CI/CD pipelines to perform comparative scans.

The core functionality demonstrated was Newtower's ability to diff HTTP responses between a "control" request (originating from the user's local residential IP or a configured baseline) and "experimental" requests (originating from specified cloud providers or regions). The tool requires an initial configuration to set up connections to different cloud environments, which, once done, becomes a "set and forget" process. Users can provide Newtower with a single URL or a list of URLs to test.

The demo showed Newtower's output in a terminal, clearly highlighting real differences in HTTP requests. The tool provides both a concise summary and detailed information on the discrepancies found. A key technical feature is its custom diffing logic, which employs the Levenshtein distance algorithm. This algorithm measures the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word into the other, effectively comparing the content of HTTP responses to reduce false positives and accurately pinpoint significant changes that indicate a network access bypass.

The web version of Newtower, available on the tools.slcyber.io platform, further enhances this capability. It offers a zero-config experience, allowing users to simply input a target URL and select regions for scanning (e.g., Cloudflare or AWS). A significant update for the web version is its ability to test connectivity from various Cloudflare data centers across the globe, including US West, US East, Western Europe, Eastern Europe, Asia, and Australia/New Zealand. This is achieved by leveraging Cloudflare Durable Objects, which allows Newtower to proxy requests through specific Cloudflare regions, providing a truly distributed and comprehensive check for regional access restrictions. This feature is particularly powerful for identifying geolocation-based whitelisting flaws, as well as general cloud-region specific access issues. The demo showcased the intuitive web interface, making advanced multi-region scanning accessible without complex setup.

Defensive Implications

▶ Watch: Exploiting trust for egress and data exfiltration (7:00)

The insights from "Testing Trust Relationships: Breaking Network Boundaries" offer crucial guidance for defenders seeking to fortify their network security posture in modern cloud environments. The primary defensive implication is a resounding call to re-evaluate and move beyond simplistic IP whitelisting as a primary access control mechanism, especially when dealing with broad cloud provider IP ranges.

Here are key actions defenders should consider:

1. Scrutinize Vendor Whitelisting Advice: Organizations must critically assess the whitelisting recommendations provided by SaaS vendors. If a vendor advises whitelisting entire cloud regions (e.g., AWS US East 1, whole Azure), this should be a red flag. Defenders should push back, request more granular IP ranges, or seek alternative, more robust authentication and authorization mechanisms.
2. Implement Granular Access Controls: Instead of broad IP whitelists, defenders should prioritize more granular controls. This includes:

• Mutual TLS (mTLS): For service-to-service communication, mTLS ensures that both client and server authenticate each other using certificates, significantly reducing the risk of unauthorized access even if an IP is whitelisted.
• Stronger Authentication and Authorization: Implement robust authentication (e.g., multi-factor authentication, strong SSO policies) and fine-grained authorization (role-based access control) at the application layer, rather than solely relying on network-level controls.
• Network Segmentation: Even within cloud environments, segmenting networks and applying strict security group rules that only allow necessary ports and protocols between specific resources can contain potential breaches.

1. Adopt Multi-Vantage Point Scanning: Traditional security scanning from a single location is insufficient. Defenders should integrate multi-source scanning into their security testing routines. Tools like Newtower or custom scripts can be used to scan their attack surface from various cloud regions (AWS, Azure, GCP) and different geographic locations. This "science experiment" approach helps identify discrepancies in application responses, revealing hidden access paths that would otherwise be missed.
2. Monitor Egress Traffic: Understand that egress whitelists can also be exploited. If an application is compromised (e.g., via RCE), broad egress rules can facilitate data exfiltration or lateral movement. Implement strict egress filtering that only allows traffic to known, necessary destinations and monitor for anomalous outbound connections.
3. Regularly Review and Audit Access Policies: Network access policies, including IP whitelists, should not be set and forgotten. Regular audits are essential to ensure that whitelists are still necessary, are as narrow as possible, and do not inadvertently grant overly permissive access as infrastructure evolves or vendor requirements change.
4. Leverage Cloud-Native Security Features: Cloud providers offer a suite of security services (e.g., AWS WAF, Azure Firewall, GCP Cloud Armor, IAM policies, VPC Service Controls). Defenders should fully utilize these features to enforce least privilege access and protect resources at multiple layers, moving away from simple IP-based allow lists.

By adopting these defensive strategies, organizations can significantly reduce their exposure to network boundary bypasses and enhance the overall security of their cloud-native applications.

Key Takeaways

• IP whitelisting is fundamentally flawed in modern cloud architectures: Relying on broad IP ranges or entire cloud regions for access control creates significant security gaps, as anyone originating from that whitelisted space can bypass intended restrictions.
• Attackers can exploit cloud infrastructure for ingress and egress: By spinning up compute resources within a target's whitelisted cloud region, attackers can access internal applications or leverage compromised applications' broad egress rules to reach other internal services.
• Traditional single-source scanning is insufficient: Security assessments must adopt a "science experiment" approach, comparing application responses from residential IPs (control) against those from various cloud providers and regions (experiments) to uncover differential access.
• Newtower automates the detection of network boundary flaws: The tool allows users to perform comparative HTTP request diffing from multiple vantage points (e.g., GitHub Actions, AWS EC2, Cloudflare Durable Objects) using algorithms like Levenshtein distance to identify subtle but critical access discrepancies.
• Broad vendor whitelisting advice is a major contributor to risk: Many SaaS providers recommend insecurely broad IP whitelists, forcing organizations to expose their internal resources to a wider, less controlled attack surface.
• Defenders must adopt granular, multi-layered security: Moving beyond IP whitelisting, organizations should implement strong authentication (mTLS, MFA), fine-grained authorization, rigorous network segmentation, and continuous multi-vantage point scanning to secure their cloud environments.

About the Speaker(s)

Michael Gianarakis is the Chief Product Officer at Searchlight and was formerly a co-founder and CEO of Assetnote. He has a deep background in offensive security, particularly within the bug bounty community, where he and his team have developed numerous innovative tools and techniques to identify and exploit vulnerabilities. His work often focuses on pragmatic approaches to security challenges, informed by real-world hacking experiences and a strong belief that much of the innovation in offensive security originates from the competitive bug bounty space.

Jordan, an engineer at Searchlight and formerly at Assetnote, also contributed to the development of the Newtower tool and the research presented in the talk. Although not present at the conference, his technical expertise, particularly in areas like the Levenshtein distance algorithm for diffing, was instrumental in creating the robust tooling discussed.