Tales from the Black Hat NOC

Tales from the Black Hat NOC

Speakers: pope (James Pope), Director of Technical Marketing Engineering, Corelight; SOC Lead for the Black Hat NOC

Conference: SAINTCON

YouTube: https://www.youtube.com/watch?v=ZXznGvlcOeY

Overview

James "Pope" Pope's "Tales from the Black Hat NOC" delivers a candid and eye-opening account of the real-world security challenges observed by the Network Operations Center (NOC) at one of the world's premier cybersecurity conferences, Black Hat. As the SOC Lead for the Black Hat NOC, Pope and his team are tasked with building, maintaining, and defending a temporary, high-stakes network environment for thousands of security professionals. This talk transcends theoretical discussions, offering a ground-level view of the pervasive vulnerabilities, misconfigurations, and outright malicious activities that persist even among a highly security-aware audience.

The presentation serves as a stark reminder that despite advancements in security tools and awareness, fundamental security hygiene often breaks down, particularly when users operate outside their controlled corporate environments. Pope highlights numerous instances where security professionals, CTOs, and even security vendors inadvertently expose sensitive data or operate infected machines due to negligence, misconfiguration, or outdated practices. The insights shared are critical for any organization or individual looking to understand the practical application of network defense and the enduring importance of basic verification.

The talk underscores the NOC's unique "black hat positive" philosophy, where legitimate attacks conducted as part of conference training are differentiated from genuinely malicious activities. Pope emphasizes the reliance on robust network visibility tools and a dedicated team of threat hunters to navigate this complex landscape, providing invaluable lessons on how to protect an organization's assets in an increasingly hostile digital world.

Background

▶ Watch: Infected CTO's laptop discovered during Black Hat talk (0:00)

The Black Hat NOC operates under unique and challenging circumstances. Its primary mission is twofold: first, to engineer and deploy a fully functional network infrastructure capable of supporting thousands of attendees, trainers, and vendors, complete with hundreds of access points, switches, and firewalls. Second, and crucially, to secure this environment against a spectrum of threats ranging from accidental misconfigurations to intentional malicious attacks. Unlike typical enterprise environments, the NOC cannot mandate the installation of endpoint agents on attendees' devices, making network visibility the cornerstone of their defensive strategy.

The NOC's security stack is primarily built on open-source tools like Zeek (for network telemetry and rich logs) and Suricata (for intrusion detection and prevention), augmented by commercial offerings like the Corelight stack. This approach emphasizes the power of free, community-driven tools for comprehensive network monitoring. A significant challenge for the NOC is distinguishing between legitimate, in-scope activities—such as attendees participating in hacking classes and intentionally running exploits—and genuinely malicious out-of-scope attacks targeting other users or the conference infrastructure. This distinction is managed through a concept called "black hat positive," where certain detections are acknowledged as valid but not acted upon because they align with the conference's educational objectives.

Pope stresses the importance of threat hunting as a proactive measure, defining it as the search for "unknown things," distinguishing it from mere detection of known indicators. The NOC leverages frameworks like MITRE ATT&CK and the lesser-known MITRE DEFEND to categorize and understand observed network behaviors. Beyond the technical aspects, the NOC also plays a vital educational role, conducting "knock tours" to explain their operations and findings to curious attendees, further fostering a culture of security awareness. This context sets the stage for the myriad of security issues they uncover year after year.

Key Findings

▶ Watch: Scale and structure of the Black Hat NOC (4:05)

The Black Hat NOC consistently uncovers a wide array of security issues, highlighting common pitfalls even among a security-conscious audience. Pope categorized these findings into several recurring themes:

1. Pre-Infected Machines: A significant number of attendees arrive at the conference with already compromised devices. Examples include:

• A CTO giving a presentation found to have an AutoIT Trojan on their machine.
• Instances of NetSupport RAT and Atomic macOS info stealer on attendee laptops.
• A particularly concerning case involved a cell phone infected with SMS spy C2, exfiltrating all text messages without the owner's knowledge, even from individuals working at security companies.
• These incidents underscore the pervasive nature of malware and the need for continuous vigilance, irrespective of an individual's security role or the presence of endpoint detection and response (EDR) solutions.

1. Pervasive Cleartext Communication (HTTP): A shocking volume of sensitive data is transmitted over unencrypted HTTP, often due to misconfigurations or poorly developed applications.

• Location Data: Weather applications and even a security tool (unnamed, but described as a "guard tool") were observed sending precise GPS coordinates over HTTP every minute.
• Voice Search and Translation: Applications attempting to guess the next word in a voice search or leveraging services like Google Translate were sending entire sentences and conversations over cleartext.
• Chat Applications: Both consumer and internal organizational chat applications were found transmitting full conversations, company names, department IDs, employee details, and even attached zip files containing database information (including past employee cell numbers) over HTTP.
• Insecure Logins and APIs: Applications would initiate connections over HTTP, sending cookies or JSON Web Tokens (JWTs) before being redirected to HTTPS, making them vulnerable to session hijacking by an attacker on the same network. Self-hosted services like Synology for images, Guacamole for remote access, OwnCloud for file syncing, and self-hosted email were frequently misconfigured to use HTTP, exposing personal photos, vacation itineraries, and critical access.
• Self-hosted GitLab instances were found exposing API keys, code, and repository access over HTTP, presenting "keys to the kingdom" for attackers.

1. Security Tools Failing to Secure: Ironically, some security tools and services were found to be sources of vulnerability themselves.

• BigFix: This endpoint management tool was observed broadcasting detailed process lists, patch levels, and severity information over cleartext HTTP, essentially performing reconnaissance for any attacker on the same network.
• Zscaler V1 Proxy: Despite users believing they were protected by a "green checkbox," the V1 proxy was observed allowing the NOC to see every URL visited, even for HTTPS sites, due to its operational model. Pope strongly recommended moving to V2.
• Splunk Forwarders: Self-hosted Splunk instances, especially for development or test environments, were often configured without TLS, causing all endpoint telemetry (application logs, host logs, login logs) to be sent in cleartext over the internet.
• Vendor Agents: A new, unnamed vendor agent, controlling both client and cloud-side infrastructure, was found to be sending all its collected logs in cleartext over HTTP, a significant security lapse in 2025.
• EDRs: One specific EDR product was noted for not only failing to stop malicious activity but also sending all its collected endpoint data, including parent-child process relationships, in cleartext.

1. Network Misconfigurations and VPN Issues:

• Multicast DNS: The NOC discovered instances where firewall rules intended to segment network zones were bypassed through multicast DNS, allowing traffic to cross boundaries it shouldn't.
• VPN Split Tunneling: A recurring issue involved VPN split tunneling misconfigurations. When a user's local subnet (e.g., at the conference or a hotel) accidentally matched an IP range configured for local routing in their VPN profile, traffic intended for the secure VPN tunnel was instead sent in cleartext over the local network. This led to highly sensitive data, such as mortgage applications and FAFSA forms, being exposed.
• DNS over HTTPS (DoH) Misconfigurations: Applications attempting to use DoH were found to be misconfigured, sending DNS queries over HTTP instead of HTTPS, undermining the intended privacy and security benefits.

1. Illegal and Out-of-Scope Activities: Despite the conference's educational focus, some attendees engaged in activities that were clearly illegal or outside the permitted scope.

• Attendees from classes, despite explicit warnings, often tested newly learned attack techniques against out-of-scope targets like payment processors or remote systems.
• Instances of SQL injection attempts and SMB over the internet were observed, with the latter exposing highly sensitive data like breach and attack simulation reports, zone identifiers, shares, portal details, and API credentials, often from large financial institutions.
• The NOC actively intervenes in such cases, leveraging their ability to quickly identify and locate individuals, often resulting in immediate cessation of the activity.

These findings collectively paint a picture of widespread security vulnerabilities, often stemming from a lack of verification, insecure defaults, or a misunderstanding of how applications and tools handle sensitive data.

Technical Deep Dive

▶ Watch: Black Hat NOC's mission and protected assets (5:55)

The Black Hat NOC's operational philosophy is deeply rooted in network visibility and the intelligent use of data. Given the inability to deploy endpoint agents on most attendee devices, their primary focus is on comprehensive network traffic analysis.

At the core of their monitoring stack are open-source tools like Zeek (formerly Bro) and Suricata. Zeek is an open-source network analysis framework that provides a high-level, scriptable language for network security monitoring. It excels at generating rich, semantic logs (e.g., HTTP logs, DNS logs, SSL logs, file activity logs) that go far beyond traditional packet captures, offering deep insights into application-layer activity. Suricata, on the other hand, is a high-performance, open-source network intrusion detection system (NIDS), intrusion prevention system (NIPS), and network security monitoring (NSM) engine. It uses signature-based detection, protocol analysis, and pattern matching to identify known threats and suspicious activities. The NOC integrates these with the Corelight stack, which commercializes and enhances Zeek data for easier deployment and analysis.

For initial investigations and basic verification, Pope strongly advocates for simple, readily available tools:

• TCPdump: A command-line packet analyzer for capturing and analyzing network traffic. Pope encourages individuals to "TCP dump that thing" to see what their applications are truly sending.
• Wireshark: A graphical network protocol analyzer that allows for deep inspection of individual packets and protocol flows.

The pervasive issue of cleartext communication (HTTP) is a central technical theme. Examples included:

• Weather apps and security tools sending GPS coordinates in plaintext HTTP POST requests to domains like weatherforecast.io. This is a fundamental failure in transport layer security.
• Voice search and translation applications transmitting entire phrases and sentences over HTTP, often leveraging services like Google Translate without enforcing HTTPS. This exposes conversational data to anyone on the network.
• Chat applications, both consumer and enterprise, were observed sending full conversations, user metadata (company, department, name, ID), and even attached files containing sensitive database information (employee directories, cell numbers) via unencrypted HTTP. This highlights a severe lack of attention to secure communication protocols in application development.
• A common pattern for insecure login/API calls involved an initial HTTP request that included a session cookie or JWT (JSON Web Token), followed by a server-side redirect to HTTPS. An attacker observing the initial HTTP exchange could capture these credentials and potentially hijack the session before the secure connection is established.

The misconfiguration of security tools themselves presented critical vulnerabilities:

• BigFix, an endpoint management tool, was observed transmitting comprehensive system information—including process lists, installed patches, and vulnerability severities—over HTTP. This effectively provides attackers with a detailed reconnaissance report without needing to scan the target. Pope urged users to "check a SSL box somewhere" in their BigFix configurations.
• Zscaler V1 proxy was noted for its inherent design flaw that allowed the NOC to observe all visited URLs, even for HTTPS sites, despite the user's perception of security. This is due to the proxy's man-in-the-middle decryption capabilities, which, when misconfigured or used with V1, expose the traffic to intermediate observers.
• Splunk forwarders, when sending data to self-hosted Splunk instances lacking TLS (Transport Layer Security), would transmit all collected endpoint telemetry—including application logs, host logs, and login events—in cleartext. This transforms a security monitoring solution into a data leakage vector.
• An unnamed vendor agent for a new cloud service was criticized for allowing HTTP communication in 2025, especially when the vendor controlled both the client agent and the cloud server, eliminating any excuse for cleartext transmission.
• One particular EDR (Endpoint Detection and Response) product not only failed to prevent malicious activity but also sent all its telemetry, including detailed parent-child process relationships, in cleartext.

Self-hosting initiatives, driven by privacy concerns, frequently resulted in insecure deployments:

• Synology network-attached storage (NAS) devices, Guacamole remote access gateways, OwnCloud file sync solutions, and personal email servers were often configured to run over HTTP, exposing sensitive data, files, and remote access credentials.
• A self-hosted AI agent workflow used for incident response was found to be accessed over HTTP, leaking sensitive details about an organization's IR processes and potentially ongoing incidents.
• Crucially, self-hosted GitLab instances, often chosen for sensitive code and IP, were misconfigured to use HTTP, exposing API keys, repository access tokens, and source code—effectively providing "keys to the kingdom."

The NOC also leverages AI and LLMs (Large Language Models) to enhance their threat hunting capabilities. They developed a system where analysts could query an LLM about IP addresses, FQDNs, or MAC addresses to gain context and relevant threat intelligence. This was integrated with Slack for ease of use, providing quick insights into network activity and mapping observations to MITRE DEFEND categories. Furthermore, LLMs are used to automate the filtering of "black hat positive" alerts by comparing observed network activity (e.g., multiple users in a class targeting the same destination with specific attack types) against the published curriculum abstracts of training sessions. This significantly reduces alert fatigue, allowing human analysts to focus on genuine threats.

Finally, the talk highlighted network segmentation failures, specifically through multicast DNS bridging what should have been isolated network zones. This demonstrated that even robust firewall deployments require continuous validation.

Demo / Proof of Concept

▶ Watch: Free open-source tools for network monitoring (Zeek, Suricata) (6:49)

The talk "Tales from the Black Hat NOC" is primarily an exposition of real-world observations and lessons learned from actively defending the Black Hat network, rather than a presentation of specific demonstrations or proof-of-concept exploits performed by the speaker. James Pope, as the SOC Lead, details numerous instances of vulnerabilities, misconfigurations, and malicious activities his team uncovered on the live conference network.

While Pope did not conduct live demos of exploiting these vulnerabilities, he did describe the process of validating some of their findings. For example, when encountering applications attempting to perform DNS over HTTPS (DoH) but doing so over HTTP, the NOC team would validate that this misconfiguration was indeed possible and demonstrable. Similarly, the process of identifying and confirming cleartext transmission of sensitive data, such as GPS coordinates or chat messages, involved direct observation of network traffic using tools like Zeek and Suricata, which effectively served as "proof" of the vulnerability in a real-world context. The core "proof" in this presentation is the extensive collection of anonymized, real-world data and anecdotes from the Black Hat NOC's operations.

Defensive Implications

▶ Watch: The human element: life as a threat hunter (8:00)

The insights from the Black Hat NOC offer critical defensive implications for organizations and individuals alike:

1. Prioritize Network Visibility: Given the prevalence of misconfigured endpoints and the difficulty of deploying endpoint agents universally, robust network security monitoring (NSM) is paramount. Tools like Zeek and Suricata (or commercial solutions built upon them, like Corelight) provide invaluable telemetry for detecting anomalies, identifying cleartext transmission of sensitive data, and understanding network behavior. Organizations should invest in detailed traffic analysis at ingress/egress points and across internal network segments.

1. Verify, Don't Trust: The "green checkbox" phenomenon and the belief that EDRs or VPNs inherently guarantee security are dangerous. Defenders must actively verify their security controls and application behavior. This means routinely performing TCPdump or Wireshark captures on critical systems, especially when connecting to untrusted networks (hotel Wi-Fi, coffee shops, conferences). Validate that VPNs are routing traffic as expected, that proxies are enforcing security, and that applications are using HTTPS for all sensitive communications.

1. Enforce HTTPS Everywhere: The sheer volume of sensitive data transmitted over HTTP is alarming. Application developers must enforce HTTPS for all communication, especially for logins, API calls, chat, location services, and any data deemed sensitive. Organizations should implement strict policies and code reviews to prevent cleartext transmission. For self-hosted services (e.g., Synology, Guacamole, OwnCloud, GitLab, email servers), proper TLS/SSL configuration is non-negotiable.

1. Secure Your Security Tools: It's a critical paradox when security tools themselves become a source of vulnerability. Organizations must rigorously audit the configurations of their EDRs, Splunk forwarders, BigFix agents, Zscaler proxies (ensure V2 is used), and other security infrastructure. Ensure all telemetry is encrypted in transit and that agents aren't inadvertently leaking sensitive system information. Hold vendors accountable for secure-by-default configurations and prompt resolution of cleartext vulnerabilities.

1. Audit VPN Split Tunneling: VPN configurations, particularly split tunneling, require careful review. Administrators should understand the implications of allowing local traffic and implement controls to prevent sensitive data from bypassing the VPN tunnel, especially when users connect from unpredictable network environments. Consider disabling local routing at conferences or high-risk locations, or providing clear guidance to users.

1. Know Your Environment (and Enrich Your Data): Effective incident response hinges on context. Defenders must have a deep understanding of their network topology, ingress/egress points, standard application behavior, and asset inventory. Enriching network logs with metadata like VLANs, department names, building locations, and device owners significantly accelerates incident identification and response. As Pope stated, knowing "that's in Denver, floor three, building 103" is far more effective than sifting through MAC tables.

1. Leverage Automation and AI for Alert Fatigue: The "needles in a haystack" problem is real. Using LLMs and automated playbooks, as demonstrated by the NOC, can help filter out benign alerts (like "black hat positives" or known non-exploitable vulnerabilities) and escalate only truly critical incidents. This allows human analysts to focus their limited attention on high-priority threats.

1. Educate Users and Developers: Continuous education for both end-users and developers is crucial. Users need to understand the risks of public Wi-Fi, the importance of VPNs, and the dangers of misconfigured applications. Developers need training on secure coding practices, enforcing HTTPS, and the secure implementation of common features (like chat, voice, and API integrations).

1. Proactive Threat Hunting: Don't just wait for alerts. Actively hunt for unknown threats and anomalies using frameworks like MITRE DEFEND. This proactive approach can uncover subtle indicators of compromise that might bypass traditional signature-based detections.

Key Takeaways

• Network visibility is paramount: In environments where endpoint agents aren't feasible or reliable, robust network security monitoring using tools like Zeek and Suricata provides essential telemetry to detect threats and misconfigurations.
• "Green checkboxes" are not enough: Organizations and individuals must actively verify their security posture, rather than blindly trusting the perceived security of EDRs, VPNs, or application interfaces. Regular use of tools like TCPdump and Wireshark is crucial for this verification.
• Cleartext communication is shockingly prevalent: Despite decades of security advancements, sensitive data—including GPS locations, private conversations, login credentials, API keys, and even incident response workflows—is frequently transmitted over unencrypted HTTP due to application flaws or misconfigurations.
• Security tools can introduce vulnerabilities: Ironically, tools designed to enhance security, such as BigFix, Zscaler V1 proxy, self-hosted Splunk forwarders, and some EDRs, can inadvertently expose sensitive data if not properly configured or if they operate with insecure defaults.
• Self-hosting requires rigorous security: While driven by privacy, the trend of self-hosting services like Synology, Guacamole, OwnCloud, and GitLab often leads to severe security lapses (e.g., lack of TLS) that expose critical data and access credentials.
• Context and automation are key to defense: Leveraging enriched network data, AI/LLMs, and automated playbooks can significantly reduce alert fatigue, allowing security teams to differentiate between benign "black hat positives" and genuine, critical threats, thereby improving response efficiency.

About the Speaker(s)

James Pope, affectionately known as "Pope," is a highly experienced and active member of the cybersecurity community. He serves as the Director of Technical Marketing Engineering at Corelight, a company that commercializes and enhances the open-source Zeek network security monitor.

Beyond his corporate role, Pope is deeply embedded in the conference circuit, notably as the SOC Lead for the Black Hat NOC, where he and his team are responsible for the demanding task of building and securing the network for one of the world's largest cybersecurity events. His involvement extends to other prominent conferences, including being a committee member for SAINTCON, helping with BSides Salt Lake, and running a monthly meetup for BSides Cache. He also plays a significant logistical role in booking keynotes and premier speakers for SAINTCON, often spending years convincing top talent to present. His extensive hands-on experience in these diverse roles gives him a unique perspective on real-world security challenges and solutions.