Modern-day SOC Evolution from Open Source to Unlimited Budget

Modern-day SOC Evolution from Open Source to Unlimited Budget

Speakers: Grifter, VP Defensive Services, Coalfire; Pope, Director Technical Marketing Engineering, Corelight

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=-_jUZBMeU5w

Overview

In this insightful ShmooCon talk, "Modern-day SOC Evolution from Open Source to Unlimited Budget," Grifter (Neil Wier) and Pope offer a comprehensive look at the essential components of a robust modern Security Operations Center (SOC). Drawing from over two decades of experience leading the Black Hat Network Operations Center (NOC) – a unique, high-stakes environment where the network itself is often a target and a testing ground – the speakers detail the journey from rudimentary, off-the-shelf solutions to a multi-million dollar, enterprise-grade security infrastructure. Their discussion serves as a practical guide for organizations grappling with the perennial question: "What security tools do we actually need, and how do we acquire them given varying budget constraints?"

The talk specifically addresses the challenges and strategic decisions involved in building a resilient SOC, whether an organization operates with an "unlimited budget" or relies heavily on open-source solutions. Grifter and Pope leverage their unparalleled experience, including the Black Hat NOC's growth from supporting 1,500 attendees and 15 classes to over 25,000 attendees and 96 classes, to illustrate the evolution of security needs and the tools required to meet them. This evolution highlights the necessity of a layered defense, emphasizing the critical role of each component in detecting, responding to, and ultimately mitigating cyber threats in an increasingly complex landscape.

Background

▶ Watch: Introduction and Black Hat's original network story (0:00)

The genesis of this talk lies in the Black Hat NOC's remarkable 23-year evolution, spearheaded by Grifter. What began with a lean setup of a Cisco 2600 router, Orinoco gold card access points, and Netgear switches for a modest 1,500 attendees, quickly transformed into a behemoth supporting 25,000 attendees and 96 concurrent classes. This expansion necessitated a radical shift in philosophy, moving from basic open-source tools like pfSense running on Soekris boxes to a sophisticated enterprise-grade network. The speakers vividly recount their "shopping spree" on the Expo floor, where the reputation of Black Hat enabled them to secure top-tier vendor solutions in exchange for brand visibility, establishing a critical foundation for their advanced security operations.

This journey from resource-constrained beginnings to a state-of-the-art operation directly informs the central dilemma addressed in the talk: how organizations, regardless of their budget, can build an effective SOC. Grifter and Pope present a nuanced discussion on the trade-offs between Capital Expenditure (CAPEX) and Operational Expenditure (OPEX). Organizations with significant upfront capital might opt for vendor solutions, benefiting from dedicated support and a "neck to choke" for accountability. Conversely, those with greater operational budgets and available personnel (like university students) might find open-source solutions more viable, allowing them to "throw bodies at it" to maintain and customize tools.

A critical point of contention also explored is the platform versus best-in-class debate. While vendors aggressively push integrated security platforms, promising seamless integration, the speakers caution against the potential for vendor lock-in and the prohibitive cost and pain of a "rip and replace" scenario should prices escalate or the solution no longer meet needs. They advocate for a strategic mix of best-in-class tools, even if it means navigating more complex integrations, to maintain flexibility and control. Their recommendations for specific tools, a core finding of the talk, are derived not only from their extensive experience at Black Hat, Defcon, and RSA but also from a survey of trusted security professionals across the industry, ensuring a balanced and practical perspective.

Key Findings

▶ Watch: Black Hat NOC's massive growth and vendor partnership (2:00)

The central finding of this talk is a definitive list of eight indispensable tool categories that form the backbone of a modern, effective SOC. Grifter and Pope argue that regardless of budget, strategic investment in these areas is paramount for robust threat detection and response. These categories include Endpoint Detection and Response (EDR), Security Information and Event Management (SIM) or Data Lake, Network Detection and Response (NDR), Security Orchestration, Automation, and Response (SOAR), Threat Intelligence Platform (TIP), Sandbox for malware analysis, User and Entity Behavior Analytics (UEBA), and Identity Verified Access.

Beyond merely listing tools, the speakers provide critical insights into the strategic decision-making process. They highlight that the choice between vendor-specific, commercial solutions and open-source alternatives is not just a budget consideration but also a reflection of an organization's operational model, skill sets, and risk tolerance. The talk underscores that while "unlimited budget" scenarios might gravitate towards commercial leaders like Splunk or CrowdStrike, highly capable and actively maintained open-source projects such as Elastic Agent, Zeek, Suricata, and MISP offer compelling alternatives for those with the internal resources to implement and manage them. The ultimate takeaway is that an intelligent, integrated approach, often combining both open-source and commercial tools, is the most pragmatic path to building a resilient and adaptive security posture.

Technical Deep Dive

▶ Watch: Grifter and Pope introduce themselves and their roles (4:20)

The core of Grifter and Pope’s presentation lies in an exhaustive technical deep dive into the eight critical tool categories essential for a modern SOC, offering both commercial and open-source recommendations, alongside compelling real-world anecdotes.

Endpoint Detection and Response (EDR)

EDR is presented as the foundational layer for any organization, especially those starting with zero security tools. It sits on user endpoints and other devices, providing crucial detection and prevention capabilities. EDR offers advanced telemetry, capturing parent-child process relationships, user actions, and command execution flows, which is invaluable for incident response (IR) teams to understand the full scope of an infection. Grifter shared a stark example from the Black Hat NOC: when they began deploying EDR agents on rental laptops used for registration, an analyst quickly identified that the golden image provided by the rental company was compromised, affecting all laptops. This allowed them to remediate the issue before any sensitive attendee data was exposed. Another instance involved a cloud environment where EDR flagged a Linux cups vulnerability on an Ubuntu server. While initially alarming, EDR telemetry revealed it was a benign installation via Chrome and Snap, preventing a massive, unnecessary infrastructure rebuild.

• Top Vendor: CrowdStrike, with Microsoft and Sentinel also highly ranked.
• Open Source: Elastic Agent, Velociraptor, and self-managed solutions using osquery, Sysmon, and Fleet.

Security Information and Event Management (SIM) / Data Lake

A SIM (or Data Lake) is crucial for centralizing all logs from various sources—EDR, switches, routers, firewalls, VPC flow logs, NDR, DNS—to enable correlation. This centralization allows for faster answers to security questions, the creation of search-based alerts, and compliance reporting. It also empowers threat hunting by providing historical data. The speakers emphasized the importance of retention periods, illustrating with an anecdote from a financial organization where they discovered an attacker had been exfiltrating "every financial transaction and trade" via FTP to a Russian IP address daily for six months. Despite 30+ analysts and a strict "FTP not allowed" policy, the activity was missed because no one looked for protocols that were supposedly blocked, and logs were only retained for six months, leaving the true duration of the breach unknown. This story highlights that even with explicit policies, network traffic and log retention are paramount.

• Top Vendor: Splunk (for SIM), Snowflake (for Data Lake).
• Open Source: Elastic (Elastic Stack), OpenSearch (for SIM), Delta Lake and Apache (for Data Lake).

Network Detection and Response (NDR)

Grifter likens NDR to "video surveillance for your network," providing visibility into actual packet flow. NDR is critical because asset management is inherently difficult, and EDR agents cannot be installed on every device, especially IC/OT (Industrial Control/Operational Technology) or IoT (Internet of Things) devices. If a device generates traffic, NDR can see it, confirming its existence. Pope shared a striking example from another large financial organization where NDR detected anomalous entropy, leading to the discovery of a compromised HVAC system. This system was part of an entire building, containing over 250 hosts, that the security team didn't even know existed due to a past acquisition and backhauled internet. This incident underscored the blind spots EDR alone can leave.

A significant discussion around NDR involved encryption. While Black Hat sees 75-80% encrypted traffic (90% in Asia, driven by privacy concerns), the speakers noted that metadata from encrypted traffic (bytes in/out, certificate info) still provides value. Furthermore, they observe that in many large organizations, at least 40% of traffic remains "in the clear," and cloud environments often exhibit even worse clear-text practices due to misconfigurations. While Black Hat doesn't perform decryption, organizations with critical assets may implement decryption solutions.

• Top Vendor: Corelight.
• Open Source: Zeek and Suricata.

Security Orchestration, Automation, and Response (SOAR)

SOAR tools streamline incident response through case management, consistent playbooks, and automation. While the orchestration and automation functions can be handled by separate operations teams, many SOCs integrate them. The goal is to reduce response time and ensure consistency across analysts. Pope explained that SOAR helps automate mundane tasks like gathering context (DNS lookups, IP ownership), allowing analysts to focus on decision-making. At Black Hat, where blocking traffic is often not an option due to training and demos, they use SOAR to deploy portals to compromised users. If a user's device exhibits suspicious activity (e.g., Bitcoin mining), a portal pops up, informing them of the compromise and inviting them to the NOC for assistance, without disrupting ongoing classes. This approach helps them find the "needle in a needle stack" of expected "bad" traffic.

• Top Vendor: Palo Alto XSOAR (formerly Demisto), Splunk Phantom.
• Open Source: Primarily custom Python scripts (described as "Python scripts in a trench coat"), Shuffle, Tracecat, and Sorc.

Threat Intelligence Platform (TIP)

A TIP should be the bedrock of any security program. It centralizes and normalizes diverse threat feeds (community, paid, "Secret Squirrel" groups), allowing organizations to prioritize Indicators of Compromise (IOCs) and, more importantly, Tactics, Techniques, and and Procedures (TTPs). While IOCs (file hashes, IPs) are valuable for immediate, high-fidelity detection, their shelf life is short. TTPs, which describe how adversaries attack, are far more strategic. A TIP that provides insights into likely attacker groups for a specific industry (e.g., oil and gas) and their methods enables threat hunters to focus on specific blind spots and red teams to emulate realistic adversary behaviors, optimizing defensive investments. Grifter noted that at Black Hat, they participate in "Secret Squirrel" briefings with Las Vegas casinos, sharing intelligence on observed threats during the intense "hacker summer camp" period.

• Top Vendor: Recorded Future.
• Open Source: MISP.
• Other: Social media platforms like Twitter (now X), Blue Sky, and Mastodon, while not formal TIPs, often serve as real-time feeds for emerging vulnerabilities and IOCs.

Sandbox

A Sandbox environment allows security teams to safely detonate suspicious files (executables, DLLs, PDFs, Word docs) to observe their behavior and generate custom threat intelligence. This is particularly useful at events like Black Hat, where new malware variations and exploits are constantly encountered. Grifter recounted an incident where he observed the Wells Fargo logo on a sandbox dashboard. Further investigation revealed a user with a misconfigured split tunneling VPN who was unknowingly sending massive amounts of sensitive financial aid documents in clear text across the network to an insecure university server. The sandbox enabled rapid identification of the user and the broader insecure practice.

• Top Vendor: VirusTotal.
• Open Source: Cuckoo V2 (a well-supported fork of the original Cuckoo Sandbox).

User and Entity Behavior Analytics (UEBA)

The speakers noted that UEBA as a standalone tool has evolved. While the "user" aspect has largely migrated into identity management solutions, the "entity" behavior analytics remains crucial but is often integrated into other SOC tools like SIMs or NDRs. UEBA focuses on establishing baselines of normal behavior for users and entities, then flagging anomalies. At Black Hat, they use behavioral analytics to profile classrooms. For example, if 80 out of 100 students in a class are attacking the same web server, it's considered expected activity. However, if one student is an outlier attacking a payment processor, that warrants investigation. Grifter issued a vital warning: ensure your environment is clean before baselining, as organizations have inadvertently baselined an attacker into their environment, making their C2 traffic appear "normal."

• Top Vendor: Palo Alto XSIAM.
• Open Source: Apache Metron (though its development is slowing, many open-source SIMs incorporate behavioral analytics).

Identity Verified Access

Identity is fundamental for controlling access and maintaining accountability. It enables organizations to centralize user management, enforce Multi-Factor Authentication (MFA), and detect anomalous login behaviors (e.g., impossible travel, sudden proliferation of devices). Pope shared a critical lesson learned from a Black Hat outage: a partner implemented a preemptive patch without testing, causing a 90-minute network outage, the largest in Black Hat history. This incident led to the implementation of an identity solution, ensuring that only NOC leads could approve network commits, thereby enforcing accountability and preventing future unauthorized changes.

• Top Vendor: Duo.
• Open Source: Authentik.

Demo / Proof of Concept

▶ Watch: Addressing common SOC questions from tours and CISO's (6:00)

While the talk did not feature a live, traditional software demonstration, the entire operational framework of the Black Hat NOC serves as a powerful real-world proof of concept for the integrated security tools discussed. The speakers implicitly demonstrate how these various components work in concert to manage one of the most hostile and dynamic network environments in the world.

A key practical application highlighted, which functions as a de facto demonstration, is their use of SOAR to interact with compromised users. Instead of simply blocking or shutting down devices—which would disrupt legitimate classes and demonstrations—the Black Hat NOC employs SOAR to automatically push web-based portals to users whose devices exhibit malicious behavior. This portal informs the user that their host is compromised and offers assistance from the NOC. This unique approach, necessitated by the Black Hat environment, effectively demonstrates how orchestration and automation can be tailored to specific operational requirements, providing information and guidance rather than punitive action, while still addressing security incidents.

Defensive Implications

▶ Watch: Defining essential tools: EDR, SIEM, and data lake (7:15)

For defenders, the insights from Grifter and Pope's talk offer a clear roadmap for building and maturing a SOC, regardless of budget. The primary implication is the absolute necessity of a multi-layered, integrated security architecture.

1. Prioritize Foundational Visibility: If starting from scratch, EDR is the first critical investment. Complement this with NDR to gain comprehensive network visibility, especially for unmanaged or IoT/OT devices that EDR cannot reach. The network truly "doesn't lie."
2. Centralize and Correlate Logs: Implement a SIM or Data Lake to centralize all security logs. This is fundamental for correlation, efficient threat hunting, compliance, and reducing the Mean Time To Detect (MTTD) and Mean Time To Respond (MTTR). Pay close attention to log retention periods to ensure historical context for long-running breaches.
3. Leverage Strategic Threat Intelligence: Move beyond just IOCs and focus on TTPs through a TIP. This enables proactive defense, guiding threat hunters to specific areas of concern and empowering red teams to emulate realistic adversary behaviors relevant to your industry.
4. Embrace Automation and Orchestration: Utilize SOAR to standardize incident response playbooks and automate repetitive tasks. This frees up analysts to focus on complex investigative work rather than data collection, contributing to the "golden hour" of incident response.
5. Analyze Behavior, But Be Wary: Integrate behavioral analytics (often within SIM, NDR, or Identity tools) to detect anomalies. However, critically, ensure your environment is clean before establishing baselines to avoid "baselining an attacker."
6. Strengthen Identity Controls: Implement robust Identity Verified Access solutions with MFA for all users. This is crucial for accountability, preventing unauthorized access, and detecting anomalous user behavior.
7. Consider Open-Source for Learning and Budget: For students or organizations with limited budgets, platforms like Security Onion and Malcolm provide excellent environments to learn and implement SOC functions. Many open-source tools rival their commercial counterparts when adequately resourced.
8. Prepare for the Future: Stay abreast of emerging technologies like Large Language Models (LLMs). The speakers foresee LLMs assisting with SOC upskilling, generating signatures/rules, abstracting complex query languages, and even performing SOAR and Breach Assessment Simulation (BAS) functions.
9. Anticipate External Pressures: Be aware that increasing regulation and cyber insurance requirements will likely steer tool choices and demand higher levels of demonstrable security posture.

Key Takeaways

• A modern SOC requires a comprehensive, integrated suite of tools including EDR, SIM/Data Lake, NDR, SOAR, TIP, Sandbox, UEBA, and Identity Verified Access.
• Organizations must strategically balance vendor solutions (CAPEX, support, "neck to choke") with open-source alternatives (OPEX, flexibility, community support) based on their budget and internal capabilities.
• Foundational visibility through EDR for endpoints and NDR for network traffic is paramount, especially for unmanaged devices and to confirm what policies claim is not happening.
• Threat intelligence should prioritize Tactics, Techniques, and Procedures (TTPs) over short-lived Indicators of Compromise (IOCs) to enable proactive defense and realistic red team engagements.
• Security Orchestration, Automation, and Response (SOAR) is crucial for standardizing incident response, automating mundane tasks, and accelerating overall detection and response times.
• Identity management with Multi-Factor Authentication (MFA) is a critical control for accountability and detecting anomalous user behavior, as demonstrated by the Black Hat NOC's experience with network outages.
• The future of SOC operations will increasingly integrate Large Language Models (LLMs) for analyst upskilling, automated rule generation, and advanced SOAR/Breach Assessment Simulation functions, alongside growing pressures from regulation and cyber insurance.

About the Speaker(s)

Grifter (Neil Wier) is a highly respected figure in the cybersecurity community with a career spanning over two decades. He currently serves as the VP of Defensive Services for Coalfire. Grifter has been the lead for the Black Hat NOC for an impressive 22 and a half years, overseeing its evolution into a world-class security operation. Beyond Black Hat, he is also the Defcon Department head, responsible for running all contests and events at Defcon for 23 years, and has led the SOC at RSA for the last six years. He is also a member of the Black Hat and Defcon review boards, solidifying his reputation as a dedicated "nerd" in the field.

Pope is the Director of Technical Marketing Engineering at Corelight and serves as the SOC lead for the Black Hat NOC, working closely with Grifter. His contributions extend beyond commercial roles into the community, where he helps run St. Con, a security conference in Utah, and founded the DC435 group in Salt Lake, which brings together 40-70 people monthly. In addition to his cybersecurity work, Pope also owns a company specializing in web accessibility, demonstrating a diverse skill set and commitment to broader technological and societal impact.