Is Visibility an Art of Possible or Impossible?

Is Visibility an Art of Possible or Impossible?

Speakers: Lu, Principal System Engineer, The Home Depot

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=IWhcbLN8ceY

Overview

In an era of increasingly complex and distributed IT environments, achieving comprehensive visibility into an organization's digital landscape has become a paramount, yet often elusive, goal for security and operations teams. This talk, delivered by Lu, a Principal System Engineer at The Home Depot, delves into the intricate challenges and strategic imperatives of building robust visibility programs. Drawing from his extensive experience in Security Information and Event Management (SIM), Security Orchestration, Automation, and Response (SOAR), and enterprise logging, Lu explores visibility not merely as a technical undertaking but as a fundamental pillar of effective cybersecurity and operational resilience.

The presentation addresses the critical question of whether true visibility is an attainable art or an impossible dream, especially for organizations scaling from medium-sized enterprises to large corporations with tens of thousands of employees, hundreds of applications, and hybrid cloud infrastructures. Lu emphasizes that visibility, at its core, is about data accessibility – empowering users and systems with the right data to make informed decisions. The talk meticulously unpacks the role of logs as the foundational data pillar, offering practical case studies, architectural considerations, and strategic insights for maturing a secure logging and monitoring program in today's dynamic threat landscape.

The relevance of this discussion cannot be overstated. As organizations accelerate their digital transformation journeys, particularly embracing public cloud technologies, the attack surface expands, and the volume and diversity of data proliferate. Without a clear, top-down strategy for visibility, security teams risk operating in the dark, unable to detect, respond to, or even prevent breaches effectively. Lu's presentation provides a vital roadmap for practitioners, highlighting the governance, architectural design, and operational processes necessary to transform the abstract concept of visibility into a tangible, actionable reality, ultimately aiming to maximize value while managing the inherent complexities and costs.

Background

▶ Watch: Speaker introduction and talk agenda (0:00)

The concept of visibility within IT and security has evolved significantly over time, moving beyond traditional monitoring to encompass a more holistic approach known as observability. Initially, monitoring focused on detecting known issues, using predefined metrics and alerts to confirm that systems were operating as expected. It was largely a proactive measure, geared towards identifying what an organization wanted to know or be alerted upon. This typically involved setting Service Level Objectives (SLOs) and Key Performance Indicators (KPIs) for operational aspects, or leveraging frameworks like MITRE ATT&CK for security monitoring to define detection coverage.

However, as IT environments grew in complexity, a new paradigm emerged: observability. This concept emphasizes a data-centric approach to understanding the unknowns within a system. Unlike monitoring, which asks "Is it working?", observability asks "Why is it not working?" by enabling users to explore and understand the internal states of a system based on externally generated data. It is inherently retroactive, allowing for deep dives into system behavior to pinpoint root causes and identify anomalous activities that were not explicitly predefined. Both monitoring and observability methodologies are widely adopted, serving as two complementary hands of IT management.

Operationally, both security and operational monitoring share commonalities. They both start with defining what needs to be monitored, employ similar analytics methods such as single signal alerting, correlation to reconstruct narratives, and machine learning (ML) or Artificial Intelligence (AI) for anomaly detection. Crucially, neither can function without data. The foundational data pillars for operational monitoring traditionally include metrics, traces, and logs. In the security domain, logs—specifically audit logs and activity logs—are equally indispensable, forming a common ground for both disciplines. From a response perspective, both areas strive for automation to enhance efficiency and engage in retrospection (e.g., post-mortems, incident reviews, threat hunting, digital forensics) to close monitoring gaps and refine detection models.

At its heart, visibility is fundamentally about data management. Without properly managed data, security technologies are effectively blinded. The intersection of metrics, traces, and logs is critical for efficient problem identification and response. Among these, logs stand out due to their high volume, diversity, and ubiquitous use across both operational and security contexts. They capture granular events and state changes, providing the raw material necessary to reconstruct events, identify malicious activities, and understand system behavior in depth. This makes the proper management of logs not just a technical task, but a strategic imperative for any organization aiming to achieve comprehensive visibility and bolster its cyber defenses.

Key Findings

▶ Watch: Defining IT visibility and varied perspectives (2:50)

The talk highlights several critical findings and strategic insights for achieving meaningful visibility in modern enterprise environments:

1. Visibility is Data Management: The core premise is that visibility is impossible without effective data management. All cybersecurity technologies and operational insights are predicated on the availability, quality, and accessibility of data, with logs being a primary driver.
2. Maturity Journey for Secure Logging: Building a robust secure logging and monitoring program is not a one-time project but a continuous maturity journey. It requires a structured approach across three distinct pillars: Governance (defining why and how), Architecture and Engineering (enabling the what and how), and Alerts, Incidents, and Response (realizing the program's value).
3. Cloud Misconfigurations are a Top Threat: As organizations migrate sensitive data and workloads to public clouds, cloud-specific threats like data leakage, financial theft, and denial of service become prevalent. The number one reason cited for these threats is cloud misconfigurations, such as excessive IAM permissions or unrestricted network ports.
4. "Everything as Code" for Program Evolution: To enable technology rationalization and adapt to evolving business needs and new technologies, adopting "everything as code" principles (Infrastructure as Code, Service as Code, Configuration as Code, and Detection as Code) is crucial. This approach standardizes processes, promotes reusability, and facilitates seamless migration and integration.
5. SOC Burnout and Alert Fatigue are Epidemic: A significant challenge faced by Security Operations Centers (SOCs) is alert fatigue, with a study by Trend Micro indicating that 70% of SOC teams are emotionally overwhelmed by alert volume. This negatively impacts both professional and personal lives, leading to missed true signals and delayed breach responses.
6. Addressing Alert Fatigue through "Three P's": Combating alert fatigue requires a multi-faceted approach focusing on People (training, business context), Process (layered tuning, quantitative severity formulas), and Product (strategic automation, feedback loops for ML models).
7. Balancing Data Volume and Visibility Goals: Given the high volume and cost of data, organizations must strategically balance collecting all possible data with focusing on high-risk assets and critical objectives. The goal is to maximize value from data while managing the inherent complexities and costs.
8. Challenges in Data Migration: Migrating large volumes of historical log data between legacy and modern systems presents significant technical hurdles. The concept of federated search (searching multiple data stores from a single layer) offers a potential solution but is still maturing in vendor offerings.

These findings collectively underscore that achieving meaningful visibility is a complex, ongoing endeavor requiring strategic planning, robust technical implementation, and a strong focus on human factors and operational efficiency.

Technical Deep Dive

▶ Watch: Explaining monitoring versus observability concepts (4:45)

The core of achieving visibility lies in effective log management, a process described through a simplified log life cycle. This cycle begins with log generators, which are the sources where logs originate. These can be diverse, residing in data centers, satellite sites, SaaS platforms, or cloud providers, and emanating from traditional compute, containerized environments, network appliances, databases, web applications, IoT devices, and mobile platforms. A critical point highlighted is that not all logs are enabled by default; proper configuration at the source is often required to capture the necessary data, especially for network appliances or specific application logs.

Once logs are generated, they enter the logging pipeline. The first component is the log receiver, which collects the raw log data. This data is then passed to the log processor, a crucial stage where logs are parsed, filtered, and organized. Data enrichment can also occur here, adding context such as geolocation, asset criticality, or user identity information, which is vital for later analysis. After processing, logs are indexed and sent to log storage. To optimize costs, storage is often tiered: lower-cost tiers (e.g., data lakes) are used for archiving historical data for long-term analysis, threat hunting, or compliance, while higher-cost tiers support real-time monitoring and rapid data retrieval for immediate response.

The ultimate purpose of log collection is log consumption. This involves using the stored logs to build dashboards for operational oversight, generate reports for compliance or trend analysis, execute queries for ad-hoc investigations, and, critically, generate alerts for security incidents. Machine learning can also be applied at this stage to detect anomalies. All these outputs feed into incident and alert response workflows. Finally, logs must be disposed of after their retention period, as they can contain sensitive information, and unnecessary retention poses a risk exposure.

With the advent of cloud computing, logging takes on new dimensions. Public cloud providers like AWS, Azure, and GCP organize resources hierarchically (organizational, cloud space, individual resource levels), and applications often consume multiple cloud components horizontally (load balancers, databases, Kubernetes, etc.). This necessitates collecting logs from various levels and components to gain full visibility. Cloud log types are broadly categorized into "write" logs (recording changes to the environment) and "read" logs (recording data access or retrieval). The speaker provides a mapping of these log types across the major cloud providers, reiterating that many critical logs are not enabled by default and require explicit configuration.

As an example, AWS CloudTrail events are discussed. CloudTrail tracks management plan activities (API and SDK calls) by users in an AWS environment and is enabled by default for management events. Data plane operations (e.g., S3 object-level API activity) can also be enabled explicitly for deeper visibility. CloudTrail logs are instrumental for detecting various threats, including data exfiltration, denial of service (due to encryption issues), identity-based detections (suspicious logins, operations from unusual geographies), and, critically, attempts by adversaries to disable logging to blind defenders. Log data from cloud environments can be exported via mechanisms like routing logs to an S3 bucket, which then triggers an SQS queue to notify connectors for ingestion into a central SIM or cloud SIM.

For program evolution and technology rationalization, the concept of "everything as code" is promoted. This includes:

• Infrastructure as Code (IaC): Codifying infrastructure for repeatable, standardized, and secure provisioning, often integrating security as policy.
• Service as Code: Codifying reusable service components, such as dashboards, to easily translate into new technologies.
• Configuration as Code: Modularizing and parameterizing configurations for easier deployment and migration.
• Detection as Code: This is particularly relevant for security. It involves building a centralized detection library using standardized formats, such as Sigma rules. Sigma rules provide a generic and open signature format that can be translated into various SIEM query languages, allowing organizations to maintain a technology-agnostic detection capability mapped to frameworks like MITRE or NIST. This significantly eases the transition between different SIEM products, as the core detection logic remains consistent.

However, a major technical hurdle remains: data migration. Migrating terabytes or even petabytes of historical log data from a legacy system to a new one is extremely challenging. Options include direct migration (often complex and vendor-dependent) or federated search. Federated search proposes a single search layer that can query multiple underlying data stores, including legacy systems. While promising, this concept faces significant technical considerations, such as query conversion (translating new query language to legacy), managing rate limits, handling egress and ingress charges, and ensuring acceptable latency. The speaker notes that while some vendors are exploring this space, a mature, widely available solution is not yet prevalent, often leading organizations to simply retain legacy systems until data ages out.

Demo / Proof of Concept

▶ Watch: Three pillars of observability: metrics, traces, logs (7:55)

The talk did not include a live demonstration or a detailed proof of concept. Instead, it focused on conceptual frameworks, architectural components, and strategic processes for building and maturing a secure logging and visibility program.

Defensive Implications

▶ Watch: Understanding the simplified log life cycle (8:40)

The insights from this talk offer several crucial defensive implications for organizations striving to enhance their security posture:

1. Develop a Top-Down Visibility Strategy: Defenders must move beyond ad-hoc logging. Start by defining organization-specific visibility goals from a strategic, top-down perspective. This involves understanding why visibility is needed, mapping these goals to specific operational and security areas, and then assessing the current state of visibility within each area to identify gaps. A roadmap should then be established to achieve desired visibility levels, covering technology, processes, and implementation.

1. Build a Robust Secure Logging Program:

• Governance: Establish clear policies, logging standards, and procedures. Incorporate compliance requirements (e.g., PCI, SOX) and develop processes for managing logging gaps and risks. Budget planning for both technology and trained personnel is critical.
• Architecture & Engineering: Select technologies based on specific organizational requirements, grilling vendors on their capabilities. Design scalable logging pipelines and choose efficient log agents. Implement data normalization to allow unified querying across diverse log types. Crucially, enforce "need-to-know" principles and least privilege for log access, sharing only relevant data with specific teams (e.g., IT vs. SOC). Maintain a comprehensive log data inventory to understand what data is collected, what is missing, and where gaps exist.
• Alerts, Incidents & Response: Own and continuously improve the detection library, integrating feedback from MITRE ATT&CK, threat hunting, and attack simulations.

1. Proactively Address Cloud Security Risks:

• Prevent Misconfigurations: Implement Infrastructure as Code (IaC) to codify infrastructure in a repeatable and standardized way. Integrate security as policy into IaC workflows to ensure cloud resources are provisioned securely before deployment.
• Vulnerability Scanning: Regularly scan cloud assets for vulnerabilities using available market tools to identify and remediate weaknesses.
• Real-time Cloud Threat Detection: Deploy cloud-native or agnostic cloud SIM solutions that can integrate with existing central SIM/SOAR platforms. Ensure these solutions leverage cloud-generated logs effectively for real-time threat detection and response, covering cloud exposure.
• Explicit Log Enablement: Understand that many critical cloud logs (especially data plane operations) are not enabled by default. Actively configure and enable necessary logging across all cloud providers (AWS, Azure, GCP) to gain comprehensive visibility. Be aware of adversaries attempting to disable logging to evade detection.

1. Combat SOC Burnout and Alert Fatigue: This is a critical area for defender resilience.

• People: Invest in continuous training for SOC analysts and engineering teams. Beyond security skills, equip them with business logic and architectural understanding of applications. This context helps them understand the impact of alerts and respond more efficiently.
• Process: Implement layered tuning within the alerting pipeline, from cyber sensors to the central SIM. Tune out known false positives (e.g., vulnerability scanners, irrelevant IP ranges). Develop quantitative severity formulas that incorporate factors like asset criticality, user risk scores, and threat intelligence to prioritize alerts effectively, allowing analysts to focus on high-criticality incidents.
• Product/Automation: Automate as much as possible for repetitive tasks and initial responses, but do so judiciously. Start with well-defined use cases, measure false positive ratios to ensure accuracy, and establish feedback loops to fine-tune automation and supervise machine learning models from security vendors (EDR, NDR, XDR) that generate alerts.

1. Embrace "Everything as Code" for Agility: To ensure the logging and monitoring program can adapt to technological shifts and business needs, adopt IaC, Service as Code, Configuration as Code, and especially Detection as Code (e.g., using Sigma rules). This creates a portable, maintainable, and version-controlled detection library that can be translated across different SIEM technologies, preserving detection logic through platform changes.

1. Strategic Data Management and Integration: Balance the desire for comprehensive data collection with the practicalities of cost and volume. Focus on high-risk assets and critical data sources first. Prioritize interoperating and integrating all data pillars (metrics, traces, logs) for enhanced, correlated visibility. Continuously evaluate and rationalize logging and SIM technologies to maximize value for money, aiming for a universal and centralized logging pipeline (e.g., using Kafka) to decouple log generation from consumption.

By systematically addressing these defensive implications, organizations can move closer to making comprehensive visibility an "art of the possible," transforming raw data into actionable intelligence and strengthening their overall cybersecurity posture.

Key Takeaways

• Visibility is fundamentally data management: Effective security and operations hinge on the ability to access, process, and analyze the right data, with logs serving as a critical foundation.
• Secure logging is a journey, not a destination: Building a mature logging and monitoring program requires continuous effort across governance, architecture/engineering, and incident response, adapting to evolving threats and technologies.
• Cloud misconfigurations are a primary attack vector: Organizations must proactively implement Infrastructure as Code with security policies and real-time cloud threat detection to mitigate risks from common cloud misconfigurations.
• "Everything as Code" drives future resilience: Adopting IaC, Service as Code, Configuration as Code, and especially Detection as Code (e.g., with Sigma rules) is essential for standardizing, automating, and future-proofing logging and detection capabilities.
• Combat alert fatigue with a holistic approach: Addressing SOC burnout requires focused efforts on people (training, business context), processes (layered tuning, quantitative severity), and product (strategic automation, feedback loops for ML models).
• Balance data volume with strategic focus: Given the scale and cost of data, organizations must prioritize logging from high-risk assets and critical business processes, integrating diverse data pillars for enhanced, actionable visibility.

About the Speaker(s)

Lu is a Principal System Engineer at The Home Depot, where he plays a pivotal role in architecting and designing the company's SIM (Security Information and Event Management) and SOAR (Security Orchestration, Automation, and Response) solutions. His expertise spans general enterprise logging and monitoring, a domain he has cultivated throughout his career.

Lu's professional journey began with a strong foundation in Splunk, working within a global finance institute for several years. He then transitioned to a Splunk Professional Services consultant role at a small consulting company in Texas, where he provided services to local government and higher education institutions. His extensive background in these diverse environments has equipped him with deep practical knowledge in building and managing large-scale logging infrastructures and security operations programs.