CloudTail: Making Heads or Tails of Selectively Retaining Multi-Cloud Logs (w/o a SIEM!)

CloudTail: Making Heads or Tails of Selectively Retaining Multi-Cloud Logs (w/o a SIEM!)

Speakers: Ela Dogjani, Associate Threat Researcher, Permiso Security

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=zbm20JDznKI

Overview

In the increasingly complex landscape of cloud computing, effective log management stands as a cornerstone of robust security and operational resilience. However, the proliferation of services and the adoption of multi-cloud strategies have transformed this essential task into a significant challenge, often characterized by overwhelming data volumes, prohibitive costs, and fragmented visibility. This talk, delivered by Ela Dogjani, an Associate Threat Researcher at Permiso Security, introduces CloudTail, an innovative open-source tool designed to address these pervasive issues by enabling selective, long-term retention of high-value multi-cloud logs without the necessity of a traditional, expensive Security Information and Event Management (SIEM) solution.

Dogjani's presentation meticulously dissects the inherent limitations of current cloud log management paradigms, from restrictive API access and short default retention periods to the complexities introduced by native log forwarding and the sheer scale of data. CloudTail emerges as a pragmatic solution, offering a unified approach to managing logs from major cloud providers like AWS and Azure. By focusing on the intelligent filtering and local storage of only the most critical security and operational events, CloudTail empowers organizations to maintain comprehensive visibility, enhance incident response capabilities, and meet compliance requirements, all while drastically reducing the financial burden typically associated with extensive log ingestion and storage.

The relevance of CloudTail extends to any organization grappling with the intricacies of securing and monitoring cloud environments, particularly those operating across multiple cloud platforms. It provides a much-needed alternative for smaller teams or budget-conscious enterprises that require sophisticated log management capabilities but cannot justify the significant investment in a full-fledged SIEM. Through its emphasis on selective retention and an intuitive configuration, CloudTail promises to simplify a critical security function, transforming what is often a headache-inducing task into a streamlined, cost-effective, and highly actionable process for making heads or tails of vital cloud events.

Background

▶ Watch: Introduction to multi-cloud log challenges and CloudTail (0:00)

The foundational importance of logs in maintaining the health and security of any computing system cannot be overstated. As Dogjani aptly describes, logs serve as the "diary" of a system, recording every action, event, and alert. From an operational perspective, they are indispensable for tracking performance, troubleshooting issues, and proactively identifying inefficiencies, such as a system nearing its resource limits. On the security front, logs are often the "first and sometimes only source of truth" when incidents occur, revealing critical details like "who did what, when, and how" – whether it's a suspicious login, an unauthorized configuration change, or a full-blown attack.

However, the advent of cloud computing, particularly multi-cloud environments, has introduced a new layer of complexity to log management. Organizations are no longer dealing with a single, homogenous system but rather an intricate web of services across platforms like AWS, Azure, Google Cloud, and SaaS providers such as Okta. Each platform generates vast volumes of logs with distinct structures and access mechanisms, making unified management a daunting task.

Dogjani highlights several critical pain points associated with existing log management approaches:

1. Direct API Access Limitations:

• Low Retention Periods: Default retention for logs can be alarmingly short. Azure Activity Logs, for instance, are retained for only 90 days by default, with extensions up to two years requiring manual configuration to Azure Storage or Log Analytics Workspace. Similarly, AWS CloudTrail's event history only lasts 90 days, though management and data events can be stored indefinitely in S3 buckets with correct lifecycle policies, which demands continuous management. This short window is insufficient for many compliance requirements or long-term forensic investigations.
• API Throttling: Cloud providers implement throttling mechanisms to prevent system overload. AWS CloudTrail's Lookup Events API, for example, permits only two requests per second. Such limitations can cause significant delays in data retrieval, create gaps in log collection, and potentially lead to missing critical events if the volume of requests outstrips the API's capacity within a specific time window.
• Limited Search Capabilities: Native search functionalities often lack the sophistication required for effective incident response. Azure Activity Logs, for example, do not allow filtering by specific operation names, forcing users to sift through irrelevant data to find events like "create user" or "add user." AWS also struggles with advanced search, lacking support for fuzzy searches or multi-attribute filtering, making it difficult to locate logs when only partial details (e.g., a fragment of a username or IP address) are known. This can lead to critical logs being overlooked.

1. Native Log Forwarding Limitations:

• Filtering Limitations: Even when logs are forwarded, native filtering tools often fall short. AWS Advanced Event Selectors in CloudTrail, while allowing filters based on resource type, event source, or read/write operations, crucially cannot filter by the event name property for management events. This forces organizations to capture all management events, leading to an overwhelming influx of "noise" that drowns out high-value actions like IAM user creation or deletion, making targeted investigations exceedingly difficult.
• Massive Log Volumes and Performance: Forwarding all logs to a centralized storage solution or a SIEM, while offering consolidation, introduces the challenge of managing sheer data volume. Real-time searches, even with powerful tools like Splunk, can slow down considerably or fail entirely when confronted with massive datasets. This performance degradation becomes a significant issue during security incidents where rapid log analysis is paramount.

1. Multi-Cloud Complexity:

• Cost: Storing and searching through vast quantities of logs across multiple platforms for extended periods can quickly become prohibitively expensive. Most organizations cannot afford to retain every log from AWS, Azure, Google Cloud, or Okta, especially when millions of events are generated daily, many of which hold little practical value for day-to-day operations or security.
• Disparate Structures: Each cloud platform logs events differently, both in structure and content. This necessitates tailored approaches for filtering, parsing, and storing logs from each provider, adding significant overhead and complexity to a unified log management strategy.

The cumulative effect of these challenges is a fragmented, costly, and often ineffective log management posture, leaving organizations with blind spots and hindering their ability to detect, investigate, and respond to security threats efficiently. The core problem is not merely collecting logs, but intelligently managing and leveraging a curated set of high-value events that truly matter for audits, compliance, and security investigations, without overwhelming budgets and systems.

Key Findings

▶ Watch: The overwhelming challenge of multi-cloud log management (2:30)

The central finding presented by Ela Dogjani is that the current state of multi-cloud log management, whether relying on direct API access or native forwarding mechanisms, is fundamentally flawed. It is characterized by prohibitive costs, overwhelming data volumes, fragmented visibility, and significant operational complexities stemming from short retention periods, API throttling, and limited search and filtering capabilities across disparate cloud platforms. Organizations are often forced to choose between ingesting an unmanageable deluge of data or risking critical blind spots due to insufficient log capture.

Dogjani's research and development demonstrate that a more strategic, cost-effective, and efficient approach is not only possible but necessary. The key finding is that by focusing on the selective retention of high-value events, organizations can dramatically improve their security posture and operational efficiency without incurring the exorbitant costs or operational overhead associated with traditional, catch-all log management solutions like enterprise SIEMs. This focused approach allows for long-term storage and rapid access to the most relevant security and operational insights, transforming the log management paradigm from one of reactive data hoarding to proactive intelligence gathering.

This critical gap in existing solutions led to the development of CloudTail. The tool's primary contribution is providing an open-source, unified, and cost-effective method to address the identified challenges. CloudTail allows organizations to "tail" their cloud logs across multiple platforms (specifically AWS and Azure) by intelligently filtering, normalizing, and storing only the events deemed most crucial for security operations, compliance, and incident response. This selective methodology directly counters the inefficiencies of traditional methods, offering a practical pathway to comprehensive log visibility without the financial burden or complexity of ingesting every single log.

Technical Deep Dive

▶ Watch: Understanding AWS CloudTrail and Azure Activity Logs (3:00)

CloudTail is an open-source tool designed to streamline multi-cloud log management by focusing on the selective retention and efficient searching of high-value security events. Its name is a deliberate nod to AWS CloudTrail, aiming to signify its role in "tailing" critical logs across various cloud environments. The tool is engineered to provide a cost-effective alternative to full-scale SIEM solutions for foundational log management, supporting both AWS and Azure platforms.

At its core, CloudTail's architecture prioritizes intelligent data ingestion and localized storage. Instead of collecting every log, which can be both costly and overwhelming, CloudTail leverages a flexible JSON-based configuration to define precisely which events are considered "high-value." This configuration file is the brain of the tool, allowing users to specify filtering rules with granular control.

Key technical features and mechanisms of CloudTail include:

1. Multi-Cloud Support:

• AWS: Integrates with AWS CloudTrail to capture API calls and resource changes. It pulls data based on specified account IDs and profile names, allowing for multi-account management within a single configuration.
• Azure: Processes Azure Activity Logs by requiring subscription IDs. It focuses on resource change tracking, service health monitoring, and security event logging.
• The tool unifies the management experience for logs from these two dominant cloud providers, eliminating the need to navigate disparate interfaces.

1. Selective Event Retention and Advanced Filtering:

• JSON Configuration: The filtering logic is entirely driven by a simple yet powerful JSON configuration file. This file allows users to define "lookup attributes" for each event, grouped under a rule name (e.g., "IAM user management").
• Attribute Key and Value: Users specify the attribute key (e.g., eventName, userAgent, sourceIpAddress) and attribute value they are interested in.
• Wildcard Matching: CloudTail supports wildcard characters for broad event categories. For instance, a rule like delete* for the eventName attribute will capture all events where the name starts with "delete," such as DeleteUser, DeleteRole, or DeleteBucket. This is crucial for tracking entire classes of potentially high-impact activities.
• JMESPath Filtering (AWS Only): For highly advanced and precise filtering, CloudTail supports JMESPath expressions specifically for AWS logs. This allows for complex queries that can filter logs based on nested attributes or conditions. An example provided is capturing CreateUser events only when the user's role is "admin," demonstrating the ability to target very specific scenarios. Notably, JMESPath filtering is not currently supported for Azure logs.
• Default High-Value Events: CloudTail comes with a pre-configured list of high-value events identified by Permiso Security's threat research team. These defaults cover critical areas like user management, resource changes, and security alerts, providing a strong baseline that users can easily customize.

1. Local Storage and Normalization:

• Instead of forwarding logs to an external, often expensive, SIEM or data lake, CloudTail stores logs locally in their raw format.
• Crucially, it also normalizes and extracts key attributes from these raw logs. This dual approach ensures that the full fidelity of the original event data is preserved for deep investigation, while extracted attributes facilitate efficient searching and querying. This minimizes storage costs without sacrificing forensic detail.

1. Scheduled Execution and State Management:

• CloudTail is designed to run on a scheduled basis, ensuring continuous log collection and processing.
• It intelligently tracks its last successful execution time, allowing it to pick up precisely where it left off on subsequent runs. This mechanism prevents duplicate event processing and ensures comprehensive coverage.
• The tool can process up to 30 days of logs in a single run, but it is optimized for frequent scheduling (e.g., daily or hourly) to maintain near real-time visibility and manage processing loads.

1. Internal Database Structure:

CloudTail utilizes a local database (likely SQLite, though not explicitly stated, common for local tools) to organize and store processed event data and operational metadata. The key tables include:

• CloudTrail events: Stores metadata and the full raw event details for AWS logs, with columns like event ID, account ID, profile name, event name, event time, and event data.
• Azure events: Analogous to the CloudTrail events table, but for Azure logs.
• Execution history: Tracks when specific rules were executed and which events were processed, facilitating auditing and state management.
• Lookup attributes and Event lookup attributes: Internal tables used to store lookup data and attribute mappings, optimizing how CloudTail queries and processes events.
• Rule matches: Records information about events that successfully matched specific rules defined in the configuration file, providing a direct link between an event and the policy that captured it.

This robust technical design allows CloudTail to operate as a self-contained, intelligent log management solution, empowering organizations to build their own extended retention repository for high-value security events.

Demo / Proof of Concept

▶ Watch: Pain points: limited retention and direct API access issues (4:30)

The demonstration of CloudTail provided a clear walkthrough of its core functionalities, focusing on the configuration of event filtering and the execution of the tool. The speaker highlighted the intuitive, JSON-based configuration file as the primary interface for defining which logs CloudTail should collect and retain.

Configuration File Walkthrough:

1. AWS Configuration:

• Under Account Profile Pairs, users specify account IDs and profile names. If no specific ID or profile is provided, CloudTail defaults to the system's configured AWS profile.
• Events are defined under lookup attributes. Each event filter is assigned a rule name, such as "IAM user management," to group similar actions.
• For each rule, attribute key and attribute value pairs are used to pinpoint desired events. While eventName is a common attribute key, other fields like userAgent or sourceIpAddress can also be used.
• Wildcard Matching: A key feature demonstrated was the use of wildcards. For example, setting the eventName to delete* would capture all events starting with "delete," such as DeleteUser, DeleteRole, or DeleteBucket. This allows for broad but targeted collection of critical activities.
• JMESPath Filtering: For more advanced scenarios, CloudTail supports JMESPath expressions. An example was given for capturing CreateUser events specifically where the user's role is "admin." This capability allows for highly granular filtering based on nested or complex conditions within the log data. For JMESPath filtering, the attribute key and attribute value fields are optional, as the JMESPath expression itself defines the criteria.

1. Azure Configuration:

• Similar to AWS, Azure configuration requires subscription IDs, which are mandatory unlike AWS account IDs.
• Users define specific operations they are interested in.
• Azure configuration also supports wildcard matching for operations.
• A notable difference is that JMESPath filtering is not currently supported for Azure logs.

Tool Execution and Output:

After configuring the JSON file, the demo proceeded to show how to run CloudTail and interpret its output.

1. Help Section and Basic Usage:

• The help section of the tool was shown, listing available command-line flags.
• The only required argument to run CloudTail is specifying the --config file.

1. Key Command-Line Flags:

• --export: Exports all processed events to a JSON file.
• --output-directory: Specifies the directory where exported JSON files should be saved. If omitted, files default to the project folder.
• --export-time-range: Allows specifying a start date and end date to export events from a particular execution timeframe.
• --database-directory: Defines where the internal database files will be stored. If not specified, the database defaults to the project directory.

1. Running with Default Configuration:

• Executing CloudTail with the default configuration file generates console output that provides real-time feedback on the processing.
• For each rule defined, the output displays:
• The lookup attribute being queried.
• Details of the query (e.g., eventName: delete*).
• The profile name associated with the rule.
• The time range of logs being queried.
• The number of events found for that particular rule.
• The database table where the results were written.

1. Exporting Events to JSON:

• The --export flag was demonstrated, without specifying an output-directory.
• The tool confirmed the creation of the output directory (if it didn't exist) and printed messages indicating the total number of events saved successfully and the name of the file where they were stored.
• Exported files are automatically named by the source name (e.g., AWS CloudTrail) and the date of the query, ensuring easy organization.
• A message also appeared for Azure logs, indicating "no events were found for Azure logs in the specified time range," demonstrating the tool's ability to report on each configured source.

1. Database Structure in Practice:

• The demo concluded by illustrating the local database structure created by CloudTail. The results are stored in distinct tables for organization:
• CloudTrail events table: Contains metadata and full raw event data for AWS logs (event ID, account ID, profile name, event name, event time, event data).
• Azure events table: Stores similar information for Azure logs.
• Execution history table: Tracks the tool's operational history, including rule execution times and processed events.
• Lookup attributes and Event lookup attributes tables: Used internally to map attributes to event data fields for efficient querying.
• Rule matches table: Identifies which events triggered specific rules defined in the configuration, providing direct traceability.

The demo effectively showcased CloudTail's ease of use, powerful filtering capabilities, and structured approach to local log storage, underscoring its potential as a practical, cost-effective solution for multi-cloud log management.

Defensive Implications

▶ Watch: Understanding the impact of API throttling on log retrieval (6:30)

CloudTail offers significant defensive implications for organizations navigating the complexities of multi-cloud security. By addressing the fundamental challenges of log management—cost, volume, and fragmented visibility—it empowers defenders with more effective tools and strategies.

1. Enhanced Visibility and Reduced Blind Spots: The primary defensive benefit is the ability to achieve comprehensive visibility into critical security events across AWS and Azure environments. By selectively retaining high-value logs, security teams can focus on truly actionable data related to user activity, resource changes, and potential threats, rather than being overwhelmed by irrelevant noise. This targeted approach ensures that critical indicators of compromise (IOCs) or anomalous behaviors are less likely to be missed, reducing blind spots that attackers often exploit.

1. Cost-Effective Long-Term Retention: Traditional SIEMs and extensive cloud storage for all logs can be prohibitively expensive. CloudTail's model of selective retention and local storage dramatically reduces costs, making it feasible for organizations to maintain logs for extended periods, far beyond the default 90 days offered by cloud providers. This extended retention is crucial for compliance with various regulatory frameworks (e.g., HIPAA, GDPR, PCI DSS) and for conducting thorough forensic investigations that may span months or even years after an incident.

1. Proactive Incident Response and Faster Investigations: With high-value logs readily available and efficiently searchable in a normalized format, security analysts can significantly shorten incident response times. Instead of battling API throttling, limited search features, or sifting through massive datasets, CloudTail provides a curated, accessible repository of critical events. This direct access to relevant data allows for quicker identification of "who did what, when, and how," facilitating rapid containment, eradication, and recovery efforts during a security incident.

1. Customizable Threat Detection and Hunting: The flexible JSON configuration, including wildcard matching and JMESPath filtering, allows defenders to tailor log collection precisely to their organization's specific threat model and risk profile. Security teams can define rules to proactively capture events associated with known attack techniques (e.g., specific API calls used in privilege escalation, data exfiltration attempts) or suspicious patterns unique to their environment. This customizability supports more effective threat hunting activities, enabling defenders to actively search for subtle indicators of compromise that might otherwise go unnoticed.

1. Reduced Reliance on Expensive SIEMs: For many organizations, particularly small to medium-sized businesses or those with budget constraints, the cost of a full-scale SIEM can be a barrier to entry for robust log management. CloudTail provides a powerful, open-source alternative for the foundational task of collecting, filtering, and storing critical cloud logs. While it may not offer the advanced correlation and alerting capabilities of a SIEM, it handles the crucial data ingestion and retention component efficiently, allowing organizations to allocate resources to other security controls or consider lighter-weight analytics solutions on top of CloudTail's output.

1. Simplified Compliance Auditing: By ensuring that relevant logs are retained for required durations and are easily searchable, CloudTail simplifies the process of demonstrating compliance during audits. The ability to quickly pull specific event types or activity logs provides undeniable evidence of security controls and operational integrity, reducing the burden and complexity often associated with compliance reporting.

In essence, CloudTail transforms log management from a reactive, resource-intensive chore into a proactive, intelligent, and cost-efficient defensive advantage. It empowers security teams to gain meaningful insights from their cloud environments, bolstering their ability to prevent, detect, and respond to cyber threats effectively.

Key Takeaways

• Multi-cloud log management is inherently complex, yet crucial for security and operations. Organizations face challenges including short default retention, API throttling, limited search capabilities, and overwhelming data volumes across disparate cloud platforms.
• CloudTail simplifies this complexity by offering selective, long-term retention of high-value events. It acts as an open-source, cost-effective alternative to traditional SIEMs for managing logs from AWS and Azure.
• The tool leverages a flexible JSON configuration for granular filtering. This includes wildcard matching and advanced JMESPath expressions (for AWS) to ensure only relevant security and operational events are captured, minimizing storage costs and noise.
• CloudTail empowers organizations to build their own extended retention repository locally. Logs are stored in their raw format, with key attributes extracted, enabling both forensic depth and efficient searching without relying on external, expensive solutions.
• Scheduled execution ensures continuous, non-duplicative log collection. CloudTail intelligently picks up from its last successful run, processing logs efficiently and maintaining an up-to-date repository of critical events.
• Defenders gain enhanced visibility, faster incident response, and improved compliance posture. By focusing on actionable data, security teams can more effectively detect threats, conduct investigations, and meet regulatory requirements without being overwhelmed by data volume or budget constraints.

About the Speaker(s)

Ela Dogjani is an Associate Threat Researcher at Permiso Security. She recently graduated as a Software Engineer and delivered this talk as her very first presentation at a security conference, marking an exciting milestone in her career. Dogjani is deeply passionate about technology, with a particular focus on cloud security, and has dedicated the past few years to expanding her knowledge in this domain. Outside of her professional interests, she enjoys reading books, cooking, and has a unique hobby of collecting perfumes.