REPLICAWATCHER: Training-less Anomaly Detection in Containerized Microservices

REPLICAWATCHER: Training-less Anomaly Detection in Containerized Microservices

Speakers: Asbat El Khairi

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=K-gKSfTl324

Overview

In the rapidly evolving landscape of containerized microservices, traditional anomaly detection systems face a formidable challenge: the relentless shift in what constitutes "normal" behavior. This talk introduces REPLICAWATCHER, a novel, training-less anomaly-based intrusion detection system designed specifically to address this issue in dynamic microservices environments. Presented by Asbat El Khairi, REPLICAWATCHER proposes a paradigm shift from reliance on static, pre-established baselines to a real-time comparative analysis of identical container instances, or replicas.

The core innovation of REPLICAWATCHER lies in its ability to detect anomalies without any prior training or subsequent retraining, a process that is both resource-intensive and impractical in environments where operational conditions constantly change. By observing the behavioral consistency among replicas, which are adopted for fault tolerance and scalability and execute analogous tasks, the system can identify deviations in a single replica as a strong indicator of a security threat. This "shield-right" approach provides continuous monitoring against zero-day attacks and runtime compromises that bypass "shift-left" prevention efforts.

REPLICAWATCHER's contributions are significant: it introduces the concept of training-less anomaly detection for microservices, presents a concrete system implementation, and demonstrates its resilience against normality shifts. The system achieved an average precision of 91.08% and recall of 98.35% across 13 attack scenarios, while maintaining negligible runtime overhead. This talk highlights a crucial advancement in securing highly dynamic, cloud-native architectures against previously unseen threats.

Background

▶ Watch: Why training-less anomaly detection is crucial (0:00)

The pervasive adoption of microservices architectures has brought immense benefits in terms of agility, scalability, and resilience. However, this dynamism introduces a critical security challenge for traditional anomaly detection systems. These systems typically rely on establishing a comprehensive baseline of normal behavior during an initial training phase. Any deviation from this baseline is then flagged as a potential anomaly. The fundamental problem, as highlighted in the talk, is that these baselines age rapidly and lose effectiveness in microservices environments.

Normality in microservices is a constantly shifting target. Changes can stem from various sources: updates to underlying base OS images, new versions of dependency packages, or even minor modifications to application code. The speaker provided a concrete example from an empirical study using the Google Online Boutique (GOB) e-commerce platform. Monitoring the cart microservice across ten versions over nine months revealed that almost every new version introduced at least one previously unseen syscall. For instance, an update from cart:0.6.0 to cart:0.7.0 involved a .NET SDK update, which, in turn, introduced the execution of the getrlimit syscall. While getrlimit is a perfectly normal operation, its absence in a prior baseline would trigger a false positive in a traditional anomaly detection system. This constant "normality drift" renders training-based approaches impractical, necessitating frequent and costly retraining.

This challenge underscores the need for a "shield-right" security approach, focusing on continuous runtime monitoring to detect and alert security teams to suspicious activities that might bypass initial "shift-left" prevention strategies. REPLICAWATCHER addresses this by proposing a novel, training-less methodology. The core insight is that identical container instances, or replicas, deployed for fault tolerance or scalability, perform similar tasks and should exhibit highly similar behavioral patterns. Any significant deviation by one replica from its counterparts can thus serve as a direct indicator of an anomaly without requiring a predefined normal baseline.

The preliminary assessment, detailed in Section III of the talk, guided the selection of features for REPLICAWATCHER. The speaker emphasized two key considerations: firstly, selecting features that broadly cover container behavior and can generically detect attacks; and secondly, identifying features that exhibit minimal noise among replicas under normal conditions to prevent false alarms. The study deployed each GOB microservice with four replicas and used Sysdig with a customized Chisel script to collect kernel events over various intervals (5s, 10s, 30s, 60s), accumulating 44,000 logs over 13 days. Feature exploration focused on three fundamental OS kernel elements: Syscalls (frequency, names, categories, latency), File Descriptors (FD) (accessed files/directories, buffer size, client IPs/ports), and Processes (executed processes, commands/arguments, executables, working directories). This meticulous feature engineering was crucial for building a system that could differentiate subtle anomalies from inherent operational noise.

Key Findings

▶ Watch: Demonstrating 'normality drift' with a real-world example (2:00)

REPLICAWATCHER's evaluation unveiled several critical findings that underscore its effectiveness and potential to revolutionize anomaly detection in microservices:

1. Exceptional Resilience to Normality Drifts: This is REPLICAWATCHER's most significant contribution. Unlike state-of-the-art training-based container HIDSes such as STIDE-BoSC, CHIDS, and CDL, REPLICAWATCHER demonstrated remarkable stability during system updates. When the Cart microservice's base OS image was updated (e.g., php:8.1.18-apache-buster to php:8.1.18-apache-bullseye introducing clock_nanosleep), or the Checkout service's Puma package was upgraded (from version 3.2 to 5.5, introducing epoll_wait, epoll_ctl, ppoll), or the Signup service's application code was updated (adding Google Geocoding API integration, introducing bind and recvmsg), traditional HIDSes experienced a significant increase in false positives. REPLICAWATCHER, by contrast, maintained a nearly steady false positive rate and high detection rate because updates are propagated to all replicas, preserving their behavioral consistency.

1. High Detection Capabilities Comparable to State-of-the-Art: REPLICAWATCHER achieved impressive detection metrics across a diverse range of attack scenarios. It yielded high AUC scores: 0.9771 for information disclosure, 0.9883 for remote privilege escalation, and 0.9970 for code execution. The average AUC for GOB scenarios was 0.9960, and for custom scenarios, 0.9827. With an empirically selected detection threshold of e = 0.3, the system achieved an average precision of 0.9248 and recall of 0.9813. While precision was slightly lower for certain attacks (e.g., PHP-LFI, CWE-434, CVE-2014-6271) and recall for subtle path traversals (CVE-2018-3760), the high AUC scores indicate that threshold tuning could further optimize performance for specific use cases. Compared to existing HIDSes, REPLICAWATCHER outperformed CDL (which had low recall due to being frequency-based) and STIDE-BoSC (which suffered from a 60% false positive rate), performing comparably to CHIDS which achieved over 96% recall and 97% precision.

1. Negligible Runtime Overhead and Scalability: The system demonstrated excellent runtime performance, growing sublinearly with an increasing number of replicas. Even with six replicas for each of eight microservices (a total of 48 pods), REPLICAWATCHER processed and classified data in under 2.25 seconds. This efficiency, attributed to Chisel's event filtering and a carefully selected feature set, makes REPLICAWATCHER highly feasible for practical deployment in production environments.

1. Crucial Role of Feature Categories: An ablation study highlighted the importance of each feature category:

• FD-based features were critical for detecting Information Disclosure (AUC dropped to 0.5735) and Remote Privilege Escalation attacks.
• Process-based features were essential for Command Injection detection, emphasizing the value of monitoring commands, executables, and arguments.
• Syscall-based features provided a performance boost for Remote Code Execution (RCE) and Remote Privilege Escalation (RPE) scenarios, particularly important for memory corruption attacks.

1. Bounded Robustness Against Evasion: While stealthy attacks, like executing a single command per snapshot during reconnaissance in a Kill Chain scenario (CWE-78 Command Injection), could temporarily evade detection, the system's detection rates rose significantly with increased command volume. Similarly, singular path traversal requests (CVE-2017-14849) caused subtle deviations that were missed, but detection improved with higher request volumes. This indicates that while evasion is possible for low-intensity activities, an attacker's ability to perform extensive malicious actions without being detected is bounded.

1. Consistent Performance Across Replica Counts: REPLICAWATCHER maintained reliable detection performance across varying replica counts, from 2 to 6. Interestingly, the False Positive Rate (FPR) was observed to be lower with two replicas, potentially due to more evenly distributed user behavior, with only moderate fluctuations as the replica count increased.

Technical Deep Dive

▶ Watch: Key criteria for selecting robust monitoring features (3:00)

REPLICAWATCHER operates under a specific threat model within a Kubernetes cluster. It assumes multiple deployments, each running a heterogeneous set of identical replicas of a microservice. These replicas are expected to execute specific tasks, adhere to microservice design principles, and run the same code version, with updates rolled out simultaneously via Deployment Controllers. Crucially, all replicas are assumed to consistently receive incoming requests, representing active user interactions. The focus is on remote network-triggered attacks where an adversary exploits a vulnerability in a public-facing microservice, delivering malicious payloads via TCP/UDP. Specifically, the system targets software-level, remote control flow hijacking attacks aimed at accessing sensitive data (e.g., authentication bypass, directory traversal), compromising filesystem integrity (e.g., file inclusion), or executing unauthorized code/commands (e.g., command injection). The fundamental assumption is that such actions deviate the behavior of the targeted replica(s) compared to their counterparts, and that attackers cannot initiate an identical attack on all replicas simultaneously without leaving detectable traces for orthogonal approaches like network traffic monitoring.

Out of scope for this work are attacks that bypass or remain inconspicuous to the syscall interface, side-channels, hardware-related attacks (e.g., Spectre), resource exhaustion, network flooding, remote passive fingerprinting, spoofing attacks, and attacks requiring physical or internal access to the Kubernetes cluster. REPLICAWATCHER is implemented as an anomaly-based IDS, leveraging Sysdig as a privileged daemonset on worker nodes, with Sysdig tampering considered outside the scope of this work.

The system's overall approach is structured into three main phases:

1. Event Chunking (Section V.A):

• The first step involves processing the continuous stream of kernel events from all running replicas. Sysdig, configured with a customized Chisel script, captures granular kernel events related to filesystem, process, and syscall activities.
• This continuous flow is then segmented into sequential "snapshots," each encapsulating the activities of replicas over a designated time interval, T_snapshot.
• Determining the optimal T_snapshot involves a trade-off: a shorter interval provides greater attack lead time but increases the risk of false alarms due to minor inconsistencies, while a longer interval reduces false alarms but sacrifices lead time. Based on the preliminary assessment (Table I in the talk), T_snapshot was experimentally set to 30 seconds, providing a balance between timely detection and robust behavior understanding.

1. Event Encoding (Section V.B):

• Within each 30-second snapshot, key features are extracted from the kernel events of each replica. These features, identified during the preliminary assessment (highlighted in bold in Table I), include aspects of syscalls, file descriptors, and processes.
• The primary goal is to compare the behavioral patterns of replicas to identify those exhibiting considerable dissimilarity. To achieve this, each replica is represented by a vector of dissimilarity scores, where each score corresponds to a distinct feature.
• For features that generate lists (e.g., distinct accessed directories), the Jaccard similarity technique is leveraged. The dissimilarity score DS_fk(R_i) for a given replica R_i and feature f_k is calculated as 1 - (1 / (n-1)) * Σ(j!=i) (Jaccard_Similarity(R_ifk, R_jfk)). A higher dissimilarity score indicates greater uniqueness in R_i's feature elements compared to its counterparts, implying a higher likelihood of it being anomalous.
• For features that generate individual values (e.g., max latency), the standard deviation is used to measure dissimilarity.
• By applying this process to each selected feature, a dissimilarity score vector v_i = [DS_f1(R_i), DS_f2(R_i), ..., DS_fk(R_i)] is generated for every replica, concisely representing its degree of alignment with its counterparts.

1. Anomaly Detection (Section V.C):

• The fundamental premise of the detection framework is that normally operating replicas exhibit only minor behavioral differences, causing their dissimilarity vectors to converge towards the origin of the vector space, represented by (0, 0, ..., 0). This origin serves as the reliable "normality region."
• Replicas whose vectors reside near the origin suggest behavioral alignment, whereas those deviating significantly indicate potential misalignment and, thus, abnormal behavior.
• To quantify this deviation, the Euclidean distance between each replica's dissimilarity vector and the origin is calculated. This distance is crucial for controlling the trade-off between detection and false alarm rates.
• The system evaluates a vector's distance against multiple threshold values, ranging from 0.1 to 1 with 0.1 increments. If the distance exceeds the employed threshold, the snapshot is labeled as anomalous. A lower threshold can improve the detection rate but may also increase false alarms. For the evaluation, an empirical threshold of e = 0.3 was selected.

Implementation Details (Section VI):

REPLICAWATCHER was implemented in Python. For evaluation, a custom e-commerce application with seven microservices (using different technologies and intentionally embedded vulnerabilities) was developed, alongside the Google Online Boutique (GOB) platform with three injected vulnerabilities. Table II in the talk outlines 13 attack scenarios across both platforms. The experimental setup involved deploying these applications with replicas on a two-node GKE cluster, with each node being an e2-standard-4 machine (4vCPU, 16GB memory) running Ubuntu and containerd. Both normal and attack modes were simulated: 30 to 50 users engaged normally, while a distinct user acted as an attacker exploiting vulnerabilities. Data collection relied on Sysdig and the custom Chisel script. For each vulnerable microservice, 1,000 normal and 250 attack snapshots were collected, split evenly across five settings with varying replica counts (2 to 6). Each snapshot captured 30 seconds of replica behavior, resulting in a dataset of 15,000 normal and 3,750 attack snapshots, equating to six days of monitoring.

Demo / Proof of Concept

▶ Watch: Final feature and monitoring interval selection (5:30)

While the talk did not feature a live demonstration in the traditional sense, the comprehensive evaluation framework described serves as a robust proof of concept for REPLICAWATCHER's capabilities. The speaker detailed an extensive experimental setup designed to rigorously test the system's performance, resilience, and scalability in realistic microservices environments.

The proof of concept involved two primary application platforms:

1. A custom e-commerce application: Developed with seven distinct microservices, each leveraging different underlying technologies (e.g., PHP, Python, Java) and intentionally embedded with a variety of vulnerabilities.
2. Google Online Boutique (GOB): A well-known microservices-based e-commerce platform, further instrumented with three injected vulnerabilities for controlled testing.

Across these platforms, a total of 13 attack scenarios were curated (as summarized in Table II of the presentation). These scenarios encompassed a broad spectrum of common web and container-related attacks, including information disclosure, remote privilege escalation, code execution, command injection (e.g., CVE-2012-1823), path traversal (e.g., CVE-2017-14849, CVE-2018-3760), and file inclusion (e.g., PHP-LFI, CWE-434).

The experimental setup was deployed on a two-node Google Kubernetes Engine (GKE) cluster. Each node was configured as an e2-standard-4 machine with 4 vCPUs and 16GB of memory, running Ubuntu and containerd. This environment provided a production-like setting for microservice deployment and orchestration.

Data collection was meticulously performed using Sysdig combined with a custom Chisel script. This allowed for granular capture of kernel events (syscalls, file accesses, process executions) from within the containers. Both normal and attack modes were simulated:

• In normal mode, 30 to 50 simulated users continuously interacted with the applications, generating typical microservice traffic and behavior.
• In attack mode, a distinct simulated user acted as an adversary, systematically exploiting the injected vulnerabilities to compromise replicas with malicious payloads.

For each vulnerable microservice, a substantial dataset was collected: 1,000 normal snapshots and 250 attack snapshots. These were distributed evenly across five different settings, varying the number of replicas from 2 to 6. Each snapshot captured 30 seconds of container behavior, aligning with the optimal T_snapshot identified during preliminary assessments. In total, the dataset comprised 15,000 normal and 3,750 attack snapshots, representing approximately six days of continuous monitoring data. This extensive dataset and rigorous experimental methodology provided the empirical evidence to validate REPLICAWATCHER's resilience against normality shifts, its detection capabilities, its robustness against evasion attempts, and its practical runtime performance.

Defensive Implications

▶ Watch: Understanding the threat model and targeted attacks (6:00)

REPLICAWATCHER presents several critical defensive implications for organizations deploying and securing containerized microservices:

1. Embrace Training-less Anomaly Detection: The most significant implication is the validation of training-less anomaly detection as a viable and superior approach for dynamic microservice environments. Security teams should consider adopting solutions that leverage replica consistency rather than relying on static baselines. This eliminates the immense burden of frequent retraining, which often renders traditional HIDS impractical in continuously evolving cloud-native setups.

1. Leverage Replica Redundancy for Security: The inherent redundancy and behavioral consistency among microservice replicas, typically used for fault tolerance and scalability, can be powerfully leveraged for security monitoring. Defenders should actively seek or develop systems that exploit this property, turning a core architectural pattern into a security advantage.

1. Strengthen "Shield-Right" Defenses: REPLICAWATCHER reinforces the necessity of a robust "shield-right" security posture. While "shift-left" practices (like secure coding, vulnerability scanning, and container image hardening) are crucial for prevention, runtime anomaly detection is indispensable for catching zero-day exploits, novel attack techniques, and sophisticated compromises that bypass initial defenses. REPLICAWATCHER serves as an effective last line of defense at the host level.

1. Multi-Layered Security is Paramount: While REPLICAWATCHER is highly effective, it is not a silver bullet. The speaker explicitly acknowledges limitations, such as brute-force attacks being obscured by load balancers or attacks simultaneously affecting all replicas. Therefore, REPLICAWATCHER should be integrated into a comprehensive, multi-layered security strategy. This includes:

• Container Isolation Mechanisms: Tools like Seccomp, SELinux, and AppArmor to restrict syscalls and capabilities.
• Network Monitoring: To detect network-level anomalies, especially for attacks that might target all replicas simultaneously or involve large-scale exfiltration.
• Resource Monitoring: To identify resource exhaustion attacks.
• Shift-Left Practices: Continuing to focus on secure development lifecycle, vulnerability management, and secure configuration of Kubernetes and container images.

1. Understand Feature Importance for Comprehensive Monitoring: The ablation study's findings provide guidance on critical monitoring points. Defenders should ensure their security solutions comprehensively monitor:

• File Descriptor (FD)-based activities: Crucial for detecting information disclosure and privilege escalation attacks.
• Process-based activities: Essential for identifying command injection and unauthorized code execution.
• Syscall-based activities: Important for detecting a wide range of attacks, including RCE and memory corruptions.

1. Consider Monitoring Granularity and Threshold Tuning: The 30-second monitoring interval (T_snapshot) offers a good balance between lead time and false positives. However, for extremely sensitive data or applications where even minimal exfiltration is critical, defenders might need to evaluate the feasibility of shorter intervals, accepting a potentially higher false positive rate, or combine with more immediate alert mechanisms. Furthermore, the uniform detection threshold of e = 0.3 might not be optimal for all microservices; per-service tuning could enhance precision and recall based on specific risk profiles and application criticality.

1. Be Aware of Evasion Limitations: While stealthy, low-frequency malicious activities can temporarily evade REPLICAWATCHER, the system's detection capability sharpens as attack intensity or frequency increases. Defenders should understand that an attacker's ability to perform extensive malicious actions stealthily is bounded. This implies that cumulative behavior analysis or combining REPLICAWATCHER with other systems that track long-term trends could further enhance detection.

1. Address Deployment Challenges and False Positives: The system acknowledges challenges like replica synchronization during rolling updates and rare bookkeeping activities causing false positives. Implementing Kubernetes readiness-probes to temporarily exclude unstable replicas and establishing processes to handle legitimate but unusual DevOps activities will be important for operationalizing such a system effectively.

In summary, REPLICAWATCHER offers a powerful, practical solution for runtime anomaly detection in dynamic microservice environments. Its training-less nature and resilience to normality drift make it a compelling component of a modern, multi-layered cloud-native security strategy.

Key Takeaways

• Training-less Anomaly Detection is Viable: REPLICAWATCHER demonstrates that effective anomaly detection in dynamic microservices can be achieved without the need for initial training or continuous retraining, overcoming a major limitation of traditional systems.
• Leveraging Replica Consistency: The core innovation is exploiting the inherent behavioral consistency of identical container replicas to identify anomalous deviations, treating such deviations as indicators of security threats.
• Resilience to Normality Drift: The system is robust against environmental changes like OS updates, dependency upgrades, and application code modifications, maintaining stable false positive rates where traditional HIDS fail.
• High Performance with Low Overhead: REPLICAWATCHER achieves high detection rates (average precision 92.48%, recall 98.13%) comparable to state-of-the-art training-based solutions, with negligible runtime overhead suitable for production deployment.
• Comprehensive Feature Monitoring is Key: Monitoring kernel events related to File Descriptors, Processes, and Syscalls is crucial for detecting a diverse range of attacks, from information disclosure to command injection and remote code execution.
• Evasion is Possible but Bounded: While stealthy, low-intensity attacks can temporarily evade detection, the system's efficacy increases with attack intensity, highlighting the limits of an attacker's ability to remain undetected.

About the Speaker(s)

Asbat El Khairi is the speaker who presented the REPLICAWATCHER research at the NDSS Symposium. The provided transcript and metadata do not contain additional details regarding their specific title, affiliation, or background beyond their name.