Phoenix: Surviving Unpatched Vulnerabilities via Accurate and Efficient Filtering of Syscall Sequences

Phoenix: Surviving Unpatched Vulnerabilities via Accurate and Efficient Filtering of Syscall Sequences

Speakers: Hugo Kermabon-Bobinnec

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=uKXsblCKXHE

Overview

The digital landscape is relentlessly challenged by unpatched vulnerabilities, posing a severe dilemma for businesses: risk exposure by keeping vulnerable services online or incur significant financial and social costs by shutting them down. Research indicates an average time-to-patch of approximately 100 days for zero-day vulnerabilities, with an additional 422 days often required for vendors to release patched container images. This protracted vulnerability window has led to catastrophic incidents, such as the Equifax data breach (CVE-2017-5638) and the Log4Shell crisis, which forced the Canadian government to disable nearly 4,000 services. The problem is particularly acute in container-based cloud services, where container images are frequently buggy, and weaker isolation mechanisms can enable adversaries to escape compromised containers and attack the underlying host system.

Existing security solutions offer only partial mitigation with significant drawbacks. System call-level filtering, like Seccomp (SECure COMPuting), can reduce the general attack surface by blocking unused system calls but fails against exploits leveraging system calls also essential for legitimate application operations. Furthermore, Seccomp filters are inherently stateless and lack deep argument inspection, leading to inaccuracies and potential false positives. Conversely, tools like Ptrace provide highly accurate monitoring and control by inspecting every system call and its arguments, but this precision comes at a prohibitive performance cost, especially for high-throughput applications like an Nginx web server, which can issue over 1,800 system calls per second. Provenance analysis solutions, such as Falco or CLARION, excel at identifying malicious behaviors, often expressed as sequences of system calls, but critically lack a direct mechanism to block these identified sequences at runtime.

Phoenix emerges as a novel solution designed to bridge this critical gap between detection and prevention. It proposes an innovative approach to prevent exploits of unpatched vulnerabilities by accurately and efficiently filtering sequences of system calls identified through provenance analysis. Phoenix cleverly combines the efficiency of Seccomp for fast pre-screening with the accuracy of Ptrace for deep argument inspection. Its core innovation lies in a dynamic Seccomp design that enables the filtering of system call sequences. This means the heavy Ptrace inspection is only triggered when Seccomp detects a potential match for a malicious sequence, and Seccomp filters are then dynamically updated to track the progression of the attack. This dual-pronged strategy offers substantial benefits: it allows for the crowdsourcing of temporary patches in the form of malicious system call sequences without requiring source code bug understanding, drastically reducing reliance on lengthy vendor patch cycles, and provides a versatile, low-level defense against a wide array of attacks.

Background

▶ Watch: Problem: Unpatched vulnerabilities and current solutions' limits (0:00)

The foundation of Phoenix rests on a thorough understanding of containerization, system call mechanisms, existing security tools, and provenance analysis. Containerization, an OS-level virtualization technology, provides isolated user spaces for applications, offering benefits such as transient deployments and enhanced observability of system interactions. However, this isolation is weaker than traditional virtual machines, making containers vulnerable to kernel exploits, such as CVE-2022-0847, which can compromise the host system. Phoenix leverages the transient nature and observability of containers to implement its protection mechanisms.

System calls serve as the fundamental interface through which applications within containers request services from the Linux kernel. By default, applications often have access to a broad range of system calls, many of which are not strictly necessary for their legitimate operation. Attackers frequently exploit vulnerabilities to coerce applications into executing malicious operations—such as privilege escalation, information leakage, or container escape—using these system calls, whether they are normally unused or even legitimate ones repurposed for nefarious ends.

Seccomp (SECure COMPuting) is a crucial Linux kernel security mechanism that allows user-space programs to restrict their access to system calls. Seccomp-BPF filters can be configured to perform various actions, including allowing, denying, or sending a signal upon matching specific system calls. Seccomp filters are widely adopted due to their negligible performance overhead. However, they suffer from two significant limitations: they are stateless, meaning they cannot remember previously observed system calls, and they lack the ability to perform deep argument inspection, which involves dereferencing pointers or structures within system call arguments to examine their actual content. This inability makes them susceptible to exploits that use legitimate system calls with malicious arguments or those that unfold across a sequence of calls. Phoenix directly addresses these limitations by integrating Ptrace and dynamically updating Seccomp filters.

Ptrace is another Linux debugging and tracing utility that allows a process to control another, enabling it to inspect and manipulate the target's execution, including its system call arguments. While Ptrace offers the deep argument inspection and statefulness that Seccomp lacks, its intrusive nature and context-switching overhead make it prohibitively expensive for continuous, high-throughput monitoring in production environments. Phoenix's design specifically aims to harness Ptrace's accuracy only when truly necessary, mitigating its performance impact.

Provenance analysis techniques, exemplified by solutions like CLARION and ATLAS, are instrumental in investigating the root causes of security incidents. These techniques construct provenance graphs, typically directed acyclic graphs, that represent the flow of information and causal relationships between subjects (processes) and objects (files, network sockets, pipes) during system execution. Such graphs provide a rich, contextual understanding of system events, even when temporal relationships are ambiguous. Phoenix leverages this capability to systematically identify the precise sequences of system calls that constitute an exploit, transforming detection into actionable prevention.

Phoenix's threat model specifically targets attacks that manifest as distinct system call sequences and their arguments. Crucially, it considers vulnerabilities that exploit system calls similar to those required by the container's normal behavior, which often bypass simpler Seccomp policies. The model assumes the integrity of Phoenix itself, the underlying infrastructure, and its audit logs. Out-of-scope threats include unknown zero-day attacks that have not yet been identified and characterized, attacks that do not involve system calls issued by the container to the host OS, and sophisticated attacks capable of tampering with Phoenix or the host infrastructure directly.

Key Findings

▶ Watch: Phoenix's three main contributions summarized (3:17)

Phoenix's comprehensive evaluation against real-world vulnerabilities and diverse application workloads yielded several significant findings, demonstrating its superior effectiveness, minimal overhead, and practical applicability:

• Unrivaled Vulnerability Mitigation: Phoenix successfully blocked all 20 real-world CVEs tested (with CVSS scores ranging from 2.1 to 7.8) across 10 popular applications (including Nginx, Tomcat, and MySQL). This stands in stark contrast to existing solutions; Docker's default Seccomp filter blocked only one CVE, while Confine and Sysfilter demonstrated varying degrees of success, often failing against CVEs that repurposed legitimate system calls. Notably, CVE-2018-14634, an integer overflow in the Linux kernel, largely evaded Confine and Sysfilter, highlighting Phoenix's unique ability to handle complex, multi-syscall exploits.
• Negligible Performance Overhead:
• Response Time: In idle and normal operational scenarios (e.g., 100 attacks/day), Phoenix introduced almost no extra overhead compared to unprotected baselines. On average, Phoenix added only 4% additional delay compared to Default Seccomp, whereas a naive Ptrace implementation increased response times by an average of 145% (e.g., 160% for Django). Even in DoS attack scenarios, Phoenix's overhead remained acceptable and significantly less than Ptrace.
• CPU Consumption: Phoenix's CPU consumption remained around 0% in idle and normal cases, peaking at a modest 7% during DoS attacks. This represents a remarkable 98% CPU saving compared to Ptrace, which continuously consumed over 5% CPU, often as much as the application itself.
• Memory Consumption: Memory consumption was observed to be constant and similar across all tested configurations.
• Criticality of Deep Argument Inspection (DAI): DAI proved indispensable for accuracy, drastically reducing false positives. For single-system call blocking, DAI reduced false positives by 77.4% on average. For sequences of two and three system calls, DAI reduced false positives by 94% and 97.2% on average, respectively, completely eliminating false positives for some CVEs. This confirms that examining system call arguments in detail is crucial for precise and effective filtering.
• Feasible Sequence Identification and Crowdsourcing Potential: The semi-automated approach to malicious sequence identification, combining automated tools with human expertise, was validated as practical. A user study showed that participants unfamiliar with specific vulnerabilities could identify reasonably small attack subgraphs (62-634 nodes/edges) in less than 30 minutes, with experts accomplishing the task in under 6 minutes. This demonstrates the feasibility of crowdsourcing temporary patches in the form of identified malicious sequences.
• Scalability and General Applicability: Phoenix's design, utilizing bounded Finite State Machines (FSMs) and efficient Seccomp pre-filtering, ensures scalability. Furthermore, the core methodology can potentially be applied to any OS process that implements an in-kernel system call restriction mechanism, extending its utility beyond containerized environments.

Technical Deep Dive

▶ Watch: Background: Containers, Seccomp limitations, and provenance analysis (4:00)

Phoenix's methodology is structured into two principal phases: Malicious Sequence Identification and Dynamic Runtime Protection.

Malicious Sequence Identification

This phase is designed to identify and characterize the specific sequence of system calls that constitutes an exploit. It facilitates a crowdsourcing model among affected users.

1. Provenance Graph Building: Phoenix continuously collects system audit logs from victim containers. It leverages CLARION, a namespace-aware and container-aware extension of SPADE, to gather detailed data including system calls, timestamps, Process IDs (PIDs), and other metadata. A critical aspect here is deep argument inspection (DAI), where Phoenix dereferences structures and pointers within system call arguments (e.g., converting a sockaddr memory address into its constituent AF_INET family, port, and IP address). This granular data is then transformed into a provenance graph using tools like CLARION and CamFlow, where system resources (processes, files, network sockets) are represented as nodes, and system calls are represented as edges, illustrating the flow of information and causal relationships. For example, in a web application attack, an alert at a specific timestamp might map to a graph showing a process opening /etc/passwd and then splicing data to a newly created pipe.

1. Root Cause Analysis and Sequence Extraction: The objective is to pinpoint the subgraph(s) within the provenance graph that are directly related to a security alert. Phoenix maps the alert to relevant nodes in the graph. Users, often security analysts, then employ existing root cause analysis solutions like DepImpact or ATLAS, combined with their domain knowledge, to identify the precise attack subgraph(s). For sequence extraction, Phoenix traces back the events in the identified attack subgraph to their original system calls in the audit logs, reconstructing a chronological sequence complete with all relevant parameters. This candidate sequence is then presented in a Graphical User Interface (GUI) for expert validation, allowing administrators to confirm the final malicious sequence and assign a corresponding action plan. For instance, an alert indicating "Super user logged in" following a CVE-2022-0847 exploit could be mapped to an execve system call. Root cause analysis might reveal a subgraph involving open, pipe, splice, and execve, which Phoenix would then chronologically order into the malicious sequence: open -> pipe -> splice -> execve.

Dynamic Runtime Protection

Once a malicious sequence is identified, Phoenix transitions to runtime protection, hardening the container against future instances of that exploit.

1. Sequence Abstraction: Phoenix transforms the incident-specific malicious sequence into a more generic version to ensure broader applicability. This involves categorizing system call parameters:

• (i) Precisely matched parameters: These are critical prerequisites for the attack (e.g., specific flags like RDONLY or well-known connection ports). Their exact values are preserved.
• (ii) Abstracted parameters: These are contextual parameters (e.g., IP addresses, file descriptor numbers, Process IDs) whose exact values vary but are shared across multiple system calls within the sequence. Phoenix replaces these with variables, ensuring consistency throughout the sequence. For example, a file descriptor like '4' might be abstracted to a variable 'X' if it's reused, and PIDs '12' and '13' could become 'A' and 'B'.
• (iii) Ignored parameters: These are incidental parameters not critical to the attack (e.g., certain strings or filenames that might change). They are removed from the sequence definition to prevent false negatives. For example, specific filenames like passwd or su might be ignored if the exploit could target other files.

1. Building Sequence State Machine (FSM): To effectively track the progression of a potential attack, Phoenix constructs a Finite State Machine (FSM). This FSM starts with an initial state and includes one state for each system call in the abstracted malicious sequence. Transitions between states occur only when an intercepted system call and its parameters satisfy specific conditions. The FSM serves three primary purposes: tracking intercepted system calls, matching system calls and their parameters against the malicious sequence, and enforcing pre-defined security actions. Transition conditions are determined by the parameter types: precisely matched parameters are always part of the conditions; abstracted parameters are part of conditions if they were defined in a prior state (i.e., their variable binding is already known), or they are defined in the current state if encountered for the first time; ignored parameters are not considered in the conditions. This design ensures both accuracy (minimizing false positives) and generality (matching different instances of the same attack).

1. Sequence State Monitoring: This is the core of Phoenix's active defense mechanism.

• Phoenix initializes the Seccomp filter to monitor specifically for the first system call in the malicious sequence.
• When Seccomp matches this initial system call, instead of blocking it directly, it triggers Ptrace by setting its return action to SECCOMP_RET_TRACE. This notifies Ptrace that a potential malicious sequence has begun, enabling Phoenix's novel efficiency-accuracy trade-off.
• Ptrace then takes over, inspecting the system call's arguments and the calling process ID with its full deep inspection capabilities.
• The FSM verifies if the intercepted system call and its arguments match the current transition conditions for the sequence. If there is no match, the system call is allowed to proceed without further intervention.
• If a match is confirmed, four critical actions occur: (i) the FSM's current state is updated to reflect progress in the attack sequence; (ii) the Seccomp filter is dynamically updated to monitor for the next system call in the malicious sequence; (iii) any new abstracted parameters identified in the current system call are defined and stored; and (iv) the user-specified security action for the current state (i.e., for this specific system call in the sequence) is enforced.

1. Security Action Enforcement: Phoenix provides users with a flexible set of security actions that can be predefined for each system call within the malicious sequence, allowing for a nuanced balance between availability and security:

• Step: Allows the system call to execute without any notice or intervention.
• Warn: Allows the system call but logs an alert to notify administrators.
• Block: Prevents the system call from executing, but the calling process is not terminated.
• Exit: Prevents the system call from executing and terminates the calling process.
• Kill: Prevents the system call from executing and initiates a shutdown of the entire container.

Users can combine these actions, for example, using "step" for initial calls that are less critical, "warn" for subsequent calls indicating escalating risk, and "block" or "kill" for truly damaging system calls. These actions can also be dynamically modified based on increasing confidence in an attack or observed impact.

A specific challenge arises when monitoring processes created outside the container's direct parent-child hierarchy, such as those spawned by docker exec or kubectl exec commands. These are "sibling" processes. Phoenix addresses this with a shim tracer that observes the container's shim process for clone or fork events. Upon such an event, the tracer interrupts the newly created process, detaches from it, and sends a SIGSTOP signal to the main Phoenix process, transmitting the new process's PID via a message queue. Phoenix then handles the SIGSTOP to resume and trace this new process as it would any other.

Implementation Details

Phoenix is implemented in C and C++ for Linux kernel v5.10, comprising approximately 2,000 lines of code. Crucially, it requires no modifications to existing applications, container runtimes (like Docker or containerd), or the Linux kernel itself, relying solely on a custom kernel module. The dynamic runtime protection module is split between user-space and kernel-space components.

• User-space Components:
• Sequence Preprocessor: A C++ module responsible for parsing the malicious sequence, typically provided in JSON format using the JSON library, and constructing the FSM based on the abstracted parameters.
• Sequence State Monitor and Ptrace: This component runs as a privileged process on the host OS. It collects container runtime information (e.g., via docker inspect), attaches Ptrace to all active container processes using PTRACE_ATTACH, and diligently traces any child processes spawned via PTRACE_EVENT_FORK, CLONE, and VFORK. It also monitors for Seccomp signals using PTRACE_EVENT_SECCOMP. A key engineering challenge—differentiating between signals from forked processes and Seccomp interruptions—is solved by comparing the status of the stopped process with Ptrace event conditions. System call arguments are retrieved using PTRACE_PEEKUSER and PTRACE_PEEKDATA.

• Kernel-space Components:
• Seccomp Filters: While Seccomp filters are installed by default on Docker and containerd, Phoenix specifically configures them to use SECCOMP_RET_TRACE for efficient pre-filtering, signaling Ptrace upon a potential match rather than immediately blocking.
• Custom Kernel Module: This module, roughly 400 lines of C code, is loaded at runtime. It provides two essential functions invoked from user space via ioctl: get_seccomp(process_pid) retrieves the address of the current Seccomp filter for a given process, and update_seccomp(*seccomp_filter) compiles a new Seccomp filter (provided as a BPF program from user space) and replaces the BPF program pointer within the task_struct of the target process. This novel mechanism enables the dynamic, on-the-fly updates of Seccomp filters, which is not natively supported or trivially possible otherwise.

Demo / Proof of Concept

▶ Watch: Phoenix's specific threat model explained (5:30)

The practical efficacy of Phoenix was rigorously demonstrated through an extensive evaluation, serving as a comprehensive proof of concept. The system was deployed on a Kubernetes cluster running Ubuntu 20.04 with Linux kernel v5.10, utilizing real-world attack datasets including the Container Breakout (CB-DS) and DongTing datasets.

A primary demonstration involved a case study assessing Phoenix's ability to prevent 20 real-world CVEs (with CVSS scores ranging from 2.1 to 7.8) across 10 popular containerized applications such as Nginx, Tomcat, and MySQL. This evaluation directly compared Phoenix against existing solutions like Confine, Sysfilter, and Docker's default Seccomp filter. The results unequivocally showed that Phoenix consistently blocked all 20 studied CVEs on every application. In stark contrast, Docker's default Seccomp filter only managed to block one CVE, while Confine and Sysfilter exhibited varying degrees of success, frequently failing to mitigate CVEs that leveraged system calls also required for legitimate application operations. For instance, CVE-2018-14634, an integer overflow in the Linux kernel, almost entirely bypassed Confine and Sysfilter, underscoring Phoenix's superior effectiveness due to its stateful system call sequence analysis and deep argument inspection.

Further demonstrations focused on performance, measuring response time, CPU, and memory consumption for applications like Django, Nginx, and MySQL under various load scenarios: idle, normal (simulating 100 attacks/day), and DoS (an attack on every request). Phoenix was compared against unprotected baselines, default Seccomp, and a naive Ptrace implementation. In idle and normal scenarios, Phoenix introduced almost no measurable overhead in response time, while the naive Ptrace solution at least doubled it (e.g., a 160% increase for Django). Phoenix consistently added only 4% additional delay on average compared to Default Seccomp, a fraction of Ptrace's 145% increase. CPU consumption for Phoenix remained around 0% in idle and normal cases, peaking at only 7% during DoS attacks, translating to a 98% CPU saving over Ptrace. These results robustly demonstrate that Phoenix effectively prevents complex exploits without imposing the prohibitive performance penalties associated with highly accurate, Ptrace-based monitoring.

The feasibility of the sequence identification approach was also demonstrated through comparisons with other tools (Strace/Auditd/Sysdig, Nimos, Madani et al., CLARION, DepImpact) and a user study. While traditional tools produced overwhelming outputs with high false positives, Phoenix's semi-automated method produced actionable sequences. The user study confirmed that participants could identify malicious sequences in less than 30 minutes, with experts achieving this in under 6 minutes, validating the practicality of the approach for real-world deployment and potential crowdsourcing. Finally, the security benefits of Deep Argument Inspection (DAI) were demonstrated by showing significant reductions in false positives—by 77.4% for 1-syscall blocking, and up to 97.2% for 3-syscall blocking—with negligible additional performance overhead, confirming DAI's critical role in achieving high accuracy without sacrificing efficiency.

Defensive Implications

▶ Watch: Phoenix methodology: Malicious sequence identification (6:00)

Phoenix offers several critical defensive implications for organizations grappling with unpatched vulnerabilities, particularly in containerized environments:

• Crucial Temporary Protection: Phoenix provides a vital lifeline during the prolonged period awaiting official vendor patches. Given that zero-day vulnerabilities can remain unpatched for over 100 days and container image fixes can take over 400 days, Phoenix offers a robust, temporary shield, minimizing the window of exposure to known exploits.
• Enhanced Precision in Attack Surface Reduction: While traditional Seccomp policies reduce the attack surface by blocking unused system calls, Phoenix goes further. It precisely blocks malicious sequences of system calls, even if individual calls within the sequence are legitimate and necessary for normal application function. This dramatically reduces the risk of exploits that cleverly repurpose legitimate functionality.
• Leveraging Crowdsourced Intelligence: The malicious sequence identification framework enables a crowdsourcing model for "temporary patches." As soon as an exploit sequence is identified by one affected entity, it can be abstracted and shared, allowing other users to harden their systems preemptively, significantly accelerating defense against new exploit variants.
• Bridging Detection and Prevention: Phoenix transforms the output of provenance analysis tools—which are excellent at detecting and characterizing attacks—into immediate, actionable runtime prevention. This bridges a critical gap in the cybersecurity kill chain, moving beyond mere detection to proactive defense.
• Granular and Adaptive Enforcement: Defenders gain fine-grained control over security responses. By defining specific actions (Step, Warn, Block, Exit, Kill) for each system call in a malicious sequence, organizations can tailor their defensive posture to balance security needs with application availability, dynamically adjusting responses as an attack progresses or confidence levels change.
• Robust Container Security: For containerized deployments, Phoenix addresses the inherent weaker isolation that makes them susceptible to container escape and host compromise (e.g., CVE-2022-0847). By actively monitoring and blocking malicious system call sequences originating from containers, it significantly strengthens the security posture of cloud-native applications.
• Improved Accuracy through Deep Argument Inspection: The integration of deep argument inspection (DAI) is paramount for minimizing false positives. Defenders can deploy Phoenix with high confidence that legitimate application behaviors will not be inadvertently blocked, a common challenge with stateless or argument-blind filtering mechanisms. This allows security teams to focus on actual threats rather than investigating benign alerts.
• Complementary Layered Defense: Phoenix is designed to complement existing security solutions rather than replace them. It integrates seamlessly with tools that block unused system calls or provide attack detection, adding a unique and powerful layer of stateful, sequence-based prevention to a holistic security architecture.

Key Takeaways

• Effective Mitigation of Unpatched Vulnerabilities: Phoenix provides a novel and practical solution for temporarily protecting containers against unpatched vulnerabilities by accurately and efficiently filtering malicious system call sequences.
• Hybrid Efficiency and Accuracy: It achieves a superior balance of efficiency and accuracy by dynamically combining the speed of Seccomp for pre-filtering with the deep argument inspection capabilities of Ptrace, triggered only when a potential threat is detected.
• Outperforms Existing Solutions with Negligible Overhead: Phoenix consistently blocked all 20 real-world CVEs tested, significantly outperforming Docker's default Seccomp, Confine, and Sysfilter. It introduced negligible performance delay (less than 4% on average) and achieved substantial CPU savings (98% less CPU consumption than Ptrace).
• Deep Argument Inspection is Critical: The integration of deep argument inspection drastically reduces false positives, improving accuracy by up to 97.2% on average for multi-syscall blocking, ensuring legitimate application functionality is preserved.
• Practical and Scalable Sequence Identification: The semi-automated malicious sequence identification approach is practical, enabling users to identify exploit sequences in under 30 minutes, facilitating the crowdsourcing of "temporary patches" to accelerate defense.
• Versatile and Complementary Defense: Phoenix's general methodology can be applied to any OS process with in-kernel syscall restriction, offering a versatile low-level defense that complements and enhances existing security solutions by turning detection into preventive action.

About the Speaker(s)

The work "Phoenix: Surviving Unpatched Vulnerabilities via Accurate and Efficient Filtering of Syscall Sequences" was presented by Hugo Kermabon-Bobinnec at the NDSS Symposium. The transcript indicates that Mr. Kermabon-Bobinnec is a researcher involved in this significant work, highlighting his expertise in cybersecurity and system-level protection mechanisms. Beyond his name and role as a presenter of this research, no further biographical details such as his title or company affiliation were provided in the transcript.