Kubernetes Attack Simulation The Definitive Guide

Kubernetes Attack Simulation The Definitive Guide

Speakers: Leo Tsaousis, Senior Security Consultant, WithSecure

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=PFeqxSD7Gh8

Overview

In the rapidly evolving landscape of cloud-native infrastructure, securing Kubernetes clusters presents unique and complex challenges for organizations. Leo Tsaousis, a Senior Security Consultant at WithSecure, delivered a comprehensive talk at DEF CON 32 titled "Kubernetes Attack Simulation The Definitive Guide," addressing the critical need for robust defense capabilities in these environments. The presentation delves into the intricacies of Kubernetes security, offering a structured approach to understanding, simulating, and ultimately detecting attacks within a cluster.

The core of Tsaousis's talk revolves around empowering blue teams to effectively measure and enhance their ability to detect and respond to threats in Kubernetes. Drawing from WithSecure's experience with mature clients, the methodology presented emphasizes a threefold adversary simulation assessment: thorough threat modeling, the development of specialized tooling for attack simulation at scale, and collaborative execution of test cases with blue teams. This research culminated in the release of Leonidas for Kubernetes, a novel, Kubernetes-native attack simulation framework designed to facilitate proactive defense building.

This article dissects Tsaousis's insights, providing a detailed exploration of Kubernetes architecture from a security perspective, common attack surfaces and adversary objectives, and the practical application of purple teaming for detection engineering. It highlights the importance of tailored security exercises, the technical underpinnings of the Leonidas tool, and crucial defensive implications for organizations operating Kubernetes at scale.

Background

▶ Watch: Introduction to Kubernetes attack simulation challenges and scope (0:00)

Kubernetes, an open-source system for automating deployment, scaling, and management of containerized applications, has become a cornerstone of modern infrastructure. However, its inherent complexity and extensibility introduce a broad attack surface that often goes unmonitored. Tsaousis begins with a concise, yet informative, introduction to the Kubernetes architecture to lay the groundwork for understanding its security challenges.

A Kubernetes cluster comprises worker machines known as nodes, which host containerized applications. These nodes are managed by a control plane, with the API server acting as the central front-end for all internal and external communications. The API server exposes a REST API, enabling clients like users (typically via kubectl) or services (e.g., CI/CD pipelines) to interact with the cluster. Each node runs a container runtime (such as containerD or older versions of Docker) and an agent called the Kubelet, which communicates with the API server to receive instructions and manage workloads.

Applications are deployed as pods, the basic building blocks, which can contain one or more containers and associated storage volumes. In practice, workloads are often deployed as managed collections of pods, such as Daemon Sets or Deployments. Beyond these, Kubernetes understands other resource types, including Secrets (for sensitive data) and Services (for network access). Resources are logically grouped into namespaces, providing a form of multi-tenancy. Authorization within Kubernetes is primarily handled by RBAC (Role-Based Access Control), which defines permissions for entities. Humans authenticate as user objects using various mechanisms (mTLS client certificates, Active Directory, IAM roles), while pods and external services authenticate via service accounts using JSON Web Tokens (JWTs).

A critical point emphasized is Kubernetes' extensibility. The platform is designed to be highly customizable, leading to diverse environments often incorporating numerous third-party technologies from the CNCF (Cloud Native Computing Foundation) landscape. Each of these technologies introduces its own set of risks, making a generic security approach insufficient. Therefore, threat modeling is presented as an essential prerequisite for any security exercise. This involves answering fundamental questions: What assets need protection? Where would attacks originate? And what are the primary motivations for attackers targeting Kubernetes?

Tsaousis enumerates common attack surfaces:

1. Container Level: The most typical origin, stemming from compromised web services or backdoored container images. If a pod has an assigned service account, an attacker can interact with the control plane, albeit ideally within a limited security context.
2. Node Level: Achieved through a container breakout, allowing the attacker to access the underlying host. This often involves impersonating the Kubelet to gain higher security privileges and potentially access resources across different namespaces.
3. External Attacks: Originating from outside the cluster, such as leaked or stolen credentials (e.g., a kubeconfig file containing a service account JWT), compromised external services, or malicious/coerced users.

Once inside, attackers pursue various objectives:

• Workload Compromise: Gaining unauthorized access to sensitive data handled by applications (proprietary software, customer data) or even backdooring applications to achieve supply chain compromise.
• Compute Resource Exploitation: The speaker notes that in the age of cryptocurrency, compute resources themselves are attractive targets for financial gain, leading to instances of cryptojacking.
• Cloud Infrastructure Pivot: Since Kubernetes clusters are often deployed as managed cloud services, attackers might use them as a stepping stone to pivot into the broader underlying cloud infrastructure.
• Persistence and Defense Evasion: Kubernetes can be used for long-term infection, enabling APT-like surveillance. It can also serve as a less monitored platform for defense evasion, allowing intruders to hide from containment measures.

Given these varied threats, Tsaousis advocates for a purple team methodology for attack simulation. Unlike stealthy, objective-based exercises, purple teaming emphasizes collaboration between offensive and defensive teams to increase familiarity with TTPs (Tactics, Techniques, and Procedures). For a novel environment like Kubernetes, this collaborative approach is deemed more valuable for helping blue teams understand attacker behaviors rather than solely identifying vulnerabilities. WithSecure's approach to purple teaming focuses on detection, offering two strategies: covering as many TTPs as possible in isolation, or chaining attacks to mimic realistic threat actor campaigns.

Key Findings

▶ Watch: Quick overview of Kubernetes architecture and core components (2:00)

The talk highlights several critical findings regarding Kubernetes security and effective defense strategies:

• Complexity and Extensibility Drive Risk: Kubernetes' highly extensible nature, coupled with its inherent complexity and reliance on numerous third-party components from the CNCF landscape, means every deployment is unique and introduces a distinct set of risks that cannot be addressed by a one-size-fits-all security solution.
• Threat Modeling is Foundational: Before any security exercise, comprehensive threat modeling is indispensable. Understanding the specific assets, potential attack origins (container, node, external), and adversary motivations is crucial for tailoring effective simulation and defense strategies.
• Diverse Attacker Objectives: Adversaries targeting Kubernetes are motivated by more than just data theft. Key objectives include exploiting compute resources for financial gain (e.g., cryptomining), using the cluster as a pivot point into broader cloud infrastructure, establishing long-term persistence, and employing Kubernetes as a platform for defense evasion due to its often-under-monitored status.
• Purple Teaming for Proactive Detection: A collaborative purple team methodology is presented as the most effective approach for improving Kubernetes security. By working together, offensive and defensive teams can gain a deeper understanding of attacker TTPs, leading to more robust and proactive detection engineering rather than just identifying vulnerabilities.
• Introduction of Leonidas for Kubernetes: A significant contribution is the release of Leonidas for Kubernetes, a dedicated, Kubernetes-native attack simulation framework. This tool allows security teams to simulate realistic attacks against their clusters, providing a practical means to test and validate their detection capabilities.
• Focus on Detection Engineering: The talk strongly advocates for a shift towards active detection engineering in Kubernetes environments. This involves leveraging diverse log sources (from the API server, Kubelet, container runtimes), understanding the context of Kubernetes resources (pods, secrets, service accounts), and developing specific detection rules (e.g., using Sigma rules provided with Leonidas) to identify malicious activities.
• Service Account Tokens as Primary Attack Vectors: The demonstration clearly illustrates how leaked kubeconfig files containing Service Account JWT tokens can serve as a primary initial access vector, emphasizing the need for stringent protection and monitoring of these credentials.

Technical Deep Dive

▶ Watch: Kubernetes extensibility and why threat modeling is crucial (4:10)

Kubernetes' architecture, while powerful, presents a complex landscape for security practitioners. Understanding its core components and how they interact is crucial for effective attack simulation and defense.

At its heart, a Kubernetes cluster consists of nodes, which are worker machines running containerized applications. These nodes are orchestrated by the control plane, a set of components that manage the cluster. The central component of the control plane is the API server, which acts as the front-end for Kubernetes. It exposes a REST API that all internal and external entities, including users (typically interacting via the kubectl command-line tool) and services (like CI/CD pipelines), use to communicate with the cluster. On each node, an agent called the Kubelet communicates with the API server, receiving instructions and managing the node's workloads. Nodes also require a container runtime, such as containerD or, in older versions, Docker, to execute containers.

Workloads are deployed as pods, the smallest deployable units in Kubernetes. A pod can contain one or more containers, whose images are pulled from registries, along with associated storage volumes. More commonly, applications are deployed as managed collections of pods, such as Deployments (for stateless applications) or Daemon Sets (which ensure a pod runs on every node). Other key Kubernetes resources include Secrets, used to store sensitive information like passwords or API keys, and Services, which define a logical set of pods and a policy for accessing them. These resources are often logically grouped into namespaces for organizational purposes, akin to tenants in a cloud environment.

Authorization within Kubernetes is governed by RBAC (Role-Based Access Control). RBAC policies define the permissions that various entities have over resources. Human users authenticate using mechanisms like mTLS client certificates, Active Directory, or IAM roles. Crucially, pods and external services are associated with service accounts, which authenticate using JSON Web Tokens (JWTs). These tokens are often mounted into pods automatically and can be a significant target for attackers.

The talk elaborates on common attack surfaces with technical specificity:

• Container Level Attacks: These originate from within a running container, perhaps due to a vulnerability in a deployed web service or a backdoored container image introduced into the cluster. If the compromised pod is associated with a service account, an attacker can leverage its JWT to interact with the API server. The scope of these interactions depends on the service account's RBAC permissions.
• Node Level Attacks: A more severe form involves a container breakout, where an attacker escapes the container's isolation and gains access to the underlying node. From the node, an attacker might attempt to impersonate the Kubelet, which typically possesses high privileges, to gain broader access to other Kubernetes resources and namespaces, potentially escalating privileges significantly.
• External Attacks: These attacks bypass the internal cluster components, focusing on external access points. Examples include leaked kubeconfig files (which often contain credentials like service account JWTs or user certificates), compromised external services with Kubernetes access, or direct attacks by malicious or coerced legitimate users.

To facilitate attack simulation, Tsaousis introduces Leonidas for Kubernetes, a Kubernetes-native framework. Leonidas operates by:

1. Attack Definition: Security practitioners define attacks, which are then used by an accompanying generator tool.
2. Code and Manifest Generation: The generator produces Python code from these definitions and standard Kubernetes YAML manifests.
3. Deployment: These YAML manifests are applied to the cluster using kubectl apply, deploying the Leonidas service itself into a specified namespace.
4. Interaction: Leonidas can be interacted with via a simple client UI or, as demonstrated, through Jupyter notebooks, allowing for programmatic execution of attack scenarios using Python.

A key technical detail in the demo involves simulating initial access via a leaked kubeconfig file. This file contains a service account token (JWT). The attacker then instantiates a client to Leonidas using this token. The first step for the simulated attacker (fictitious threat actor OF815) is to enumerate their own permissions using the self subject review API call. This critical API endpoint allows an entity to determine what actions it is authorized to perform. The demonstration reveals that the compromised service account has create, read, update, and delete permissions on pods and secrets, highlighting a common misconfiguration. Following this, the attacker enumerates pods within the cluster, identifying crucial workloads like patient DB and H controller.

Demo / Proof of Concept

▶ Watch: Enumerating common Kubernetes attack surfaces: container, node, external (4:55)

The practical demonstration provided a clear, end-to-end simulation of an attack against a Kubernetes cluster using the Leonidas for Kubernetes tool. The scenario was designed to showcase initial access, permission enumeration, and resource discovery, mirroring realistic threat actor methodologies.

The setup involved a live Kubernetes cluster with a dashboard visible, confirming its operational status. Within this cluster, a namespace named Dharma Prods was highlighted. This namespace contained a deployment called H controller, which in turn comprised two pods: one representing a database (patient DB) and another performing sensitive operations (H controller). This provided a plausible target environment for the simulated attack.

The first step in the demonstration was to deploy the Leonidas attack simulation framework into the target cluster. This was achieved using an accompanying generator tool. The generator first processed predefined attack definitions to create Python code, and then generated standard Kubernetes YAML manifests. These manifests were subsequently applied to the cluster using kubectl apply, resulting in the deployment of the Leonidas service within the Dharma Prods namespace. Once Leonidas was running, its API was exposed locally on port 5000 using a kubectl port-forward command, allowing for external interaction.

Interaction with Leonidas was demonstrated through a Jupyter Notebook, which provided a flexible and scriptable environment for orchestrating the attack. The speaker navigated to a notebook specifically crafted for a fictitious threat actor, designated OF815. This notebook contained Python code designed to interact with the Leonidas service running inside the cluster.

The simulated attack sequence commenced with initial access, a common entry point for adversaries. OF815 gained access through a leaked kubeconfig file that contained a Service Account JWT (JSON Web Token). This token was then used to instantiate a client to the Leonidas service from within the Jupyter Notebook, effectively simulating the attacker's compromised identity.

With initial access established, the next crucial step for OF815 was permission enumeration. The Jupyter Notebook executed a call to the Kubernetes self subject review API. This API allows an authenticated entity to determine its own permissions within the cluster. The results of this call were displayed, clearly showing that the compromised service account possessed create, read, update, and delete (CRUD) permissions on both pods and secrets. This finding immediately highlighted a significant security weakness, as broad permissions on secrets could lead to sensitive data exposure.

Following permission enumeration, the demonstration proceeded to pod enumeration. The attacker (via the Jupyter Notebook and Leonidas) listed the pods running within the cluster. The output confirmed the presence of patient DB, the Leonidas deployment itself, and the H controller pods, confirming the attacker's ability to discover running workloads. This step is critical in a real attack scenario for identifying potential targets for further exploitation or lateral movement.

The demonstration effectively illustrated how Leonidas can be used to simulate key stages of an attack, from initial access via compromised credentials to internal reconnaissance, providing blue teams with concrete examples of malicious activity to detect.

Defensive Implications

▶ Watch: What attackers aim for inside a compromised Kubernetes cluster (6:00)

The insights from Leo Tsaousis's talk offer crucial guidance for defenders seeking to fortify their Kubernetes environments. The primary takeaway is the necessity of a proactive, collaborative approach to security, shifting from reactive vulnerability patching to robust detection engineering.

1. Prioritize Threat Modeling: Given the unique and extensible nature of each Kubernetes deployment, a generic security posture is insufficient. Defenders must undertake thorough threat modeling to identify specific assets, potential attack vectors (container, node, external), and relevant adversary objectives for their unique environment. This foundational step ensures that defensive efforts are tailored and effective.

1. Embrace Purple Teaming: The talk strongly advocates for a purple team methodology. Blue teams should actively collaborate with red teams to understand attacker TTPs in a practical, hands-on manner. This iterative process of simulating attacks and immediately testing detection capabilities significantly enhances the blue team's understanding of attacker behavior and improves their ability to write effective detection rules.

1. Focus on Detection Engineering: A core defensive implication is the need for dedicated detection engineering in Kubernetes. This involves:

• Leveraging Comprehensive Log Sources: Defenders must collect and analyze logs from all critical Kubernetes components, including the API server (for all cluster interactions), Kubelet (for node-level activities and container lifecycle events), and container runtimes (for container-specific processes and syscalls). These logs are the raw material for detecting malicious activity.
• Developing Kubernetes-Native Detections: Generic host-based detections may not suffice. Defenders need to develop detections that understand Kubernetes objects, API calls, and the relationships between pods, service accounts, and namespaces. The speaker explicitly mentions releasing Sigma rules alongside Leonidas, providing a valuable starting point for building these native detections.
• Monitoring for Privilege Escalation: Pay close attention to activities that indicate privilege escalation, such as a service account attempting to use the self subject review API to enumerate unexpected permissions, or any signs of container breakouts and attempts to impersonate the Kubelet.

1. Secure Service Accounts and kubeconfig Files: The demonstration highlighted the critical risk posed by leaked kubeconfig files containing Service Account JWTs. Defenders must:

• Implement strict RBAC policies following the principle of least privilege for all service accounts.
• Regularly audit service account permissions to ensure they align with their intended function.
• Protect kubeconfig files with the same rigor as other critical credentials, using strong access controls, encryption, and monitoring for unauthorized access.
• Monitor for unusual usage patterns of service account tokens, especially from unexpected sources or for unauthorized API calls.

1. Monitor for Cryptomining and Resource Abuse: Given that compute resources are a prime target, defenders should implement monitoring for unusual resource consumption (CPU, memory, network egress) within pods and nodes. Spikes in resource usage, particularly from unexpected or newly deployed workloads, can indicate cryptomining operations.

1. Address Third-Party Risks: Acknowledge that Kubernetes' extensibility means numerous third-party technologies introduce their own risks. Defenders must track all components from the CNCF landscape deployed in their clusters, assess their individual security posture, and ensure they are patched and configured securely.

1. Utilize Attack Simulation Tools: Tools like Leonidas for Kubernetes are invaluable for testing and validating defensive capabilities. By regularly simulating realistic attacks, blue teams can identify gaps in their visibility, detection logic, and incident response procedures before a real adversary exploits them. This proactive testing is essential for continuously improving a cluster's security posture.

Key Takeaways

• Threat Modeling is Non-Negotiable: Kubernetes' complexity and extensibility demand comprehensive, environment-specific threat modeling as the foundational step for any effective security strategy.
• Purple Teaming is Paramount for Detection: A collaborative purple team approach, focusing on understanding attacker TTPs, is the most effective way to build and mature Kubernetes detection capabilities and proactively strengthen blue teams.
• Attackers Target Diverse Objectives: Beyond data theft, adversaries aim for compute resource exploitation (e.g., cryptomining), pivots into underlying cloud infrastructure, establishing persistence, and using Kubernetes for defense evasion.
• Leonidas for Kubernetes Empowers Defenders: The new Kubernetes-native attack simulation framework, Leonidas, provides a practical tool for simulating realistic attacks, enabling organizations to test and validate their detection engineering and incident response processes.
• Robust Detection Engineering is Essential: Defenders must prioritize developing Kubernetes-native detections by leveraging comprehensive log sources (API server, Kubelet, container runtime) and applying the principle of least privilege to service accounts, actively monitoring for suspicious API calls, permission enumerations, and resource abuse.
• Secure Service Account Credentials: Leaked kubeconfig files containing Service Account JWTs represent a critical initial access vector; stringent protection, auditing of RBAC permissions, and continuous monitoring of service account activity are vital.

About the Speaker(s)

Leo Tsaousis is a Senior Security Consultant at WithSecure, based in Manchester, UK. His professional focus lies primarily in threat simulation and conducting collaborative security assessments. Tsaousis's expertise is instrumental in helping clients understand and improve their capabilities to detect and respond to sophisticated attacks across diverse environments, including novel container orchestration platforms like Kubernetes. His work at WithSecure directly informed the research and development behind the Leonidas for Kubernetes tool and the purple team methodologies presented in this talk.