A Commencement into Real Kubernetes Security

A Commencement into Real Kubernetes Security

Speakers: Jay Beale, CTO, InGuardians; Mark Manning

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=5YHcw-qj094

Overview

In "A Commencement into Real Kubernetes Security," Mark Manning and Jay Beale challenge conventional wisdom surrounding Kubernetes security, urging practitioners to shift their focus from theoretical, "scariest" threats to practical, real-world attack vectors. The talk highlights a significant disconnect between what is commonly taught or assumed about securing Kubernetes environments and the actual vulnerabilities exploited by adversaries in production. Manning and Beale, both seasoned practitioners in Kubernetes penetration testing and cloud security, leverage their extensive consulting experience to expose these disparities.

The core message revolves around the idea that many organizations misprioritize their security investments, often chasing high-profile, complex vulnerabilities like container breakouts or zero-day exploits, while neglecting more common and easily exploitable misconfigurations in areas like Role-Based Access Control (RBAC) and admission control. This talk serves as a "commencement" for attendees, aiming to equip them with a more nuanced and practical perspective on defending Kubernetes, emphasizing risk prioritization and continuous education over rigid adherence to checklists or fear-driven remediation.

This article will delve into the speakers' arguments, explore the technical underpinnings of container security, detail a compelling demonstration of compromising a "compliant" Kubernetes cluster, and outline actionable defensive strategies for organizations striving for genuinely effective Kubernetes security.

Background

▶ Watch: Speakers introduce challenging Kubernetes security assumptions (0:00)

Kubernetes, at its essence, orchestrates machines (nodes) to run containers within a self-healing, scaling cluster, automatically managing workload placement. The talk explores Kubernetes security from the deepest layers of the Linux kernel, where system calls and container breakouts occur, all the way up to the cluster's API level, addressing misconfigurations that can lead to full cluster compromise.

A foundational concept in container security is the understanding that containers are not a security boundary in the same way virtual machines are. They share the same Linux kernel, making them inherently less isolated. While Linux namespaces provide process separation, they do not constitute true sandboxing. This shared kernel architecture immediately raises concerns about container breakouts, where an attacker escapes the confines of a container to gain access to the underlying host. To mitigate this, technologies like Seccomp (Secure Computing), BPF (Berkeley Packet Filter), Landlock, and LSM (Linux Security Modules) are employed to filter and restrict the system calls a container can make to the kernel.

The industry often promotes the idea that "if you really care about security, you should be building custom seccomp BPFs" or running sophisticated runtime security tools. However, Manning and Beale identify two distinct modes of Kubernetes usage: "Remote Code Execution (RCE) as a Service" environments (e.g., platforms running customer-provided code) and simpler workloads (e.g., Nginx deployments). While RCE as a service environments necessitate stringent kernel-level protections, the latter, which constitutes the majority of deployments, often over-invests in these complex measures.

A significant challenge for runtime security tools, which often leverage eBPF or LSMs to monitor system calls, is the Time of Check Time of Use (TOCTOU) race condition. This vulnerability arises when a security tool checks a system call's parameters for malicious intent, but an attacker manipulates those parameters between the check and the kernel's execution. For example, a benign file open("/tmp/test") could be changed to open("/etc/shadow") by an attacker using ptrace on their own process. While mitigating TOCTOU is possible by hooking both CIS_ENTER and CIS_EXIT and comparing parameters, this approach doubles the performance overhead, introduces significant I/O, and consumes more CPU, making it impractical for most production environments. Even advanced tools like Tracee offer a "secure tracing" mode that layers an additional LSM on top of BPF programs to detect these race conditions, underscoring the inherent complexity and resource intensity of achieving robust kernel-level security.

Key Findings

▶ Watch: Kubernetes basics and talk's deep technical scope (4:00)

The talk's central findings pivot on a critical re-evaluation of Kubernetes security priorities:

• Over-prioritization of Container Breakouts: For most organizations not running RCE-as-a-service environments, container breakouts are less likely and less impactful threats compared to misconfigurations at the cluster level. The perceived "sexiness" of breakouts often overshadows more mundane but prevalent vulnerabilities.
• Custom seccomp Profiles Can Be Less Secure: Despite the common recommendation to create custom seccomp profiles for hardening, the speakers demonstrated that these custom profiles often inadvertently allow dangerous system calls (like clone or BPF capabilities) due to the complex nature of container runtimes and the sheer volume of syscalls they emit, making the default seccomp profile often safer for general use cases.
• Compliance Checklists Offer a False Sense of Security: Strict adherence to compliance guidelines, such as eliminating all image CVEs (Common Vulnerabilities and Exposures) or achieving perfect CIS Benchmark scores, can be misleading. Many CVEs in libraries are not exploitable within the context of a specific container's workload (e.g., the vulnerable function is never called). Similarly, CIS Benchmarks, while valuable, contain numerous items requiring manual verification, particularly in critical areas like RBAC and admission control, which automated tools often miss.
• Overprivileged RBAC and Misconfigurations are Primary Attack Paths: The most common and exploitable vulnerabilities in Kubernetes clusters stem from overly permissive RBAC roles, insecure configurations, and weak admission control policies. These allow attackers to perform lateral movement, escalate privileges, and gain control over the cluster without needing to exploit kernel-level vulnerabilities.
• Lack of Practical Roadmap: Organizations often struggle with a practical roadmap for applying security best practices to their clusters. Unilateral assertions of "must be CIS compliant" without clear, actionable guidance lead to security theater rather than genuine risk reduction.
• Human Bias in Risk Analysis: Humans tend to focus on the "scariest" and most interesting risks (e.g., supply chain attacks, zero-day kernel exploits) rather than the most likely and practical threats (e.g., misconfigured secrets, overprivileged service accounts). This cognitive bias leads to misallocated security resources and an ineffective defense posture.

Technical Deep Dive

▶ Watch: Demystifying containers and runtime security breakouts (4:40)

The technical aspects of the talk focused on two main areas: the complexities of container-level hardening with seccomp and the practical exploitation of cluster-level misconfigurations.

The Pitfalls of Custom Seccomp Profiles

Mark Manning detailed the challenges of creating effective custom seccomp profiles. While seccomp can restrict system calls, its implementation in a Kubernetes environment is fraught with difficulty. For instance, an organization wanted to secure an image resizing microservice using ImageMagick, a notoriously vulnerable application, by confining it with a custom seccomp profile. The idea was to allow only the necessary system calls for image processing.

Tools like strace, zaz, oci-seccomp-bpf-hook, and Inspector Gadget are commonly used to profile a running container and generate a list of its required system calls. These tools capture system calls emitted during the container's operation, aiming to produce a JSON-formatted seccomp profile that can then be managed in Kubernetes using the Security Profiles Operator (SPO). SPO treats these profiles as Kubernetes objects, integrating them into an infrastructure-as-code workflow.

However, the demonstration revealed a critical flaw: a simple "Hello ShmooCon" C program, when run directly, made only 17 system calls. The exact same program packaged in an Alpine Linux container and run in Kubernetes made 1,888 system calls. This dramatic increase is due to the underlying container runtime (e.g., containerd, runc) and its dependencies requiring many more system calls, including privileged operations like clone (used for creating new namespaces/containers), socket (for network communication), and even BPF (which can allow loading arbitrary eBPF programs, potentially leading to kernel compromise). This means that a custom seccomp profile generated from observing a seemingly simple application might inadvertently allow dangerous syscalls that are part of the container's operational overhead, effectively making the profile less secure than the default Docker/containerd seccomp profile which is maintained by experts like Jess Frazelle.

To address this, Manning developed setcomp-diff, an open-source tool that ptraces a pod, extracts its seccomp profile, and disassembles the seccomp BPF bytecode. This allows security practitioners to compare profiles, identify differences, and specifically highlight dangerous system calls that might have been accidentally allowed. The tool provides a web interface and command-line options, enabling users to analyze and validate their custom seccomp configurations.

For environments requiring true sandboxing, such as RCE-as-a-service, the speakers suggested alternative technologies like gVisor or Firecracker microVMs. gVisor, in particular, was highlighted for recent I/O performance improvements, making it a more viable option for robust container isolation. The analogy to SELinux was drawn: both seccomp and SELinux are powerful but complex, often leading to misconfigurations that can either break applications or create new vulnerabilities if not handled by experts.

Cluster-Level Attacks: The "Compliant, CVE-Free" Hack

Jay Beale then shifted focus to cluster-level attacks, demonstrating how a seemingly secure, compliant, and CVE-free Kubernetes cluster could be completely compromised. This part of the talk directly challenged the efficacy of checklist-driven security.

He critiqued common compliance practices:

• CVE Remediation: Many organizations spend excessive time remediating CVEs in image libraries that are never actually used or are not exploitable within the container's context.
• CIS Benchmarks: While valuable hardening guides, CIS Benchmarks often have manually checked items, particularly in critical areas like RBAC and admission control, which are difficult to automate and thus frequently overlooked.

The demonstration used a microk8s cluster, pre-hardened with a CI security add-on, achieving a perfect score against CIS Benchmarks and having no exploitable CVEs in its images. This "perfectly compliant" cluster was then attacked.

Demo / Proof of Concept

▶ Watch: Kernel-level security for containers: seccomp, BPF, Landlock (6:00)

Jay Beale's demonstration was a multi-stage attack against a "compliant" Kubernetes cluster, showcasing how misconfigurations in RBAC and admission control can lead to full compromise.

1. Initial Access via JupyterHub:

• The attacker starts with access to a JupyterHub instance within a "data science cluster" (simulating a compromised user or service).
• Initial attempts to list pods fail because no service account token is mounted in the current container, which is a good security practice.

1. Lateral Movement and Service Account Token Theft:

• The attacker discovers internal load balancers and services within the namespace, including a "grad student service host" and a code-server (VS Code in the browser) instance, which is password-protected.
• By listing processes (ps) inside the JupyterHub container, the attacker notices other processes, including a pause container and a NodeJS server. This indicates multiple containers within the same pod, sharing the same PID namespace.
• Exploiting the shared PID namespace, the attacker navigates to the /proc directory of another container within the pod (e.g., the code-server container) and discovers a mounted service account token.
• The token is extracted and used to set up a kubectl alias, enabling interaction with the Kubernetes API using that service account's permissions.

1. Privilege Escalation within the Namespace:

• Using the stolen token, the attacker checks permissions (kubectl auth can-i list secrets). The service account (e.g., jupyterhub-default) is found to have patch pods and get/list secrets permissions. The patch pods permission is crucial.
• The attacker discovers that a specific code-server image is explicitly forbidden by a cluster policy, yet the existing password-protected code-server instance uses a similar image.
• Leveraging the patch pods permission, the attacker patches an existing, "broken" grad-student-hub pod to use the forbidden code-server-non image. This effectively replaces a legitimate container with one controlled by the attacker.
• Inside the newly "patched" code-server container, the attacker gains a shell. Confirming the jupyterhub service account's permissions, the attacker uses get secrets to find and decode an image pull secret (e.g., Mor and Mindy), which contains credentials for the internal container registry.

1. Node Breakout via Privileged Pod:

• With registry credentials, the attacker builds a custom container image (e.g., Alpine with a reverse shell and Paradis, Jay Beale's open-source post-exploitation tool).
• The attacker then launches a privileged pod using this custom image. The pod definition includes hostPID: true (allowing access to the host's PID namespace) and privileged: true (granting full capabilities). These two settings, when combined, facilitate a container breakout.
• Upon launching the privileged pod, a reverse shell connects back to the attacker's machine.
• From the shell, the attacker uses nsenter --target 1 --all to enter the host's namespaces (targeting pid 1, typically systemd or init), effectively breaking out of the container and gaining a shell on the underlying node.

1. Cluster-Wide Compromise using Paradis:

• On the compromised node, the attacker copies the Paradis binary to a standard path (/usr/bin).
• Running Paradis on the node automatically harvests sensitive information: the kubelet certificate and secret key (allowing authentication as the node itself), all service account tokens used by pods on that node, and secrets/certificates.
• Using Paradis's kubecontrol --try-all feature, the attacker iterates through all discovered service account tokens and the kubelet credentials to find the most powerful identity.
• The attacker discovers that the metallb controller service account has permissions to create and delete admission control webhooks. This is a critical discovery for achieving persistence and cluster-wide control.

1. Persistence with Mutating Admission Controller:

• Leveraging the metallb controller's permissions, the attacker creates a mutating admission controller webhook. This involves deploying a small Python web server that acts as the webhook.
• The webhook is configured to intercept every new or updated pod deployment in the cluster.
• The malicious webhook then injects a sidecar container (e.g., named "istio" to blend in) into every incoming pod manifest.
• This sidecar could be used for various malicious activities, such as crypto mining (demonstrated as a common real-world use case by attackers, referencing the Tesla compromise), data exfiltration, or establishing persistent command and control. The sidecar often goes unnoticed in standard dashboards.

1. Cub Hound Analysis:

• The talk concluded with a brief demonstration of Cub Hound (from DataDog), a tool similar to BloodHound for Active Directory, which can map out attack paths in Kubernetes. Cub Hound successfully identified the entire attack chain demonstrated, from the initial JupyterHub pod to the creation of the mutating admission controller, highlighting the interconnectedness of seemingly minor misconfigurations.

This elaborate demo effectively proved that a cluster deemed "compliant" and "CVE-free" by automated checks and benchmarks could still be fully compromised through a series of logical steps exploiting overprivileged RBAC and weak admission control.

Defensive Implications

▶ Watch: Containers are not a security boundary (6:40)

The talk offers critical defensive implications, urging a paradigm shift in Kubernetes security:

• Prioritize RBAC Review and Least Privilege: Rigorously review and enforce the principle of least privilege for all Role-Based Access Control (RBAC) configurations. Focus on who can create pods, patch pods, read secrets, and manipulate critical cluster resources like admission control webhooks. Overprivileged service accounts are the most common entry point for escalation. Use tools to audit RBAC effectively.
• Implement Robust Admission Control Policies: This is a crucial defense layer. Implement Admission Controllers (e.g., Pod Security Standards, Kyverno, OPA Gatekeeper) to prevent the deployment of dangerous pod configurations. Specifically, block:
• Privileged containers (privileged: true).
• Containers using host PID namespaces (hostPID: true).
• Containers mounting host paths or with excessive capabilities.
• Images from unapproved registries or those known to be vulnerable.
• Reference a comprehensive list of pod configurations that lead to breakouts (as mentioned by the speakers).
• Contextualize CVE Remediation: Don't blindly chase every CVE reported in image scans. Prioritize remediation based on whether the vulnerable library or function is actually used and exploitable within the container's runtime context. Focus on minimal, hardened base images (e.g., Chainguard Wolfi images) to significantly reduce the attack surface.
• Understand and Address Practical Threats: Shift focus from "scariest" threats (e.g., Linux kernel zero-days, container breakouts for non-RCE workloads) to practical, common misconfigurations that attackers routinely exploit. This includes insecure ConfigMaps storing secrets, weak network policies, and unauthenticated dashboards.
• Invest in Continuous Education and Practice: Kubernetes is complex and evolves rapidly. Organizations must invest in ongoing training and hands-on practice for their security and development teams to truly understand the platform's attack and defense surface.
• Validate Custom seccomp Profiles Carefully: If custom seccomp profiles are deemed necessary (e.g., for RCE-as-a-service), use tools like setcomp-diff to thoroughly validate them. Ensure they are more restrictive than the default and do not inadvertently allow dangerous system calls. For most general workloads, the well-maintained default seccomp profile is often safer and more performant.
• Consider Sandboxing for High-Risk Workloads: For environments running untrusted code (RCE-as-a-service), consider strong sandboxing solutions like gVisor or Firecracker microVMs rather than relying solely on seccomp for isolation.
• Effective Risk Communication: Security teams must effectively communicate real-world risks to management, compliance teams, and board members. This involves translating technical vulnerabilities into business impact and advocating for investment in practical, high-likelihood mitigations rather than chasing theoretical "scary" risks. Do not let external checklists dictate an organization's security posture without a clear understanding of their practical applicability.

Key Takeaways

• Kubernetes security is a nuanced and continuously evolving field; there is no "silver bullet" solution.
• Overprivileged RBAC configurations and weak admission control policies are the most common and practical attack vectors in Kubernetes, not necessarily container breakouts.
• Blindly remediating all CVEs in container images or striving for perfect compliance checklist scores can be a misallocation of resources if the context of exploitability is ignored.
• Custom seccomp profiles are notoriously difficult to implement correctly and can often be less secure than the well-maintained default profiles due to the complex syscall surface of container runtimes.
• Robust admission controllers are essential for enforcing security policies at deployment time, preventing the introduction of dangerous configurations like privileged pods.
• Effective Kubernetes security requires continuous education, hands-on practice, and a pragmatic approach to risk prioritization, focusing on the most likely threats to an organization.

About the Speaker(s)

Mark Manning, also known as "anti_tree," is an experienced security professional with a background in both offensive and defensive security. He hails from Rochester, NY, and has contributed significantly to the container security space. Mark previously worked at Snowflake building RCA (Root Cause Analysis) and service-related systems, and spent a considerable time at NCC Group as a pentester, where he was instrumental in developing their container security practice. His work focuses on the practical aspects of securing complex cloud-native environments.

Jay Beale is the CTO at InGuardians, where he leads their Kubernetes and Cloud penetration testing efforts. Jay is a prominent figure in the Kubernetes security community, known for co-creating the popular Kube-CTF (Capture The Flag) at Defcon. He also trains professionals in Kubernetes attack and defense at Black Hat and is a prolific developer of open-source security tools, including Bastille Linux and Paradis, which was featured in the talk's demonstration. Jay's expertise lies in uncovering and exploiting vulnerabilities in Kubernetes clusters and developing practical defensive strategies.