From Interview Questions to Cluster Damage: Adventures in k8s Cluster Hacking

From Interview Questions to Cluster Damage: Adventures in k8s Cluster Hacking

Speakers: Amit Serper

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=Le4K2ZlqHgg

Overview

In this insightful talk from BSides NYC, Amit Serper, a security researcher at CrowdStrike, delves into the often-overlooked security implications of Kubernetes' inherent complexity and native features. Co-prepared with Travis Low, the presentation explores how common interview questions about Kubernetes can unravel into sophisticated attack vectors, allowing adversaries to exfiltrate sensitive data and even cause widespread cluster damage using only built-in functionalities. The talk highlights that despite Kubernetes' omnipresence in modern cloud infrastructure, its "overengineered" nature and reliance on YAML configurations often lead to critical misconfigurations that security teams frequently miss.

Serper emphasizes that the talk isn't about zero-day exploits but rather about abusing the system through its intended features. He demonstrates practical techniques for disrupting core services like DNS and covertly extracting information by manipulating admission webhooks and exploiting misconfigured internal API endpoints. This perspective is crucial for anyone involved in cloud-native development, operations, or security, as it underscores the importance of deep understanding and meticulous hygiene in Kubernetes environments. The lessons learned are derived from real-world observations during interviews, where a surprising lack of fundamental Kubernetes security knowledge among candidates prompted deeper investigation into these attack patterns.

The core message is a stark warning: the very features designed to make Kubernetes powerful and flexible can become potent weapons in the hands of an attacker who understands the system's intricate interdependencies. By showcasing how seemingly innocuous missteps can cascade into catastrophic failures or silent data breaches, Serper provides a compelling case for a more rigorous approach to Kubernetes security, moving beyond superficial configurations to a truly holistic understanding of its attack surface.

Background

▶ Watch: Introduction: The 'Internet of Cubes' and YAML complexity (1:00)

Kubernetes, often described as a "series of cubes" underlying much of the modern internet, is an immensely powerful container orchestration platform. It manages the lifecycle, scaling, and networking of containerized applications across vast fleets of machines. However, this power comes at a significant cost in complexity. Serper, a veteran in low-level vulnerability research, admitted his initial apprehension, finding Kubernetes "too high level" and "so overengineered," with "so many components" and "moving parts."

The architecture of Kubernetes involves a control plane (housing components like the API server, etcd for configuration storage, the scheduler for workload placement, and the controller manager) and multiple worker nodes (running kubelet agents and container runtimes). These components, along with fundamental resources like pods (the smallest unit of compute), deployments, services, and namespaces, are deeply interdependent. A misconfiguration in one area can lead to cascading failures across the entire cluster, potentially rendering thousands of nodes useless.

Security in Kubernetes is equally complex, involving RBAC (Role-Based Access Control), service accounts, network policies, pod security contexts, and the management of secrets. A critical vulnerability highlighted by Serper is Kubernetes' default handling of secrets: they are merely Base64-encoded, not encrypted, making them easily readable if an attacker gains access. Similarly, Config Maps, used for configuration files, can also expose sensitive information if not properly secured.

A significant challenge in Kubernetes security is visibility. Serper notes the lack of a "single central log" for all activities, with audit logs covering some aspects but leaving others opaque. The distinction between the control plane and runtime environments further complicates monitoring. Moreover, the dynamic, scalable nature of Kubernetes means that traditional security approaches often fall short. Attackers can leverage insecure internet-facing services, such as a vulnerable WordPress deployment, as a beachhead to gain access to a Kubernetes node. From there, they can pivot to other parts of the network or cloud environment. Serper humorously points out that "all the attacks are relevant again," with attackers resorting to noisy, old-school network scanning (e.g., a /16 scan) within the cluster, indicating a gap in stealthier, Kubernetes-native reconnaissance methods. This backdrop of inherent complexity, default insecurities, and poor visibility forms the foundation for the attack scenarios explored in the talk.

Key Findings

▶ Watch: Talk's promise: Abusing native K8s features for data exfiltration (3:20)

The talk reveals several critical vulnerabilities stemming from Kubernetes' design and common operational practices, demonstrating how native features can be weaponized for significant impact:

1. Cascading Cluster Damage via CoreDNS Manipulation: Serper demonstrated that by gaining sufficient privileges, an attacker can effectively disable CoreDNS across an entire Kubernetes cluster. Initial attempts to simply delete CoreDNS pods are insufficient as existing connections persist and new pods can still be launched (due to kubelet resolving container images via the host's DNS). However, by patching the CoreDNS DaemonSet with a non-existent node selector label and then scaling its deployment replicas to zero, attackers can evict CoreDNS pods from all nodes and prevent new ones from spinning up. This leads to a cascading failure where new DNS queries fail, applications behave unexpectedly, and the cluster becomes largely inoperable, creating a "Fight Club"-esque scene of digital destruction.

1. Covert Data Exfiltration via Admission Webhooks: A highly potent technique involves deploying a malicious mutating admission webhook. These webhooks are designed to intercept and potentially modify requests made to the Kubernetes API server (e.g., creating pods, secrets, or config maps). An attacker with sufficient privileges can register their own webhook to intercept requests for sensitive resources. The demonstration showed a Python-based webhook intercepting a secret creation request and logging the username and password in cleartext, effectively exfiltrating data directly from the control plane before it's even persisted to etcd. This method is stealthy and leverages a core Kubernetes extension mechanism.

1. Stealthy Cluster Reconnaissance via Exposed API Endpoints: Traditional network scanning (/16 scans) within a compromised pod is noisy and inefficient. Serper revealed that many Kubernetes clusters run tools like kube-cost, Argo, or Grafana, which often expose internal metrics API endpoints. If these endpoints are not properly segmented by network policies, an attacker can make a single HTTP GET request to these internal service names (e.g., kube-cost-cost-analyzer.kubecost.svc.cluster.local). This single request can yield a comprehensive overview of the entire cluster, including all namespaces, pod names, and their corresponding internal IP addresses, providing a rich target list without generating suspicious network traffic.

These findings collectively illustrate that a deep understanding of Kubernetes' internal workings and component interactions is essential for both offense and defense, as the most effective attacks often leverage the system's own design principles against itself.

Technical Deep Dive

▶ Watch: Speaker's journey: From low-level to high-level Kubernetes skepticism (5:00)

Kubernetes' architecture, while robust, offers numerous avenues for exploitation if not meticulously secured. The talk centered on three primary technical abuses: DNS manipulation, admission webhook compromise, and API endpoint exploitation.

Kubernetes Core Components Refresher

Before diving into the attacks, Serper provided a quick refresher on Kubernetes' core components:

• Control Plane: The brain of the cluster, including the API server (the front-end for Kubernetes), etcd (a distributed key-value store for cluster state), the scheduler (assigns pods to nodes), and the controller manager (runs controller processes).
• Worker Nodes: The machines (physical, VMs, or cloud instances) that run containerized workloads. Each node runs a kubelet agent (Kubernetes agent) and a container runtime (e.g., Containerd, CRI-O).
• Pods: The smallest deployable unit, encapsulating one or more containers, storage, and network resources.
• Deployments: Define the desired state for pods, managing their creation and scaling.
• Services: An abstraction defining a logical set of pods and a policy for accessing them (e.g., stable IP address, load balancing).
• Namespaces: Provide a mechanism for isolating groups of resources within a cluster, though Serper clarifies they are "just like a drawer, an organizational thing" and do not imply network isolation by default.
• Secrets and Config Maps: Used to store sensitive data (like passwords, API keys) and configuration data, respectively. Critically, Kubernetes' default handling of secrets involves Base64 encoding, which is not encryption and can be easily decoded.

DNS Manipulation: The Reverse DevOps Jenga

The first attack targets CoreDNS, Kubernetes' internal DNS server, which resolves service names to cluster IPs using a pattern like service_name.namespace_name.svc.cluster.local. This simplifies development by abstracting IP addresses but creates a single point of failure.

1. Initial CoreDNS Disruption: An attacker might first attempt to delete the CoreDNS pod(s). Serper explains that this causes new DNS queries to fail and introduces 5-second DNS resolution timeouts, leading to application errors and slowness. However, existing connections remain unaffected, and surprisingly, new pods can still start.
2. Kubelet's DNS Role: The reason new pods start is due to the kubelet's role. When a pod is created, the kubelet on the worker node is responsible for pulling the container image. This image pull request is resolved by the host's DNS server, not the cluster's CoreDNS. Once the pod starts, its internal resolv.conf (managed by kubelet) points to CoreDNS, so internal DNS lookups will fail.
3. Achieving Total DNS Failure: To cause a complete cluster meltdown, an attacker with sufficient privileges (which Serper notes are often granted by default to WordPress-like deployments) needs to prevent CoreDNS from running anywhere. This is achieved in two steps:

• Patching the DaemonSet: CoreDNS runs as a DaemonSet in the kube-system namespace, ensuring a CoreDNS pod runs on every node. The attacker patches this DaemonSet to include a node selector for a non-existent label. For example:

This command modifies the DaemonSet's specification, telling it that CoreDNS pods should only be scheduled on nodes labeled non-existent=true. Since no nodes have this label, all existing CoreDNS pods are evicted.

• Scaling Down Replicas: While the DaemonSet change prevents new CoreDNS pods from spinning up on existing nodes, old replicas might still be managed by a separate deployment. To ensure complete eradication, the attacker also scales down the CoreDNS deployment:

This combination guarantees that no CoreDNS pods are running, leading to a "cascading failure" where applications fail, logs cease, and the cluster becomes unresponsive.

Data Exfiltration via Admission Webhooks

Admission Webhooks are a powerful Kubernetes feature that allows external services to intercept, validate, and/or modify requests to the Kubernetes API server before they are persisted.

• Validating Webhooks: Act as a "judge," allowing or denying a request.
• Mutating Webhooks: Can modify a request before it's processed.

An attacker can leverage a mutating admission webhook to intercept sensitive data. The flow is:

1. A user/application sends a request (e.g., kubectl create secret...) to the API server.
2. After authentication and authorization, the request reaches the admission webhooks.
3. A malicious mutating webhook, registered by the attacker, intercepts the request.
4. The webhook extracts sensitive information (e.g., the raw data of a secret or config map) from the request payload.
5. The webhook can then exfiltrate this data (e.g., log it to an external server or simply store it within the compromised pod's logs) before allowing the request to proceed (or denying it, if desired).

Serper demonstrated a simple Python Flask application acting as such a webhook, configured to monitor create operations on pods, secrets, and configmaps. When a secret was created, the webhook grabbed the Base64-encoded secret data, decoded it, and logged the cleartext username and password.

Stealthy Reconnaissance via Exposed API Endpoints

Instead of noisy network scans, attackers can exploit misconfigured internal API endpoints. Many Kubernetes management and observability tools (like kube-cost, Argo CD, Grafana) expose metrics or status endpoints that contain extensive information about the cluster.

If network policies are not strictly enforced, a compromised pod in one namespace might be able to access these endpoints in another namespace. By making a single HTTP GET request to a well-known internal service endpoint (e.g., http://kube-cost-cost-analyzer.kubecost.svc.cluster.local/metrics), an attacker can retrieve a wealth of information. The demonstration showed how such a request could return a formatted table detailing all namespaces, pod names, and their corresponding internal IP addresses, providing a complete map of the cluster's running components and their network locations without generating any suspicious network scan traffic. This highlights a critical oversight in many Kubernetes deployments where internal service-to-service communication is often unrestricted.

Demo / Proof of Concept

▶ Watch: K8s architecture: Over-engineering and cascading failure points (5:30)

Amit Serper provided compelling live demonstrations to illustrate each attack vector, underscoring the practical feasibility of these techniques.

1. CoreDNS Cluster Damage:

• The demo began by showing a test-pod running busybox attempting to resolve internal DNS names within the cluster, initially succeeding.
• Serper then executed the kubectl patch daemonset coredns -n kube-system --patch '{"spec":{"template":{"spec":{"nodeSelector":{"non-existent":"true"}}}}}' command. This command modified the CoreDNS DaemonSet to only schedule pods on nodes with a non-existent label, effectively evicting all running CoreDNS pods.
• Following this, he used kubectl scale deployment coredns -n kube-system --replicas=0 to ensure no old replicas could spin up.
• Re-running the DNS resolution command from the test-pod now resulted in "connection refused" errors to the CoreDNS service, demonstrating that internal DNS was completely down. This confirmed the cascading failure effect.

1. Admission Webhook Data Exfiltration:

• Serper presented a simple Python Flask application acting as a validating admission webhook server. The code was designed to listen on /v-validate and parse incoming Kubernetes API requests, specifically looking for kind: Secret resources. Upon receiving such a request, it would extract the uid and the data field (which contains the Base64-encoded secret).
• The webhook was configured to monitor create operations on pods, secrets, and configmaps.
• He then used kubectl create secret generic my-secret --from-literal=username=testuser --from-literal=password=supersecret to create a new secret in the cluster.
• Crucially, the webhook server's logs immediately displayed the intercepted request, showing the Base64-encoded username and password, which the Python script then decoded and printed in cleartext. This vividly demonstrated how an attacker-controlled webhook could silently exfiltrate sensitive credentials as they are being created or updated in the cluster.

1. API Endpoint Cluster Reconnaissance:

• Serper deployed a new pod running a simple Python script (app.py). The purpose of this script was to make a single HTTP GET request to a specific internal Kubernetes service endpoint.
• The target was a hypothetical kube-cost metrics endpoint, represented by the internal DNS name kube-cost-cost-analyzer.kubecost.svc.cluster.local.
• The app.py script fetched the data from this endpoint and formatted it into a readable table.
• Upon inspecting the logs of the deployed app.py pod (kubectl logs), a complete table was displayed, listing all namespaces, their corresponding pod names, and their internal IP addresses within the cluster. This showcased how a single, stealthy request to a misconfigured internal endpoint can provide an attacker with a full, detailed map of the cluster's topology, bypassing the need for noisy network scans.

Each demo effectively validated the theoretical attack vectors, providing tangible evidence of how Kubernetes' native features can be abused for both destructive and covert malicious activities.

Defensive Implications

▶ Watch: Deep dive into K8s dependencies and security configuration issues (6:30)

The vulnerabilities highlighted in this talk underscore the critical need for robust security practices in Kubernetes environments. Defenders must move beyond basic configurations and adopt a deep, architectural understanding to mitigate these risks.

1. Strict RBAC Hygiene: The most fundamental defense is the Principle of Least Privilege. As Serper noted, many production Kubernetes setups, including the example of a WordPress blog, often run with overly permissive service accounts that possess administrative privileges.

• Action: Audit and enforce strict RBAC policies. Ensure that workloads, especially those exposed to the internet, are granted only the absolute minimum permissions required to function. A WordPress deployment should never have permissions to manipulate DaemonSets, scale deployments, or manage admission webhooks. Regularly review ClusterRoles, Roles, ClusterRoleBindings, and RoleBindings.

1. Monitor and Control Admission Webhooks: Admission webhooks are powerful extension points, making them prime targets for abuse.

• Action: Implement continuous monitoring for the creation, modification, or deletion of validating and mutating admission webhooks. Any new or altered webhook, particularly those affecting sensitive resources like secrets, configmaps, or pods, should trigger high-priority alerts. Kubernetes security products should be leveraged to provide this detection capability.
• Action: Limit who can create or modify webhooks through RBAC. Only trusted administrators should have such permissions.
• Action: Regularly audit the configurations of existing webhooks to ensure they are legitimate and correctly configured.

1. Robust Network Policies: Serper stressed that Kubernetes namespaces alone do not provide network isolation. This makes Network Policies indispensable.

• Action: Treat Network Policies as a firewall for your Kubernetes cluster. Define explicit ingress and egress rules for every namespace and workload.
• Action: Crucially, segment access to internal metrics and management API endpoints (e.g., from kube-cost, Argo, Grafana). Only authorized services or monitoring agents should be able to access these endpoints. A compromised WordPress pod, for instance, should be entirely blocked from reaching such internal infrastructure services.
• Action: Establish clear ownership and responsibility for Network Policy management within the organization. Avoid silos where app teams deploy services without security oversight, leading to "everything is open" defaults.

1. Enhanced Visibility and Logging: Given the complexity of Kubernetes and the distributed nature of its logs, comprehensive visibility is paramount.

• Action: Aggregate and centralize Kubernetes audit logs, kubelet logs, container runtime logs, and application logs into a single security information and event management (SIEM) system.
• Action: Implement runtime security monitoring (e.g., using eBPF-based tools) to detect suspicious process execution, file access, and network activity within pods and on nodes.
• Action: Monitor for unusual DNS query patterns or failures, which could indicate attempts to disrupt CoreDNS.

1. Secure Secret Management: Kubernetes' default Base64 encoding for secrets is insufficient.

• Action: Integrate with external secret management solutions (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) that provide true encryption at rest and in transit, as well as robust access control and auditing.
• Action: Utilize External Secrets Operator or similar tools to inject secrets securely into pods from external vaults.

By diligently implementing these defensive measures, organizations can significantly harden their Kubernetes clusters against the types of sophisticated, native-feature abuses demonstrated in this talk, transforming them from potential liabilities into resilient platforms.

Key Takeaways

• Kubernetes' Complexity is a Double-Edged Sword: While powerful, the "overengineered" nature and intricate interdependencies of Kubernetes components create a vast attack surface, often leading to subtle but critical misconfigurations.
• Native Features Can Be Abused: Attackers don't always need zero-days. Core Kubernetes functionalities like DNS, admission webhooks, and internal API endpoints can be leveraged for severe cluster damage, data exfiltration, and stealthy reconnaissance.
• Base64 is Not Encryption: Kubernetes' default handling of secrets by Base64 encoding them means they are easily readable if an attacker gains access to the cluster's state. Secure secret management is non-negotiable.
• Strict RBAC and Network Policies are Paramount: Overly permissive service accounts and a lack of granular network segmentation are common and dangerous defaults. Implement the principle of least privilege and robust firewall-like Network Policies to restrict lateral movement and access to sensitive internal services.
• Monitor Control Plane Extensions: Admission webhooks are powerful and, if compromised, can silently intercept and exfiltr sensitive data. Continuous monitoring for new or modified webhooks is a critical defensive control.
• Comprehensive Visibility is Essential: Due to fragmented logging and the dynamic nature of Kubernetes, a holistic monitoring strategy aggregating all relevant logs (audit, runtime, application) is crucial to detect and respond to sophisticated attacks.

About the Speaker(s)

Amit Serper is a seasoned security researcher with over two decades of experience in the field. He describes his background as rooted in low-level vulnerability research, malware analysis, and working with tools like IDA disassemblers and shellcodes. Currently, Amit leads the security research efforts for Linux and cloud technologies at CrowdStrike. He possesses a unique perspective, having transitioned from deep system-level work to the high-level abstractions of Kubernetes, which he admits he initially found "too high level" and "overengineered."

Travis Low (co-author, but not present at the talk) is Amit Serper's co-worker and "partner in crime" at CrowdStrike, where they both work on the cloud security team. Travis is described as Amit's "Kubernetes god," providing deep expertise and answers to Amit's questions about the platform. He played a significant role in preparing the content and research for this talk.