Kubernetes Security: Attacking and Defending Modern Infrastructure

Kubernetes Security: Attacking and Defending Modern Infrastructure

Speakers: Lenki Aley, Security Engineer, Google; Max von Lichtenberg, Security Researcher

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=iCzUv6-9zrI

Overview

In an era where containerization and orchestration have become foundational to modern application deployment, understanding the intricate security landscape of Kubernetes is paramount. This talk, delivered by Lenki Aley, a security engineer at Google, and Max von Lichtenberg, a security researcher, delves deep into the multifaceted challenges and solutions surrounding Kubernetes security. Presented at RSAC 2024, the session provides a comprehensive exploration of both offensive techniques employed by adversaries and the robust defensive strategies that organizations must implement to safeguard their containerized infrastructure.

The speakers meticulously break down the evolution of application deployment, from traditional bare-metal servers to virtual machines and finally to the ubiquitous container architecture, highlighting the inherent benefits and the new security paradigms each shift introduces. They then proceed to dissect the core components of Kubernetes, establishing a clear understanding of its architecture before detailing the most common attack vectors and the corresponding defensive measures. This talk is crucial for security professionals, developers, and infrastructure engineers who are grappling with the complexities of securing highly dynamic and distributed Kubernetes environments.

The importance of this presentation cannot be overstated. As Kubernetes continues to gain traction, becoming the "operating system of the internet" for many, its attack surface expands, attracting sophisticated adversaries. Aley and von Lichtenberg not only enumerate the specific tactics, techniques, and procedures (TTPs) observed in the wild but also equip the audience with actionable insights into building resilient defenses. Their discussion covers everything from foundational security principles like least privilege and network segmentation to advanced topics such as admission controllers and GitOps methodologies, making it an indispensable guide for anyone responsible for Kubernetes security.

Background

▶ Watch: Talk introduction and agenda overview (0:00)

The journey to modern application deployment has been marked by a continuous quest for efficiency, scalability, and isolation. Initially, applications ran directly on dedicated machines, offering straightforward process isolation but proving to be cost-ineffective due to underutilized resources. The advent of virtual machine (VM) architecture introduced the hypervisor, enabling multiple isolated operating systems to run on a single physical host. While a significant improvement in resource utilization, VMs remained relatively heavy and cumbersome to move.

The next evolutionary leap brought forth the container architecture, leveraging a container runtime (like Docker) to package applications with all their dependencies, configurations, and source code into lightweight, portable units called containers. These containers utilize clever operating system virtualization features, primarily namespaces, cgroups (control groups), and capabilities, to achieve a degree of isolation. Namespaces control what a process can see (e.g., process IDs, network interfaces), cgroups manage resource allocation (CPU, memory, network I/O), and capabilities fine-tune process privileges. Crucially, containers are not full virtual machines but rather isolated processes sharing the host kernel, which presents a unique set of security considerations.

Managing a few containers is manageable, but orchestrating hundreds or thousands across a distributed system demands a powerful framework. This is where Kubernetes (K8s) enters the picture, described by the speakers as the "operating system of the internet" due to its pervasive role in modern infrastructure. Kubernetes is a robust, open-source platform designed for automating the deployment, scaling, and management of containerized applications.

To understand Kubernetes security, one must first grasp its fundamental concepts and architecture:

• Services: The way Kubernetes exposes applications, typically via public IP addresses and ports.
• Pods: The smallest deployable units in Kubernetes, encapsulating one or more containers that share network and optionally file system namespaces.
• Nodes: The virtual or physical machines that host and run pods.
• Cluster: A collection of nodes that collectively run a Kubernetes application.
• Deployment: A configuration that defines how pods are allocated and managed across the cluster.

A Kubernetes cluster is composed of several key components:

• Control Plane Components (Master Nodes):
• kube-apiserver: The main control plane for the cluster, exposing the Kubernetes API. All communication with the cluster goes through this component.
• etcd: A highly available key-value store used for storing all cluster data, configuration maps, and secrets.
• kube-scheduler: Responsible for scheduling newly created pods onto available nodes.
• kube-controller-manager: Monitors and manages pods, ensuring they run correctly according to their deployments.
• Node Components (Worker Nodes):
• kube-proxy: Manages network traffic between the node and services, facilitating communication.
• kubelet: The agent that runs on each node, communicating with the API server and ensuring containers are running in a pod.
• Container Runtime: Software like Docker or containerd that runs containers on the node.

This intricate architecture gives rise to a distinct Kubernetes threat model. An attacker gaining control of a container can attempt to escape to its pod, then to the node, and subsequently escalate privileges to compromise other worker nodes or even the master nodes, ultimately gaining control over the entire cluster. This multi-layered environment presents a complex challenge for security, as highlighted by the existence of the OWASP Kubernetes Top 10 list, which enumerates the most critical security risks in Kubernetes environments.

Key Findings

▶ Watch: Kubernetes explained: orchestration and key terms (2:00)

The talk's core findings revolve around a comprehensive breakdown of attack techniques and corresponding defensive strategies, framed within the context of a specialized threat matrix for container environments. Max von Lichtenberg introduced this Microsoft Threat Matrix for Container Environments, which in 2021 consolidated tactics and procedures affecting container environments. This matrix, while similar to traditional Linux threat matrices, accounts for the unique characteristics of containerized deployments, for instance, initial access shifting from a machine to a cluster, and malicious code execution targeting containers or the cluster rather than just a machine. This matrix was updated in 2022 with contributions from independent researchers, reflecting observed real-world attack patterns.

The speakers meticulously detailed common attack techniques across various phases of the kill chain:

• Initial Access: Emphasizing how attackers first gain a foothold, often not by directly targeting Kubernetes but through broader cloud attack vectors.
• Execution: Describing methods for running malicious code within the cluster.
• Persistence: Outlining techniques adversaries use to maintain their presence after an initial compromise.
• Privilege Escalation: Detailing how attackers expand their access from a compromised component to higher-privileged parts of the cluster.
• Defense Evasion: Covering strategies to avoid detection and hide malicious activities.

On the defensive side, the talk highlighted a layered approach, introducing specific Kubernetes-native security controls and best practices:

• Network Policies: For micro-segmentation and restricting inter-pod communication.
• Encryption: Ensuring data is protected both in transit and at rest.
• Access Control: Implementing the principle of least privilege through mechanisms like Pod Security Policies (PSPs) (for older clusters) and Pod Security Admission (PSA) (modern approach).
• Audit Logging: For comprehensive visibility into cluster activities.
• Hardening Kubernetes Components: Securing the core infrastructure itself.

A significant finding was the power of privileged containers as a privilege escalation vector, demonstrated by a simple command that grants full host access. Conversely, the talk stressed the importance of robust access control mechanisms, specifically the evolution from PSPs to PSAs, as critical for preventing such attacks. The presentation concluded with a concise "10 things to think about" checklist, synthesizing the key defensive findings into actionable advice for anyone deploying or managing Kubernetes clusters, serving as a practical guide for enhancing security posture.

Technical Deep Dive

▶ Watch: Deep dive into Kubernetes architecture components (3:40)

The technical deep dive commenced with a detailed exploration of attack techniques, categorized by their phase in the kill chain, followed by an equally thorough examination of defensive measures.

Initial Access:

Attackers seek to establish a foothold in the cluster.

• Cloud Credentials: Traditional phishing techniques are still highly effective in stealing cloud provider credentials, granting access to management UIs or consoles, which then serve as an entry point to Kubernetes. MFA bombing can bypass multi-factor authentication.
• Compromised Registry: Attackers can push malicious images or dependencies into container registries, leading to supply chain attacks. The speakers mentioned an "X-axis vulnerability" in this context, implying a specific type of vulnerability exploited in this scenario.
• kubeconfig Files: These files contain credentials for connecting to a cluster. Simple GitHub dorking (e.g., searching for filename:kubeconfig) can expose these sensitive files, allowing attackers with developer machine access to inherit their privileges.
• Application Vulnerabilities: Since web applications run within containers, common vulnerabilities like Remote Code Execution (RCE) or SQL Injection (often covered by the OWASP Top 10) can provide an entry point.
• Exposed Sensitive Interfaces: Historically, the Kube-UI was often exposed to the internet. Today, other management UIs or dashboards for Kubernetes-related projects can be exposed with default or weak credentials, offering easy access.

Execution:

Once inside, attackers aim to run their code.

• Application Exploit: Exploiting a vulnerable application within a container to execute arbitrary code.
• kubectl exec: If an attacker gains access to a Kubernetes account with sufficient permissions, they can use kubectl exec to run commands inside a running container.
• Running New Containers: Attackers can deploy their own malicious containers, often seen with crypto miners. If a container is a scratch container (minimal base image), kubectl exec might not be viable, necessitating a new container deployment.
• Sidecar Injection: A stealthier method involves deploying a sidecar container within an existing pod. Since pods can have multiple containers, a malicious sidecar can hide among legitimate ones, making detection harder.

Persistence:

Maintaining access after the initial compromise is critical.

• Backdoor Container: Simply deploying a malicious container designed to run continuously as a backdoor.
• Kubernetes Cron Job: Utilizing Kubernetes' equivalent of a Linux cron job, attackers can schedule malicious containers to run at specific times, making them harder to detect as they spin up and down.
• Writable HostPath Mount: If an attacker can create a new container with a writable hostPath mount, they gain write access to the underlying node's file system. This allows them to modify crucial files like authorized_keys for SSH access or add new cron jobs to the host machine.
• Malicious Admission Controller: An admission controller intercepts requests to the Kubernetes API server before they are persisted. If an attacker compromises an admission controller, they can inject malicious containers into legitimate workloads, ensuring their code is deployed as part of valid operations.

Privilege Escalation:

Moving laterally or vertically to gain higher privileges.

• Compromising Additional Cloud Resources: A compromised container might have access to cloud resources like S3 buckets or identity provider credentials, which can be leveraged for further attacks.
• Privileged Containers: A potent technique involves creating a privileged container. The speakers demonstrated this with a "280 character tweet" command that effectively creates a process with full access to the host's PID namespace, network namespace, and all files in the system. This grants the attacker control equivalent to having root on the host machine.
• Compromising cluster-admin Role: The cluster-admin role is the equivalent of a super user in Kubernetes. If an attacker compromises a container or service account with this role, they can deploy any malicious workload, read all secrets from etcd, or essentially control the entire cluster.

Defense Evasion:

Tactics to avoid detection.

• Clearing Container Logs: Deleting application logs (e.g., Nginx logs) generated by the container.
• Deleting Kubernetes Events: Kubernetes tracks all cluster changes as events. Attackers can use kubectl delete events --all to erase these records and hide their activity.
• Typosquatting: Naming malicious pods or containers similarly to legitimate ones to blend in and evade detection by human operators.
• Connecting from a Proxy Server: Launching attacks from within the Kubernetes network or via a VPN/proxy to obscure their origin.

Defensive Measures:

Lenki and Max then transitioned to comprehensive defensive strategies:

• Network Policies: These are powerful tools for network segmentation. By default, pods have no network restrictions. Network policies, implemented by the CNI plugin (e.g., Flannel, Calico, Istio), allow administrators to define precise rules for ingress traffic (who can talk to a pod) and egress traffic (who a pod can talk to), based on namespaces, labels, or IP ranges.
• Encryption:
• Encryption in Transit: Kubernetes network traffic is not encrypted by default. This can be achieved by generating TLS certificates using certificates.k8s.io or external tools like cert-manager.io. Some CNI plugins also offer this natively.
• Encryption at Rest: For secrets, config maps, and other data stored in etcd, encryption configuration can be enabled, supporting algorithms like AES-CBC and AES-GCM. For stronger security, an External Key Management System (KMS) should be integrated. Note that this doesn't encrypt file systems; for that, application-level encryption or storage integration with encrypted volumes is required.
• Access Control: Adhering to the principle of least privilege is critical.
• Pod Security Policies (PSPs): An older, namespace-scoped or cluster-scoped mechanism using roles and role bindings (or cluster roles and cluster role bindings) to enforce security configurations (e.g., privileged: false) on pods. This prevents the deployment of privileged containers.
• Pod Security Admission (PSA): The modern approach, acting as a "gatekeeper" or checkpoint. Before any request (like creating a pod) is fully processed, it must pass through admission controllers. Tools like OPA Gatekeeper and Kyverno provide flexible policy enforcement.
• Audit Logs: Kubernetes offers a flexible audit logging mechanism that, when enabled, logs every operation in the cluster. It can be tuned to log specific operations (e.g., creating pods, services) and output to the file system or a SIEM solution. However, enabling audit logs consumes additional CPU and memory resources.
• Hardening Kubernetes Components:
• Isolating Control Plane: Don't expose Kubernetes components directly to the internet. Use an ingress controller and separate network interfaces for public vs. internal management traffic.
• Protecting Kubelet Process: Disable anonymous authentication and enable TLS authentication for the Kubelet API.
• Protecting etcd: Put a firewall in front of etcd, enable encryption at rest, and use a KMS.
• Third-Party Authentication for API Server: Integrate OIDC (Open ID Connect) or build a custom proxy for authentication to the API server.

The speakers also touched upon the emerging challenges of deploying Kubernetes at the edge (e.g., KubeEdge, K3s, MicroK8s, MiniKube), where resource constraints and diverse environments amplify security concerns.

Demo / Proof of Concept

▶ Watch: Introduction to Kubernetes common attack techniques (5:25)

While the talk did not feature a live, interactive demonstration of an attack or defense, it effectively presented a powerful conceptual proof of concept for privilege escalation using privileged containers. Lenki Aley highlighted a "280 character tweet" command, which, when executed with sufficient cluster credentials, could deploy a privileged container.

This command serves as a stark illustration of how a simple Kubernetes deployment manifest can grant a container full access to the host machine's PID namespace, network namespace, and file system. By running such a container, an attacker gains capabilities equivalent to having root access on the underlying node, allowing them to manipulate host files, install backdoors, or pivot to other systems. This example, though presented conceptually, powerfully conveyed the gravity of misconfigurations or compromised credentials in a Kubernetes environment and underscored the immediate need for robust access control and security policies like Pod Security Policies or Pod Security Admission to prevent such deployments.

Defensive Implications

▶ Watch: Initial access: cloud credentials, compromise registry, kubeconfig (6:05)

For defenders, the insights from this talk translate into a multi-layered, proactive security strategy for Kubernetes environments. The primary implication is the necessity of moving beyond traditional host-based security to embrace Kubernetes-native controls and a holistic approach that covers the entire software supply chain and runtime environment.

Key defensive actions include:

1. Secure the Supply Chain: Do not deploy images from untrusted sources. Regularly scan container images for vulnerabilities using tools like Socket or DeepFactor. This addresses the "Compromised Registry" and "Application Vulnerabilities" attack vectors.
2. Protect Sensitive Data: Avoid storing sensitive data directly in Kubernetes secrets. Instead, leverage external Key Management Systems (KMS) or utilize Kubernetes' encryption configuration for secrets at rest.
3. Encrypt All Communications: Implement encryption in transit for all network traffic within the cluster using TLS certificates (via certificates.k8s.io or cert-manager.io) and ensure encryption at rest for all persistent data.
4. Enforce Security Contexts and Resource Limits: Prevent crypto miners and other resource-intensive malicious workloads by enforcing security contexts that restrict container privileges and setting resource consumption limits (CPU, memory) on all workloads.
5. Implement Network Segmentation: Utilize Network Policies with a CNI plugin that supports them (Flannel, Calico, Istio) to strictly control ingress and egress communication between pods and services. This is crucial for limiting lateral movement.
6. Robust Access Control: Always enable and meticulously configure access control. For older clusters, implement Pod Security Policies (PSPs) to prevent privileged containers. For modern deployments, adopt Pod Security Admission (PSA) with policy engines like OPA Gatekeeper or Kyverno. Crucially, always adhere to the principle of least privilege and avoid using the powerful cluster-admin role for regular operations.
7. Comprehensive Audit Logging and Monitoring: Enable and configure Kubernetes audit logs to monitor for suspicious activity, unusual API calls, or unauthorized changes. Integrate these logs with a SIEM solution for centralized analysis and alerting.
8. Keep Kubernetes Up-to-Date: Regularly patch and update Kubernetes versions to benefit from the latest security fixes and features.
9. Harden Control Plane Components: Isolate Kubernetes control plane components from direct internet exposure using ingress controllers and network segmentation. Protect the kubelet process by disabling anonymous authentication and enabling TLS. Secure etcd with firewalls, encryption at rest, and KMS integration. Implement third-party authentication (e.g., OIDC) for the API server.
10. Adopt GitOps Methodology: Implement a GitOps practice for managing the cluster configuration. This methodology ensures all changes go through a version-controlled, auditable, and peer-reviewed process, preventing unilateral, potentially malicious, or misconfigured deployments.

Defenders should also regularly audit their clusters using tools like Kube-bench (which checks against CIS benchmarks) and Net-scout (for cloud security audits with Kubernetes-specific components) to identify misconfigurations and vulnerabilities proactively. The complexity of Kubernetes demands a continuous security posture management approach, integrating these defensive strategies into the entire DevOps lifecycle.

Key Takeaways

• Kubernetes introduces a complex, multi-layered threat model: Attackers can escalate privileges from a compromised container to a pod, then to a node, and ultimately to the entire cluster, necessitating a holistic security approach.
• Preventing privileged containers is paramount: Misconfigurations or compromised credentials can allow deployment of privileged containers that grant full host access, making robust Pod Security Policies (PSPs) or Pod Security Admission (PSA) essential.
• Network segmentation via Network Policies is critical: By default, pods have unrestricted communication; Network Policies are vital for micro-segmentation, limiting lateral movement of attackers within the cluster.
• Secure the entire software supply chain: Regularly scan container images for vulnerabilities, avoid untrusted sources, and protect sensitive data using external KMS rather than native Kubernetes secrets.
• Comprehensive visibility and control are non-negotiable: Implement detailed audit logging for all API activities and enforce the principle of least privilege through strict access control and GitOps practices to prevent unauthorized changes and detect malicious actions.

About the Speaker(s)

Lenki Aley is a Security Engineer at Google, bringing extensive expertise in securing complex cloud-native environments. His professional background at one of the world's leading technology companies provides him with deep insights into cutting-edge security challenges and solutions in the Kubernetes ecosystem. Beyond his corporate role, Lenki is also a dedicated volunteer at the Pacific Hacker Association, demonstrating his commitment to community engagement and knowledge sharing within the security domain.

Max von Lichtenberg is a Security Researcher, actively contributing to the understanding and mitigation of emerging threats. His work involves probing systems for vulnerabilities and developing defensive strategies. Like Lenki, Max is also a volunteer at the Pacific Hacker Association, highlighting a shared passion for cybersecurity and a commitment to fostering a secure technological landscape through research and education. Together, their combined experience provides a well-rounded perspective on both the offensive and defensive aspects of Kubernetes security.