Zero Trust in the Matrix: Hardening Kubernetes for the AI Frontier

Zero Trust in the Matrix: Hardening Kubernetes for the AI Frontier

Speakers: Apur, Security Engineer, Microsoft

Conference: BSides Seattle

YouTube: https://www.youtube.com/watch?v=qyup9y7ClXU

Overview

As every company races to become an AI company, the infrastructure running large language models is rapidly becoming the most attractive target on the network. In this fast-paced closing talk at BSides Seattle, Apur, a security engineer at Microsoft with research interests in Kubernetes, containers, and application security, laid out why traditional security assumptions collapse when AI workloads enter the picture.

The core thesis is simple but powerful: in traditional application security, developers control the validation layer between user input and code execution. With AI applications, the LLM sits in the middle, and prompt injection means user input can effectively become code. The talk walks through three major attack patterns observed in the wild against AI infrastructure on Kubernetes, then pivots to five concrete defensive principles and actionable remediation steps that defenders can implement immediately.

This talk is particularly relevant because the attack techniques Apur describes require no exploits at all. Attackers simply use the tools that AI developers built, asking the LLM nicely to exfiltrate credentials or harvest cloud metadata. The gap between what security teams think they deployed and what is actually running in production is where the real danger lives.

Background

▶ Watch: Threat model shift: LLM sits between user input and code execution (0:23)

The modern AI application architecture typically involves user input flowing through an API gateway to an LLM inference pod, which may communicate with a vector database and, critically, execute tools for code execution, web fetching, and file access. This tool execution layer is where the security risk concentrates. Unlike traditional applications where input validation is deterministic, LLM behavior is unpredictable by design. The threat model shift means that the validation layer security teams relied on for years no longer functions as expected.

Apur referenced real-world patterns from active research, noting that specific findings are pending responsible disclosure. The attack techniques he demonstrated, however, are drawn from production environments where AI sandboxes were found running as root, without egress filtering, and with the cloud metadata service wide open. This is not a theoretical concern; it is a pattern that is, in Apur's words, "shockingly common."

Key Findings

▶ Watch: SSRF through vision APIs — TOCTOU redirect bypass explained (4:00)

The talk identified three major attack patterns against AI infrastructure on Kubernetes:

Sandbox Escapes: AI code execution sandboxes frequently run as root because they need to install pip packages or because pods were restarted in privileged mode during debugging. Attackers craft prompts that get the LLM to execute code exfiltrating environment variables (like /etc/passwd) to a C2 server or hitting the IMDS service at 169.254.169.254 to harvest cloud credentials. No exploit is needed; the attacker just uses the built-in tools.

SSRF Through Vision APIs: Asking an AI to "describe the image at this URL" triggers a server-side fetch. If the URL points to the IMDS IP, this becomes SSRF-as-a-service. Even with blocklists, attackers use time-of-check vs. time-of-use (TOCTOU) bypasses via 307/302 redirects. The URL passes validation at request time, but the actual fetch follows a redirect to an internal IP through a different code path.

Cross-Tenant Isolation Failures: Multi-user AI platforms share infrastructure where predictable IDs, internal APIs that trust anything from internal networks, and session state without tenant context create the conditions for cross-tenant attacks. Shared services like databases, caches, and job bootstrapping code (such as E2B, a code execution interface used by AI providers) lack tenant context, making them the weakest layer.

Technical Deep Dive

▶ Watch: Five zero trust principles for AI workloads (6:44)

Apur presented side-by-side comparisons of intended vs. actual pod configurations. The textbook secure pod manifest specifies non-root execution, locked-down file systems, and denied egress. In practice, AI frameworks requiring runtime pip install, or pods restarted in privileged mode during debugging, default back to root access with unrestricted network access and a direct line to the cloud metadata server.

For SSRF defense, the recommended pattern involves parsing the URL, resolving DNS, checking the IP, and critically, revalidating after every redirect in a while loop. Each time a redirect is encountered, DNS is re-resolved and the IP is rechecked. This defeats the TOCTOU bypass. The key rule: never use allow_redirects=True blindly.

For network policy, Apur showed a Kubernetes YAML configuration that explicitly blocks the metadata service by IP, blocks all RFC1918 private ranges, and only allows HTTPS egress to approved destinations. The critical insight is the deny block: explicitly deny internal ranges rather than relying solely on allow lists.

Pod security context recommendations include: run as non-root (non-negotiable), use pre-built images with everything baked in rather than runtime package installation, read-only root file systems to prevent persistence, drop capabilities to reduce attack surface, and use seccomp profile runtime default to block dangerous syscalls.

Apur also addressed the container isolation debate: standard containers share the host kernel, meaning one kernel exploit gives host access. Kata containers or gVisor containers run workloads in lightweight VMs with guest kernels, providing hardware-enforced VM boundaries. A hypervisor escape is significantly harder to achieve than a container escape. Azure uses this approach for confidential containers, and Kata is open source, integrating well with containerd.

For IMDS hardening, the recommendations are: require IMDSv2 tokens on AWS with a hop limit of one (blocking containers from reaching it since the token request won't traverse more than one network hop), and on GCP and Azure, use workload identity so pods authenticate with their own identity rather than the node service account, eliminating the need for IMDS entirely.

Demo / Proof of Concept

▶ Watch: Kata containers vs standard containers — hardware-enforced isolation (8:00)

While the talk did not include a live exploit demonstration, Apur walked through concrete attack code showing how a crafted prompt gets the LLM to exfiltrate /etc/passwd or environment variables to a C2 server, and how a second command harvests cloud credentials from the IMDS endpoint. He also showed actual Kubernetes YAML for both the vulnerable and hardened configurations, along with Python-style pseudocode for the SSRF defense redirect validation loop. The attack patterns are drawn from active research against production environments.

Defensive Implications

▶ Watch: Monday morning action plan: audit egress, block IMDS, add Falco rules (10:45)

Apur provided a clear "Monday morning" action plan organized by urgency:

Immediate actions: Audit egress to determine what AI pods can actually reach. Block IMDS with network policy and IMDSv2. Enable workload identity. Check if sandboxes are running as root. Add Falco rules for metadata server access, DNS lookups to metadata URIs, outbound traffic to non-allowlisted IPs, and cross-namespace traffic.

Longer-term actions: Evaluate gVisor or Kata containers for code execution workloads. Consider service mesh for tenant isolation with mTLS.

For detection, Apur recommended Falco, an open-source Kubernetes-native runtime detection tool, for writing custom rules targeting these specific scenarios. On Azure, Defender for Containers has built-in alerts for metadata server access.

The five zero trust principles for AI workloads are: never trust LLM output, sandbox does not mean secure (add real isolation), egress filtering is mandatory, block IMDS period, and tenant context must flow through internal APIs.

Key Takeaways

• AI workloads require fundamentally new threat models because the LLM is untrusted and tools extend the attack surface in ways traditional applications do not
• Audit what is actually running in production, not just what was intended to be deployed; the gap between intended and actual pod configurations is where attacks succeed
• SSRF via AI is an evolution of traditional SSRF vectors; vision APIs, web fetching, and tool use are all SSRF surfaces that require revalidation after each redirect
• Multi-tenancy is genuinely hard; shared services without tenant context are the attacker's golden target, and identity-aware access must replace network-position trust
• Block IMDS with network policy, require IMDSv2, and migrate to workload identity as the correct long-term approach
• Kata/gVisor containers provide hardware-enforced isolation for untrusted AI code execution workloads at acceptable performance overhead

About the Speaker(s)

Apur is a security engineer at Microsoft with research interests in Kubernetes, containers, and application security. His research focuses on the security implications of AI infrastructure, with specific findings from active engagements pending responsible disclosure. He presented this talk as the closing session at BSides Seattle, delivering actionable defensive guidance drawn from real-world attack patterns observed against production AI deployments.