What’s In Your Burrito?

What’s In Your Burrito?

Speakers: Caleb, Raunak

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=wQs1jaHV1Nw

Overview

In their engaging RSAC 2024 presentation, "What’s In Your Burrito?", Caleb and Raunak, security engineers from Chipotle, delivered a refreshing take on Kubernetes security, deliberately steering clear of the pervasive AI discussions. Their talk offered a practical, "biggest bang for your buck" approach to securing Kubernetes environments, particularly relevant for organizations with lean security teams, drawing insightful parallels between managing a large-scale restaurant chain and deploying secure cloud-native applications. The core message emphasizes that robust security isn't about exhaustive control checklists but about strategically identifying and implementing high-impact measures that address the most significant risks.

The speakers highlighted the unique challenges faced by small security teams in the quick-service restaurant (QSR) industry, where resources are limited, and the focus must remain on core business objectives. They articulated a philosophy of deep embedding with development teams, fostering collaboration over confrontation, to bake security into the software development lifecycle (SDLC) from its earliest stages. This pragmatic perspective is crucial for any organization grappling with the complexities of Kubernetes, offering a roadmap to enhance security posture without overwhelming already stretched resources.

This article delves into their recommendations, exploring how the principles of consistency, speed, and ingredient traceability in the food supply chain translate directly into securing Kubernetes clusters. From design-time considerations to runtime monitoring, Caleb and Raunak provided actionable strategies and tool recommendations for protecting critical assets, with a particular emphasis on customer data and credentials, which are paramount in industries like QSR.

Background

▶ Watch: Introduction to Kubernetes security and burrito analogy (0:00)

The landscape of Kubernetes security is notoriously complex and often overwhelming. As highlighted by the speakers, the Microsoft Kubernetes Threat Matrix, inspired by the MITRE ATT&CK framework, illustrates the sheer breadth of tactics and techniques attackers can employ, from initial access to privilege escalation. For many organizations, especially those with small security teams, attempting to map controls to every single threat can quickly lead to paralysis. This complexity is exacerbated by common misconfigurations, which remain a persistent problem. The speakers cited alarming research from Aquasec (August 2023) which found approximately 350 publicly accessible Kubernetes API servers among Fortune 500 companies, underscoring that even fundamental security practices, such as placing clusters behind a VPN or bastion host, are frequently overlooked by cloud teams prioritizing rapid deployment.

Caleb and Raunak frame Kubernetes security through an extended analogy of running a restaurant at scale. Just as a restaurant manages a food supply chain, sourcing ingredients with traceability, modern applications rely on hundreds of open-source dependencies. The need for traceability extends from farm to table in culinary terms, mirroring the need to trace software packages from their source repository, through the pipeline, to the final container image in a Kubernetes environment. Furthermore, just as culinary expectations and malicious food threats evolve, so do software vulnerabilities and security standards, requiring continuous iteration and adaptation.

Their talk focuses on securing the entire Software Development Lifecycle (SDLC), recognizing that security cannot be an afterthought applied only at deployment. They segment the SDLC into critical phases: the design phase (architecture, threat modeling, documentation), local developer systems (protecting against malicious packages), the build system (controls like EPP, image verification), and finally the Kubernetes environment itself. Within Kubernetes, they differentiate between the control plane (often opaque in managed services like AKS, EKS) and the worker nodes running workloads, acknowledging the ancillary services (key vaults, databases) that are integral to the overall security posture. This holistic view forms the foundation for their practical, high-impact security strategy.

Key Findings

▶ Watch: Understanding business risk in Kubernetes security (4:00)

Caleb and Raunak's core findings revolve around a pragmatic approach to Kubernetes security, emphasizing high-leverage controls for lean teams. They demonstrated that by focusing on a select set of foundational security practices, organizations can achieve significant risk reduction, covering a substantial portion of common attack techniques outlined in the Kubernetes Threat Matrix.

A significant finding was the revelation that implementing a focused set of basic controls can mitigate approximately 20 out of the 40 common Kubernetes threats identified in the Microsoft Threat Matrix. This highlights their "biggest bang for your buck" philosophy, proving that strategic security investments yield disproportionate benefits.

Their key contributions and findings include:

• Golden Paths and Templates: Establishing standardized, secure architectural templates and shared libraries for developers significantly reduces the burden on small security teams and ensures consistent security practices.
• Image Signing and Verification: Mandating the signing of all Docker images with X.509 certificates throughout the pipeline and enforcing verification at deploy time is a critical control against supply chain attacks and unauthorized image deployment.
• Admission Controllers as a Critical Gate: Leveraging mutating and validating admission controllers with Rego policies is paramount for enforcing security configurations (e.g., no root access, resource limits, Linux security modules) before workloads enter the cluster.
• Continuous Vulnerability Scanning of Registries: Shifting vulnerability scanning from solely pipeline-time to continuous monitoring of the container registry (ACR) prevents production deployments of images with newly discovered critical vulnerabilities.
• Runtime Monitoring with eBPF: Employing Extended Berkeley Packet Filter (eBPF) sensors (like Falco) for runtime monitoring allows for real-time detection and response to kernel-level calls and suspicious activities within containers.
• External Attack Surface Monitoring (EASM): Acknowledging that not everything can be controlled within pipelines, EASM provides continuous assessment of publicly exposed assets, catching misconfigurations that might evade internal checks.
• Foundational Importance of IAM and RBAC: Strong Identity and Access Management (IAM) and Role-Based Access Control (RBAC) policies are repeatedly stressed as fundamental, capable of mitigating 5-10 common threats alone.
• Architectural Assurance: Tools like Namespace Hound help validate network segmentation and communication patterns between namespaces, preventing unintended lateral movement.

These findings collectively advocate for a layered security approach that integrates controls across the entire SDLC, focusing on automation, policy enforcement, and continuous validation to maximize security posture with minimal overhead.

Technical Deep Dive

▶ Watch: Kubernetes threat landscape and misconfiguration examples (4:40)

The technical deep dive of Caleb and Raunak's talk systematically dissects Kubernetes security across the entire SDLC, from initial design to runtime operations, providing specific tools and methodologies.

Design Stage: Blueprinting Security

The security journey begins in the design phase. To enable their two-person team to scale, they advocate for golden paths and templates. These pre-approved architectural patterns and templated configurations (e.g., for microservices or authentication mechanisms) allow developers to build securely by default, reducing the need for constant security team oversight. They also emphasize building shared libraries for common functionalities, such as managing JSON Web Tokens (JWTs), ensuring secure implementations are readily available.

Threat modeling is introduced as a crucial early step. Rather than a rigid framework, they suggest asking fundamental questions: What are we building? How does data flow between components? What are the potential threats? What controls can be applied? This leads to defining test cases to validate security design goals later.

Documentation is also a key security asset. Beyond architecture diagrams, it's essential to scan documentation for accidentally embedded secrets. They manage security-related work through an "SSS" (Security, Scalability, and Supportability) dashboard, tracking requests and progress. An illustrative example used for the design phase was implementing multi-factor authentication (MFA) for customer accounts, which involves securing API backends, customer databases, and third-party SMS/email services.

Build Stage: Securing the Supply Chain

The build stage focuses on securing the pipeline itself and the software supply chain. First, the pipeline infrastructure (e.g., GitHub Actions, Jenkins, Azure DevOps) must be secured. Whether managed or self-managed, these environments are prime targets. They recommend Endpoint Protection Platform (EPP) controls, comprehensive audit logging, secure secrets vaulting, and strict network segmentation to limit outbound connections to only necessary deployment targets.

Within the pipeline, developers use templates that embed security tooling:

• SCA (Software Composition Analysis) and SAST (Static Application Security Testing) tools detect vulnerabilities and code weaknesses.
• End-of-life (EOL) software detection (e.g., using Xeol) identifies deprecated packages that pose security risks.
• Crucially, image signing is implemented. Every Docker image passing through their pipeline is signed with an X.509 certificate, proving its authenticity. This enables verification at deploy time, ensuring only trusted images enter the cluster.
• They advocate for standardized base images (e.g., a specific .NET image) that Chipotle's security team patches centrally, abstracting patch management from developers who might not understand the underlying Linux vulnerabilities.
• Finally, least privilege IAM is enforced for pipeline service accounts, limiting their ability to deploy only to specific namespaces or clusters.

For the open-source supply chain, they promote using the Open Source Security Foundation (OSSF) Scorecard. Developers check this scorecard for new packages, with a policy requiring a score above 8. This helps identify well-maintained and secure libraries. They also warn against typosquatting and malicious base images, citing real-world examples like the XZ Utils attack, where a malicious contributor masqueraded as legitimate.

Deploy Time: The Admission Gate

The deploy time acts as a critical security checkpoint, akin to a restaurant's final quality check. This is primarily managed by Kubernetes Admission Controllers.

• Authorization (RBAC) is the first check, ensuring the user/service account has permission to deploy.
• A mutating admission controller can make slight, predefined modifications to YAML manifests based on Rego policies (Open Policy Agent's policy language). This can, for instance, inject sidecars or adjust default settings.
• A validating admission controller enforces strict policies, failing deployments that don't comply. Policies include:
• No shell or root access within containers.
• Enforcing compute and memory limits to prevent runaway pods.
• Defining Linux security modules (LSMs) like AppArmor, SE Linux, and Seccomp to restrict kernel-level calls a container can make. For example, blocking access to ETCD even for root users.
• Assessing service account privileges and their inherited cloud permissions (e.g., access to storage accounts or databases) to build an overall risk profile for the application.

After admission control, image verification ensures that the deployed image was indeed signed by the organization's trusted pipeline. This blocks attempts to deploy unsigned or tampered images, even if a legitimate image was initially used in the pipeline.

Vulnerability management at deploy time shifts to scanning the container registry (ACR) continuously. While pipeline scans are useful, new critical vulnerabilities can emerge between build and deployment. Scanning the registry ensures that no image with a critical or high vulnerability (as per their policy) can be deployed, even if it was clean when initially built. This avoids breaking developer builds due to last-minute vulnerability discoveries while ensuring production readiness.

Runtime and Posture: Continuous Vigilance

Once deployed, runtime monitoring is crucial. Analogous to restaurant restaurateurs ensuring consistency and inventory, eBPF sensors provide deep visibility into kernel-level calls made by containers. Tools like Falco leverage eBPF to detect and react to suspicious activities, defining policies to block backdoor attacks or monitor API calls.

Continuous validation extends to:

• ACR vulnerability scanning: Maintaining a current risk profile for all images.
• External Attack Surface Monitoring (EASM): Tools like those from Project Discovery help identify publicly exposed assets (storage accounts, key vaults, databases) that might have slipped through internal controls.
• Control validation: Ensuring that all security tools (SAST, SCA) are actually running and effectively scanning all repositories, not just reporting green checks.

Finally, Kubernetes posture management ensures ongoing compliance with standards like Pod Security Standards (PSS) or CIS controls. Tools like Kubescape and MKAT (for managed K8s) assess the environment against best practices. Architecture assurance tools, such as Namespace Hound, validate that network segmentation and communication patterns between namespaces adhere to the intended secure architecture, preventing unauthorized lateral movement, especially in large clusters with 50+ namespaces. This ties back to data classification and lineage, understanding where sensitive data resides and how it flows. IAM reviews of machine accounts and RBAC policies are continuously performed to ensure least privilege.

Network policy and service mesh implementations using tools like Cilium or Calico define precise communication rules between applications within the cluster, further strengthening isolation.

Demo / Proof of Concept

▶ Watch: Tracing open source dependencies like a food supply chain (6:20)

The talk focused on conceptual frameworks, strategic approaches, and tool recommendations rather than a live demonstration or proof of concept. The speakers illustrated their points with real-world scenarios and analogies from their experience, explaining how various security controls and tools function and integrate within a robust Kubernetes security posture.

Defensive Implications

▶ Watch: Security must start early in the SDLC (7:50)

The strategies outlined by Caleb and Raunak offer a comprehensive and actionable framework for defenders, particularly those in lean security teams.

1. Prioritize High-Impact Controls: Defenders should adopt a "biggest bang for your buck" mindset, focusing on controls that cover the most significant threats rather than attempting to implement every single security recommendation. This means identifying the critical assets (e.g., customer data) and the most likely attack vectors.
2. Embed Security Early in the SDLC: Shift security left by integrating with development teams from the design phase. Establish golden paths, secure templates, and shared libraries to make it easier for developers to build secure applications by default. This reduces friction and scales security efforts.
3. Harden Pipeline Infrastructure: Recognize that the CI/CD pipeline is a primary injection point. Implement EPP controls, comprehensive audit logging, secrets vaulting, and strict network segmentation for build systems to minimize their attack surface.
4. Mandate and Verify Image Signing: Implement image signing with X.509 certificates for all first-party and third-party container images. Crucially, enforce image verification via admission controllers at deployment time to ensure only authenticated and untampered images enter the cluster.
5. Leverage Admission Controllers for Policy Enforcement: Utilize mutating and validating admission controllers with Rego policies (e.g., via Gatekeeper) to enforce critical security configurations. This includes preventing deployments with root access, limiting compute/memory resources, and defining Linux security modules (AppArmor, SE Linux, Seccomp) to restrict container capabilities.
6. Shift to Continuous Registry Scanning: Supplement pipeline vulnerability scans with continuous scanning of the container registry (ACR). This ensures that newly discovered critical vulnerabilities in existing images are identified and remediated before they can be deployed to production, preventing build breaks and maintaining security posture.
7. Implement Runtime Monitoring with eBPF: Deploy eBPF-based runtime monitoring solutions like Falco to gain deep visibility into container behavior at the kernel level. This enables real-time detection and response to anomalous activities, unauthorized process execution, and privilege escalation attempts.
8. Proactively Monitor External Attack Surface: Implement External Attack Surface Monitoring (EASM) tools (e.g., Project Discovery suite) to continuously discover and assess internet-facing assets. This helps identify misconfigurations or shadow IT that might expose sensitive data or provide initial access for attackers.
9. Strengthen IAM and RBAC: Regularly review and enforce least privilege principles for all service accounts and machine identities within Kubernetes and the broader cloud environment. Strong IAM and RBAC policies are foundational to preventing privilege escalation and unauthorized access.
10. Validate Security Tool Effectiveness: Don't just rely on green checks in CI/CD. Proactively validate that SAST and SCA tools are actually running, scanning all intended repositories, and effectively identifying issues. This ensures that the security controls are not just present but also operational.
11. Enforce Network Segmentation: Utilize network policies and service meshes (e.g., Cilium, Calico) to strictly control communication between namespaces and pods within the cluster. Tools like Namespace Hound can help visualize and validate these communication paths to prevent unintended lateral movement.
12. Build Relationships with Developers: Foster strong relationships with application development teams. This collaboration is crucial for successfully implementing security controls, gaining buy-in for golden paths, and ensuring that security is seen as an enabler, not a blocker.

Key Takeaways

• Prioritize Pragmatic Security: For lean teams, focus on high-impact controls that offer the "biggest bang for your buck," addressing the most critical threats identified in frameworks like the Microsoft Kubernetes Threat Matrix, rather than exhaustive checklist adherence.
• Secure the SDLC from Design to Runtime: Implement security controls across the entire software development lifecycle, starting with architecture and threat modeling, through pipeline hardening (image signing, SCA/SAST), admission control at deploy time, and continuous runtime monitoring (eBPF).
• Image Signing and Admission Controllers are Critical Gates: Mandate X.509 certificate-based image signing and enforce verification via mutating and validating admission controllers (e.g., Gatekeeper with Rego policies) to ensure only trusted, compliant images are deployed.
• Continuous Vulnerability Management Beyond the Pipeline: Shift vulnerability scanning to continuous monitoring of the container registry (ACR) to proactively identify and remediate critical vulnerabilities that emerge between build and deployment, preventing them from reaching production.
• Foundational Controls for Identity and Network: Strong IAM/RBAC policies for service accounts and robust network segmentation (using tools like Cilium/Calico and Namespace Hound) are essential, mitigating a significant portion of common Kubernetes attack paths.
• Foster Developer Partnerships: Building strong relationships with development teams and establishing "golden paths" for secure development is paramount for successful security adoption and scaling security efforts within an organization.

About the Speaker(s)

Caleb and Raunak (who also goes by Ron) are security engineers at Chipotle, where they are instrumental in establishing and leading the application security discipline. Their journey began as interns together at Starbucks in 2019, giving them extensive experience in the quick-service and retail industries. They have a background in both security engineering and software development, having worked at various startups before reuniting at Chipotle. They represent a lean security team, emphasizing the importance of being close partners with developers rather than solely relying on control checklists. Their philosophy focuses on identifying the most impactful security measures to achieve maximum benefit with limited resources, a principle they effectively applied to the complex domain of Kubernetes security.