Unprivileged Containers: Shaving Yaks To Get the Toothpaste Back In the Tube

Unprivileged Containers: Shaving Yaks To Get the Toothpaste Back In the Tube

Speakers: Matt Carroll, Tech Lead, Incident Detection and Response Team, Yelp

Conference: 44CON

YouTube: https://www.youtube.com/watch?v=mnNrraXYJx4

Overview

Matt Carroll's 44CON talk, "Unprivileged Containers: Shaving Yaks To Get the Toothpaste Back In the Tube," delves into the arduous journey Yelp undertook to secure its containerized development environment. The presentation is a candid recounting of a year-and-a-half-long project aimed at resolving a critical security vulnerability: the inherent root access granted by Docker's default operational model in a multi-tenant development setup. Carroll uses two memorable idioms to frame the challenge: "getting the toothpaste back in the tube," representing an incredibly difficult-to-reverse action, and "shaving yaks," symbolizing seemingly unrelated, yet dependent, tasks necessary to achieve a primary goal.

This talk is crucial for any organization utilizing Docker or similar container technologies, especially in shared development or multi-tenant environments. It highlights a fundamental security flaw that, if unaddressed, allows any developer to escalate privileges to root on a shared machine, bypassing audit controls. Carroll, a tech lead on Yelp's incident detection and response team, meticulously details the theoretical solutions, the profound technical obstacles encountered during implementation, and the innovative, sometimes unconventional, methods employed to overcome them. His narrative serves as a vital case study, emphasizing that generic security advice often glosses over the "love crafty and hellscape" of real-world implementation, where theory meets practice in a "big old war of attrition."

The significance of this talk extends beyond container security; it's a testament to the complexities of modern infrastructure, the trade-offs between security and developer velocity, and the necessity of deep technical understanding to remediate seemingly simple problems. Carroll's honest appraisal of what worked, what didn't, and what he would do differently provides invaluable lessons for security professionals and engineers alike. It underscores the importance of context-specific security solutions, leveraging business models and workflows, and challenging the orthodoxy when faced with intractable technical challenges.

Background

▶ Watch: Introduction to talk title and speaker's role at Yelp (0:00)

Yelp's journey with containers began early, long before Docker was considered production-ready. The company transitioned from a monolith to statically deployed microservices and eventually developed its own Platform as a Service (PaaS) called "Pasta." This platform was built on Mesos, Marathon, and Docker, a stack that predates the widespread adoption of Kubernetes. Pasta aimed to provide a streamlined environment for developers primarily building HTTP services, offering associated tooling like yelp compose (inspired by Docker Compose) to abstract away common tasks such as network instantiation, dependency management, and running acceptance tests. This approach significantly boosted developer velocity by allowing engineers to "ship their machine to production."

The core problem, however, emerged from Docker's default operational model in Yelp's shared development infrastructure. Multiple developers shared "dev boxes," creating a multi-tenant environment. As early as 2014, just five months after Matt Carroll joined Yelp, an internal ticket was opened highlighting a critical security flaw: "anybody can root a Dev box now." The issue stems from Docker's daemon (dockerd) running as root by default. The Docker CLI communicates with dockerd over a Unix domain socket. If a user has access to this socket, they effectively have root privileges on the host machine.

Carroll vividly demonstrated this vulnerability with a live demo. By running a simple Docker command that mounts the host's home directory, copies /bin/bash into it, and then applies the SUID (Set User ID) bit (chmod +s), a user can execute that copied bash binary as root outside the container. This is a direct violation of the principle of least privilege, as it grants unconstrained root access without any meaningful audit trail.

The technical roots of this problem lie in Docker's design choices. Docker gained market dominance by prioritizing ease of development and rapid iteration. This often meant abstracting away complex Linux containerization primitives (like namespaces and cgroups) and running its core components with maximum privileges. The entire Docker stack—dockerd, containerd, and containerd-shim—runs as root. Furthermore, Docker lacks coherent local authorization (AuthZ) and useful logging. While mTLS can be configured for authentication to the Docker socket, it's often cumbersome for local Unix domain sockets, and Docker doesn't map authenticated users to specific containers, leading to no tracking of who launched what. Existing AuthZ plugins are limited, often acting like a "game of whack-a-mole" against path traversal or mount options. Crucially, due to socket communication, immutable login UID information is lost, making auditing impossible. In essence, "access to the Docker socket is root without audit."

Key Findings

▶ Watch: Overview of the talk's agenda and main sections (3:50)

The central finding of Carroll's work is the confirmation that Docker's default privilege model inherently creates a significant security vulnerability in shared development environments, granting unprivileged users effective root access without audit. This realization catalyzed a complex, year-and-a-half-long project to "get the toothpaste back in the tube."

To address this, the theoretical solution converged on Rootless Docker, specifically leveraging unprivileged username namespaces. This kernel feature allows the remapping of UIDs (User IDs) within a container, such that a root user inside the container maps to an unprivileged user outside. The key benefit is that any container escape would only lead to privilege escalation within that user's specific namespace, not to root on the host system.

However, the theoretical solution quickly ran into practical "yak shaving" challenges. A critical discovery was the selfishness of dockerd, being fundamentally single-tenant. Running a separate dockerd instance per user for rootless operation meant an N-fold duplication of all base images, leading to prohibitive disk costs on shared dev boxes. This problem was elegantly solved by adopting Podman, an alternative OCI-compliant container runtime. Podman's crucial "additional image stores" feature allows for read-only, shared base image layers, eliminating the duplication issue while still allowing users to build on top of them with copy-on-write layers.

Another significant finding was the need for ID mapped mounts to transparently translate file UIDs. When a system user pulls base images into the shared store, they retain the original root ownership outside the user's namespace. Simply changing permissions (chmod or chown) would break the crucial development-to-production parity. ID mapped mounts, a relatively new kernel feature, provided the ability to bind mount these shared base images and dynamically remap their UIDs to match the user's unprivileged sub-UIDs within their namespace, without altering the underlying files. This feature, however, required Linux kernel 5.19 or newer for overlay2 filesystem support, which serendipitously became available in Ubuntu's Luna Lobster (kernel 6.2) and was backported to Jammy LTS, proving to be a timely and indispensable enabler for the project.

Finally, the project highlighted that even with a robust technical solution, implementation is a "love crafty and hellscape." Challenges ranged from minor API incompatibilities with Podman's Docker compatibility layer (e.g., slow dpkg queries) to intricate networking complexities requiring custom, mixed-privileged systemd units and scripts, and even obscure kernel page size limits affecting mount options. The journey underscored that "talk is cheap" and a deeper understanding of underlying mechanisms is paramount to successful security remediation.

Technical Deep Dive

▶ Watch: Disclaimer: This is a story about tradeoffs, not direct advice (4:40)

The core of Yelp's solution revolved around transitioning from a privileged Docker daemon to a rootless container environment for developers, primarily using Podman. This required a multi-faceted approach, addressing storage, networking, and compatibility issues.

The theoretical foundation for rootless containers relies on unprivileged username namespaces. Linux containers are built upon various namespaces (network, mount, IPC, etc.). Username namespaces allow for the remapping of UIDs and GIDs (Group IDs) within a container. For instance, uid 0 (root) inside a container can be mapped to an unprivileged UID (e.g., 1337) on the host. This mapping is defined in files like /etc/subuid and /etc/subgid, which allocate a range of high-numbered, contiguous UIDs and GIDs to a specific user for use within their namespaces. The critical security benefit is that if a container escape occurs, the attacker only gains privileges as the unprivileged host user, not as root.

However, moving to a per-user rootless Docker daemon (or Podman instance) introduced a significant challenge: disk space. Docker's storage backend, typically using overlayFS, is single-tenant. This meant that if 10-20 developers shared a dev box, each running their own dockerd, every commonly used base image would be duplicated multiple times. Given Yelp's use of "fat base images" and the practice of pre-pulling them, this amounted to a projected cost of "hundreds of thousands of dollars per year" for additional EBS storage, with no guarantee of accuracy or future cost stability.

The solution to the disk space problem came from Podman and its additional image stores feature. Podman, being OCI-compliant, produces build artifacts compatible with Docker. Crucially, Podman can run in a daemonless mode, where the podman run command is the main program, launching containers as child processes, which helps with login UID tracking. The additional image stores feature allows for defining read-only image repositories that multiple Podman instances can share. Images pulled or built by users can still be based on these read-only images, leveraging copy-on-write for their specific layers.

The architecture for image storage was designed as follows:

1. A dedicated unprivileged system user (the ubuntu user in Yelp's case) pulls all common base images into a shared, central location.
2. These base images are then mounted read-only into /var/ucs/uid/base (with ucs being an abbreviated, shorter path to avoid later issues, as discussed below).
3. Each user's read-write image layers are stored in /var/ucs/uid/per_user.
4. A custom Go program was developed to clean up base images, walking the image graph to ensure dependent images are not prematurely removed.

A subsequent problem arose: even with shared read-only image stores, the base images pulled by the ubuntu system user would retain their original UIDs and GIDs (e.g., root ownership for /etc/shadow) outside the user's namespace. This meant that an unprivileged user's Podman instance, even with UID remapping, might not be able to read or modify these files correctly, as their remapped root might not match the true root of the base image. The critical constraint was that chmod or chown could not be used, as this would create a disparity between the development artifact and the production deployment, leading to "nothing will work properly" in production.

This led to the discovery of ID mapped mounts. This kernel feature allows for a bind mount to specify UID/GID mappings, effectively translating file ownership transparently within the mount namespace. For example, a file owned by root on the host could be presented as owned by the user's remapped root (their sub-UID 0) inside their container environment. This was a perfect fit, allowing the read-only base images to be correctly permissioned for each user without modification. A significant hurdle was that ID mapped mounts were only supported as a lower layer for overlay2 (the common Docker/Podman storage driver) starting with Linux kernel 5.19. Luckily, Ubuntu's Luna Lobster (6.2 kernel) and its backport to Jammy LTS provided this crucial functionality. The feature was so new that Yelp had to package a kernel developer's example program, as mount options weren't yet integrated into the standard mount command.

The implementation faced a peculiar issue related to Linux kernel page size: mount options for overlay2 had to fit within a 4KB kernel page. Yelp's initial, overly descriptive path names for image layers, combined with the number of layers, caused the mount options string to exceed this limit. This resulted in consistent crashes at the 44th image layer for their monolithic base image. The fix was to shorten the path prefix for image layers (e.g., from var/unprivileged-container-storage/uid/... to var/ucs/uid/...), which increased the limit to 72 layers, resolving the issue.

Networking also presented challenges. Podman's default slirp4netns for single containers works well, but for a multi-container yelp compose network, more sophisticated routing was needed. The solution involved a mixed-privileged approach:

1. A privileged systemd daemon would set up a yelp_compose network for each user upon login.
2. A "wedge" container was used to keep the network namespace open, as Podman would otherwise close it if no containers were running.
3. Inside this network namespace, a process would insert one end of a veth pair, set up default routing, and perform NATting and routing on the host. This provided a functional yelp_compose network that "just worked" for the 80% use case.

Finally, integrating Podman via its compatibility API (which mimics the Docker API) wasn't seamless. It required backporting various patches from upstream to fix bugs. A humorous example was Podman's method of querying package versions: it would hit dpkg, which is notoriously slow, causing timeouts and performance issues. The solution was a custom, cached dpkg wrapper that splatted package versions to a text file, allowing Podman to read from a local cache instead of repeatedly querying dpkg.

To manage the rollout and debug issues, Yelp developed:

• doc_proc: A TCP proxy for the privileged Docker socket that logged metadata (command, CWD, login UID) about calling processes. This allowed auditing and eventually served as an AuthZ decision point (forwarding to Open Policy Agent) to restrict privileged Docker usage.
• pod_doctor: A diagnostic script that collected common information, posted it to Slack, and eventually incorporated autofixes for recurring issues. This served as a living "laundry list" of priorities for the team.

The entire system was orchestrated via systemd units, which, despite some "regrets" about its complexity for dependency management, handled the setup of ID mapped mounts, networking, and user lingering (a systemd feature to keep user-level processes alive after logout, also used to launch user-level units on boot).

Demo / Proof of Concept

▶ Watch: Yelp's platform evolution: microservices, Pasta, early Docker adoption (5:10)

Matt Carroll's presentation included two live demonstrations that effectively illustrated both the problem and the solution.

The first demo showcased the critical security vulnerability inherent in Docker's default, privileged operation on a shared development machine. Carroll executed a series of commands:

1. echo DOCKER_HOST: To confirm the default Docker socket was being used, indicating a privileged Docker daemon.
2. docker run -v $HOME:$HOME ubuntu:latest bash -c "cp /bin/bash $HOME/pwned_bash && chmod +s $HOME/pwned_bash": This command launched an Ubuntu container, mounted the user's home directory as a volume, copied /bin/bash to a new file named pwned_bash within the home directory, and then applied the SUID bit (+s) to it. The SUID bit ensures that when pwned_bash is executed, it runs with the permissions of its owner, which in this case is root because Docker runs as root.
3. $HOME/pwned_bash -c "whoami": Executing the SUID-bit-enabled pwned_bash directly from the host's home directory.

The immediate output was root. This clearly and chillingly demonstrated how easily any developer with access to the Docker socket could gain unfettered root access on the shared dev box, highlighting the severity of the problem Yelp was trying to solve.

The second demo, conducted after describing the implementation of the unprivileged container solution, mirrored the first to prove the remediation. Carroll again executed the same sequence of commands:

1. export DOCKER_HOST=/run/user/$UID/podman/podman.sock: This crucial step redirected the Docker CLI to communicate with the user's specific Podman socket, indicating that the unprivileged, per-user Podman instance was in use.
2. docker run -v $HOME:$HOME ubuntu:latest bash -c "cp /bin/bash $HOME/pwned_bash && chmod +s $HOME/pwned_bash": The exact same container command was run.
3. $HOME/pwned_bash -c "whoami": Executing the SUID-bit-enabled pwned_bash.

This time, the output was the user's unprivileged UID, specifically a high-numbered UID that was part of their allocated sub-UID range, rather than root. This confirmed that even with the SUID bit set, the execution context remained within the user's unprivileged namespace, preventing a host-level root escalation.

To further illustrate the granular control and auditing capabilities of the new system, Carroll presented pitr_bcc output. pitr_bcc is Yelp's open-source eBPF tracing solution that logs the entire calling process tree for network connections. The output for a curl 44con.com command executed within an unprivileged container clearly showed:

• At the bottom, systemd running as root.
• Immediately above it, a user-level systemd unit with the login_uid set to Carroll's actual user ID. This is significant because, once the login_uid is set, it is immutable and propagated to all child processes, even if they temporarily gain root privileges within their namespace.
• Further up the chain, the curl 44con.com command itself, with the username displayed as nobody (which Carroll clarified was actually a high sub-UID, not the literal nobody user).

This pitr_bcc output served as a powerful visual confirmation that the system was correctly enforcing unprivileged execution, propagating the login_uid for auditability, and effectively isolating container processes within their designated user namespaces, even when they internally believed they were running as root.

Defensive Implications

▶ Watch: Yelp Compose and the critical shared development infrastructure (6:45)

The experience at Yelp provides several critical defensive implications for organizations grappling with container security, particularly in multi-tenant or shared development environments:

1. Mandate Rootless Container Runtimes for Development: The most direct implication is to move away from privileged Docker daemons in shared environments. Solutions like Podman (with its daemonless mode and additional features) or Docker Rootless should be the default for developers. This fundamentally shifts the security boundary, ensuring that container escapes lead only to an unprivileged user's context, not host root.
2. Leverage Unprivileged Username Namespaces: Defenders must understand and actively utilize unprivileged username namespaces. This core Linux kernel feature is the bedrock of rootless container security, enabling the remapping of UIDs and GIDs within a container to unprivileged host UIDs. This prevents arbitrary root access from containerized workloads.
3. Implement ID Mapped Mounts for Shared Storage: For environments where shared base images or volumes are necessary, ID mapped mounts are indispensable. This feature allows transparent UID/GID remapping for bind-mounted filesystems, ensuring that shared resources have correct permissions for unprivileged users without modifying the underlying files. This is crucial for maintaining dev/prod parity and avoiding breakage. Ensure your kernel version (5.19+ for overlay2 support) supports this.
4. Audit and Authorize Docker Socket Access: For any remaining privileged Docker socket access (e.g., for specific administrative tasks), implement robust auditing and authorization. Tools like doc_proc (a proxy logging metadata about callers) combined with an Open Policy Agent (OPA) or similar policy engine can enforce granular access controls, ensuring only authorized users or groups can perform privileged actions, and every action is logged with immutable login_uid information.
5. Be Prepared for Custom Tooling and Deep Technical Diving: As demonstrated by the "shaving yaks" narrative, generic security advice often requires significant custom engineering. Defenders should be prepared to:

• Develop custom solutions for image cleaning, network orchestration, or other environment-specific challenges.
• Address API incompatibilities or performance bottlenecks in upstream tools (e.g., the dpkg caching solution).
• Deeply understand Linux kernel primitives (namespaces, cgroups, mount options) and filesystem behavior (overlayFS, copy-on-write) to diagnose and solve complex issues.

1. Prioritize Developer Velocity and User Experience: Security solutions will only be adopted if they are minimally disruptive to developer workflows. The "paved path" approach, focusing on minimizing friction and integrating solutions seamlessly into existing tooling (like yelp compose and the Docker CLI compatibility), is key to achieving buy-in and sustained compliance. pod_doctor exemplifies providing immediate self-service troubleshooting.
2. Challenge Generic Best Practices and Assess Risk Contextually: Carroll emphasizes that the security industry often recommends remediations that cost more than the accepted risk, often without a deep understanding of the underlying mechanisms. Defenders should:

• Critically evaluate the true cost and complexity of security recommendations.
• "Meet stakeholders where they live" by understanding business models and workflows to tailor solutions.
• Recognize that "just use rootless containers" drastically understates the engineering effort involved.

1. Stay Current with Kernel and Runtime Features: The timely availability of kernel 5.19+ for ID mapped mounts was a project enabler. Defenders need to stay abreast of new kernel features and container runtime developments that can provide more secure and efficient solutions.

By embracing these principles, organizations can transition from a vulnerable, privileged container environment to a robust, unprivileged one, significantly enhancing their security posture while maintaining high developer productivity.

Key Takeaways

• Docker's default privileged model is a critical security vulnerability in shared development environments, allowing easy root escalation without audit.
• Rootless container technologies (like Podman or Docker Rootless) are the fundamental solution for securing multi-tenant containerized development, but their implementation is complex and requires significant engineering effort.
• Unprivileged username namespaces and ID mapped mounts are indispensable kernel features for managing user permissions and shared image storage in rootless environments without compromising dev/prod parity.
• Deep technical understanding and custom tooling are essential to overcome the "love crafty and hellscape" of real-world implementation, addressing issues from disk space management to networking and API compatibility.
• Security solutions must be context-specific and prioritize developer velocity to achieve adoption. Minimizing disruption and integrating with existing workflows are key to success.
• Generic security advice often oversimplifies complex problems. Security professionals must understand the true cost and underlying mechanisms of remediations, rather than just repeating "best practices."

About the Speaker(s)

Matt Carroll is a Tech Lead on the Incident Detection and Response Team at Yelp. With a decade of experience at Yelp, his career trajectory has spanned roles from CIS admin to Site Reliability Engineer (SRE) and, most recently, security. He spent his early years at Yelp on the operations team (now Production Engineering) before moving to security, where he has contributed to three different security teams. Carroll describes his approach to problem-solving as using "structures as a hammer," indicating a pragmatic and often deeply technical methodology.

He acknowledges that the unprivileged containers project, a year-and-a-half endeavor, concluded just as he transitioned to his current role in incident detection and response, humorously noting its "nonse" connection to his new team's mission. Carroll's candidness about the project's difficulties, his own emotional investment, and the lessons learned (including the value of peer support from colleagues like Mateo, Zach, and Daniel) underscores his practical, experience-driven perspective on complex security challenges.