Analyzing volatile memory on a Google Kubernetes Engine node

Analyzing Volatile Memory on a Google Kubernetes Engine Node

Speakers: Marcus Hallberg, Detection Engineer, Spotify

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=r0JGLcRLW9E

Overview

In this insightful talk at BSides NYC, Marcus Hallberg, a Detection Engineer at Spotify, unveiled a novel approach to conducting volatile memory analysis on Google Kubernetes Engine (GKE) nodes. The presentation addressed a critical gap in cloud-native forensics: gaining deep visibility into the runtime state of containerized applications and the underlying host when traditional logging mechanisms fall short or direct host access is restricted. Hallberg demonstrated how a combination of open-source tools—avml, dwarf2json, and Volatility 3—can be leveraged to capture and analyze memory snapshots from GKE nodes running Container Optimized OS (COS).

This method offers an invaluable alternative for incident responders and security engineers operating within large-scale Kubernetes environments. By providing a "photographic piece" of a node's memory at a specific point in time, it allows for the discovery of ephemeral processes, hidden activities, and command histories that might otherwise evade detection through standard logs or disk-based forensics. The talk is particularly significant for organizations like Spotify, which operate vast GKE infrastructures with stringent security policies preventing direct SSH access to production nodes, thereby necessitating innovative, container-native forensic techniques.

Background

▶ Watch: Introduction and talk agenda overview (0:00)

Spotify operates a formidable infrastructure on Google Cloud Platform, spanning over five GCP regions globally, managing hundreds of production clusters, each potentially running up to 4096 nodes. This environment hosts more than 4,000 live services across 3,000-plus namespaces, underscoring the critical need for robust security and forensic capabilities. Hallberg began by providing a high-level overview of GKE architecture, defining key terms such as cluster (multiple nodes), node (cloud VM), namespace (resource isolation), pod (smallest application resource), and container (executable image). He explained that the control plane and API server manage these resources, scheduling pods and monitoring usage, while Google manages the underlying infrastructure components.

A core concept for this talk is the kernel, which acts as the crucial intermediary between applications and the underlying hardware (storage, network, CPU, RAM) via system calls. Understanding the kernel's state offers profound insights into an entire infrastructure, revealing running processes, file system interactions, and overall operational context. The problem Hallberg sought to address was how to gain this deep insight, particularly "what a user did after they accessed a container," when existing monitoring tools or logs are insufficient. While streaming telemetry and disk snapshots provide valuable data, volatile memory snapshots offer an additional layer of forensic detail, capturing the live state that other methods might miss.

A significant challenge on GKE nodes is the use of Container Optimized OS (COS), which prevents the direct use of kernel modules for memory acquisition. This necessitates alternative approaches. Hallberg emphasized that while commercial tools exist for streaming telemetry of containerized workloads, his method provides an open-source, portable alternative, particularly useful for targeted troubleshooting or as a comparative analysis tool. Memory snapshots, despite the rise of streaming telemetry, remain a vital forensic technique, offering a precise point-in-time view of system activity and aiding in comprehensive incident response when combined with other data sources.

Key Findings

▶ Watch: Spotify's approach to GKE container workloads (2:29)

The central finding of Marcus Hallberg's talk is the successful development and demonstration of an open-source methodology for performing volatile memory analysis on Google Kubernetes Engine nodes running Container Optimized OS (COS). This technique addresses a critical gap in cloud forensics by enabling deep inspection of live system state within highly dynamic and restricted environments.

The key discoveries and contributions include:

1. Feasibility of Open-Source Toolchain: Hallberg unequivocally demonstrated that a combination of avml for memory acquisition, dwarf2json for symbol file generation, and Volatility 3 for analysis can effectively be used on GKE. This provides a powerful, accessible, and auditable alternative to commercial solutions or custom kernel module development (which is not viable on COS).
2. Undocumented VM Linux File Acquisition Method: A pivotal breakthrough was the discovery of a public, albeit undocumented, Google API endpoint that allows for the download of the exact VM Linux file (the unencrypted kernel binary) corresponding to a specific COS build ID. This build ID is conveniently exposed within the GKE node's image name. This capability is crucial because the VM Linux file acts as a "map" for Volatility 3 to correctly interpret the raw memory dump. Without it, detailed analysis of kernel structures is impossible on COS. Hallberg obtained permission from Google engineers to disclose this method.
3. Container-Native Acquisition Strategy: To overcome the restriction of not being able to build kernel modules on COS and the general policy of disallowing SSH access to production nodes (as is the case at Spotify), Hallberg devised a strategy involving a custom privileged container. This container, equipped with avml, is deployed to the target GKE node with access to the host file system (/proc/kcore), performs the memory dump, and can then be removed, ensuring minimal footprint and adherence to security policies.
4. Demonstrated Forensic Efficacy: The live demo successfully extracted valuable forensic artifacts from a GKE node's memory, including:

• Process lists (pslist) showing active processes like watch, python, sh, and netcat (nc).
• Process trees (pstree) illustrating parent-child relationships, crucial for understanding execution flow (e.g., netcat being a child of a python script).
• Evidence of ephemeral activity, such as a netcat listener process remaining active in memory even after the script file that launched it was deleted from disk.
• Partial bash history, though acknowledged as tricky and not always complete, still providing potential indicators of compromise.

1. Addressing Gaps in GKE Logging: The method directly tackles the limitations of standard GKE logging, which often provides generic kubectl exec entries without detailing the specific commands run interactively within a container. Memory analysis provides the granular context needed to understand post-execution activities.

These findings collectively present a robust, open-source framework for GKE volatile memory forensics, significantly enhancing the visibility available to security practitioners in cloud-native environments.

Technical Deep Dive

▶ Watch: GKE basics: Cluster, Node, Pod, Container (3:20)

The technical implementation of Marcus Hallberg's GKE memory analysis hinges on a three-pronged approach: memory acquisition, symbol file generation, and forensic analysis, all orchestrated using open-source tools and custom scripts.

1. Memory Acquisition from GKE Nodes

The first challenge is to acquire the live kernel memory from a GKE node. On COS, traditional methods involving kernel modules are not viable. Hallberg's solution leverages the /proc/kcore pseudo-file system entry, which represents the entire physical memory of the running node in the ELF core file format. The size of this file corresponds to the physical RAM size plus an additional four kilobytes, meaning dumps can be substantial (ranging from 800 MB to 30 GB in Spotify's tests).

To perform the dump, Hallberg utilizes avml (Acquire Volatile Memory for Linux), an open-source tool originally developed by Microsoft. avml is capable of dumping /proc/kcore in a compressed format, which is crucial for managing the size of the acquired data.

The critical hurdle is granting a container access to the host's /proc/kcore. Standard container security practices strongly advise against accessing the host file system. Hallberg's approach involves deploying a custom privileged container directly onto the target GKE node. This container is configured with specific security contexts to allow access to the host's file system. Once the memory dump is complete, this privileged container can be removed, minimizing its attack surface and operational presence on the node. The speaker demonstrated how this privileged container definition is managed using Terraform for infrastructure as code.

2. Interpreting the Memory Dump with a Symbol File

A raw memory dump is unintelligible without a "map" to interpret the kernel's internal data structures. This map is provided by a symbol file, which is derived from the VM Linux file—the unencrypted binary of the kernel itself. Obtaining the correct VM Linux file for a GKE node running COS presents a significant challenge. The VM Linux file found directly on a COS host is often not the complete or accurate version needed for comprehensive analysis.

Hallberg's breakthrough here was the discovery that the build ID of the COS image is embedded within the GKE node's image name (e.g., cos-105-17412-108-0). Knowing this build ID, it's possible to query an undocumented but public Google API endpoint to download the precise VM Linux file corresponding to that specific COS version. This revelation is a cornerstone of the entire methodology, as it provides the necessary kernel debugging symbols.

Once the VM Linux file is obtained, the open-source tool dwarf2json is used to build the symbol file. This tool parses the DWARF debugging information within the VM Linux file and converts it into a JSON format that Volatility 3 can readily consume.

3. Forensic Analysis with Volatility 3

With the compressed memory dump and the generated symbol file in hand, the final step is analysis using Volatility 3. Volatility 3 is a powerful, open-source framework for extracting digital artifacts from volatile memory (RAM) samples. It uses a plugin-based architecture to inspect various aspects of a system's runtime state.

Hallberg's testing setup involved a dedicated forensic compute engine (a separate VM in GCP). This engine is responsible for:

1. Pulling the compressed memory dump from a Google Cloud Storage (GCS) bucket (where the avml pod uploads it).
2. Decompressing the memory dump.
3. Querying the Google API to obtain the COS build ID.
4. Downloading the VM Linux file via the undocumented API.
5. Building the symbol file using dwarf2json.
6. Running Volatility 3 plugins against the memory dump and symbol file.

The demonstration highlighted several key Volatility 3 plugins, including:

• linux.pslist.PsList: Lists all active processes.
• linux.pstree.PsTree: Displays processes in a hierarchical tree, showing parent-child relationships.
• linux.bashhistory.BashHistory: Attempts to extract bash command history. Hallberg noted this plugin can be "tricky" and highly dependent on timing and shell interaction modes, often yielding incomplete results.
• Other mentioned plugins for comprehensive analysis include linux.map_files.Map_Files (mapped files), linux.modules.Modules (loaded kernel modules), and linux.mount.Mount (mount points).

The entire process, from deploying the privileged container to running Volatility 3 analysis, is orchestrated by a Python script. This script automates the kubectl exec commands, avml execution, GCS uploads, and the subsequent analysis steps on the forensic engine, providing a streamlined workflow for GKE volatile memory forensics.

Demo / Proof of Concept

▶ Watch: High-level GKE cluster architecture explained (4:30)

Marcus Hallberg's live demonstration provided a compelling proof of concept for his GKE volatile memory analysis methodology. The setup involved a single GKE node with two key pods: an attacker pod and an avml pod, all managed via Terraform in a dedicated GCP project.

Attacker Pod Activity:

Hallberg first accessed the attacker pod using kubectl exec. This pod, based on an Nginx image, was initially clean. He then executed a script named actions.sh which performed several interactive commands, including:

• Running a watch command to continuously curl the node's IP address, simulating persistent network activity.
• Executing a Python script. This Python script was designed to perform a more "malicious" action: it created a temporary shell script, which then downloaded and fired up a netcat (nc) listener. Crucially, after launching the netcat process, the Python script deleted the temporary shell script file from the disk. This scenario is vital for demonstrating the value of memory forensics, as the file would no longer be present for disk-based analysis.

The speaker showed that while some commands appeared in bash history after exiting and re-entering the shell, the history was not always comprehensive, highlighting the limitations of relying solely on shell logs.

Memory Acquisition:

After simulating the attacker activity, Hallberg initiated the memory acquisition process via his Python orchestration script. This script:

1. Created a pre-signed URL for uploading the memory dump to a GCS bucket.
2. Executed avml within the privileged avml pod on the target GKE node. This process accessed /proc/kcore and generated a compressed memory dump. The speaker noted that this process can take some time, and the dump size varies significantly (800MB to 30GB depending on node RAM and activity).
3. Uploaded the compressed memory dump to the designated GCS bucket.

Forensic Analysis on the Compute Engine:

The orchestration script then shifted to a forensic compute engine VM. Here, it:

1. Downloaded and decompressed the memory dump.
2. Queried the Google API to retrieve the COS build ID from the GKE node's image name.
3. Downloaded the corresponding VM Linux file using the previously mentioned undocumented public API.
4. Used dwarf2json to build the symbol file from the VM Linux file.
5. Finally, executed Volatility 3 plugins to analyze the memory dump using the generated symbol file.

Demo Results and Findings:

The Volatility 3 output provided concrete evidence of the simulated malicious activity:

• linux.pslist.PsList: Clearly showed the watch command, the python script, sh (shell), nc (netcat), and the avml process itself, all active on the GKE node.
• linux.pstree.PsTree: Displayed the hierarchical relationships, revealing sh and nc as child processes of the python script. This provided critical context on how the netcat listener was initiated. Importantly, the netcat process was still active and listening, even though its initiating script file had been removed from the disk, demonstrating the ability of memory forensics to capture ephemeral artifacts.
• Full Process Information: Detailed command-line arguments and process details for watch, the "malicious script" (Python), and the netcat listener were visible, offering a complete picture of their execution.
• Bash History: While some commands from actions.sh were visible, Hallberg reiterated that the bashhistory plugin is often unreliable for a complete record, emphasizing the need for memory analysis to complement logs.

The demo successfully validated the entire workflow, demonstrating that deep, volatile memory forensics is indeed achievable on GKE nodes, providing insights beyond what traditional logs or disk-based methods can offer, especially in environments with restricted host access.

Defensive Implications

▶ Watch: Understanding the kernel and its importance (6:20)

The methodology presented by Marcus Hallberg offers several crucial defensive implications for organizations operating on Google Kubernetes Engine, particularly those with stringent security postures.

1. Enhanced Incident Response Capability: This open-source approach provides a powerful tool for incident responders to gain deep visibility into GKE nodes during a compromise. When standard logging (e.g., generic kubectl exec entries) fails to provide sufficient detail on malicious activity within a container, memory snapshots can reveal running processes, active network connections, and command execution that are otherwise hidden.
2. Detection of Ephemeral Artifacts: The ability to capture processes like netcat that are active in memory even after their initiating script files have been deleted from disk is a significant advantage. This helps detect sophisticated attackers who employ fileless or living-off-the-land techniques to evade disk-based forensics.
3. Complementary to Existing Security Tools: Hallberg stressed that this method is not a replacement but a strong complement to commercial streaming telemetry tools (e.g., Sysdig, Lacework) and other forensic data sources like disk snapshots and network PCAPs. Integrating memory analysis rounds out the forensic picture, providing a more comprehensive understanding of an incident.
4. Overcoming Access Restrictions: For organizations like Spotify that prohibit direct SSH access to production GKE nodes, this container-native memory acquisition method is invaluable. It allows for forensic data collection without violating established security boundaries or requiring elevated, persistent host access.
5. Targeted Forensic Investigations: The use of node selectors allows security teams to precisely deploy the privileged avml container to specific GKE nodes that are exhibiting suspicious behavior or are identified as potential points of compromise. This enables focused investigations without impacting the entire cluster.
6. Resource Management and Operational Security: Defenders must be acutely aware of resource consumption when implementing this technique. Deploying a privileged container that performs a memory dump can consume significant CPU, RAM, and disk resources. Overconsumption can lead to the pod being evicted or the node rebalancing, which could disrupt production workloads or cause the forensic operation to fail. Careful resource calculation and sizing of the privileged container are essential to ensure operational stability and the success of the forensic task. Future improvements include automating resource calculation prior to deployment.
7. Leveraging Open-Source for Transparency and Cost-Effectiveness: The reliance on open-source tools (avml, dwarf2json, Volatility 3) means the methodology is transparent, auditable, and cost-effective. Organizations can adapt, extend, and integrate these tools into their existing security stacks without vendor lock-in.
8. Future Enhancements: Hallberg also outlined future defensive improvements, such as automating the entire deployment and analysis pipeline, and integrating it with other forensic data streams (disk dumps, network pcaps) to build a more holistic incident response platform. This could potentially extend to automated responses, like isolating or terminating a compromised node, although such actions require careful consideration in production environments.

In essence, Hallberg's work empowers GKE defenders with a powerful, flexible, and open-source capability to peer into the volatile memory of their containerized environments, significantly enhancing their ability to detect, analyze, and respond to threats in cloud-native settings.

Key Takeaways

• Volatile memory analysis is achievable on GKE nodes running Container Optimized OS (COS), despite the limitations on kernel module usage and restricted host access.
• A crucial enabler is the discovery of an undocumented, public Google API that allows for the retrieval of the precise VM Linux file using the COS build ID exposed in the GKE node's image name.
• The methodology relies on a powerful open-source toolchain: avml for memory acquisition from /proc/kcore, dwarf2json for symbol file generation, and Volatility 3 for deep forensic analysis.
• Acquisition is performed via a custom, temporary privileged container deployed to the target GKE node, which accesses the host file system and uploads a compressed memory dump to GCS.
• Memory forensics provides critical visibility into ephemeral processes and command execution (e.g., netcat listeners, detailed kubectl exec activity) that are often missed by standard GKE logs or disk-based analysis.
• Careful resource management is essential when deploying privileged containers for memory dumps, as overconsumption can lead to pod eviction or node rebalancing, impacting forensic efforts and production stability.

About the Speaker(s)

Marcus Hallberg is a Detection Engineer on the security team at Spotify. He is passionate about enhancing security visibility and forensic capabilities in cloud-native environments. Outside of his technical pursuits, Marcus is an avid hiker, folk dancer, and traveler, with a particular fondness for rock climbing. He also possesses a unique skill of mimicking accents, a talent he admits drove his mother "insane" during his childhood. Marcus emphasizes that the work presented in this talk is part of his personal research and contributions at Spotify, and he does not claim to be a kernel or Kubernetes expert, inviting collaborative discussion and further contributions from the community.