Metal-as-a-Disservice: Exploiting Legacy Flaws in Cutting Edge Clouds

Metal-as-a-Disservice: Exploiting Legacy Flaws in Cutting Edge Clouds

Speakers: Bill Demirkapi

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=EtGhHCr3VLE

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Bill%20Demirkapi%20-%20Metal-as-a-Disservice%20Exploiting%20Legacy%20Flaws%20in%20Cutting-Edge%20Clouds.pdf

Overview

The rise of GPU-focused cloud providers has created a new class of infrastructure security problems. Unlike established hyperscalers such as AWS, Azure, and Google Cloud — which have invested more than a decade hardening their hardware security stories — the new wave of machine-learning-oriented bare metal providers often resell capacity from smaller data centers with little to no firmware hygiene, network isolation, or decommissioning discipline. Bill Demirkapi spent spring 2025 renting GPUs across these platforms to study their attack surfaces, and what he found was alarming: systemic vulnerabilities spanning boot security, baseboard management controllers (BMCs), PCI device firmware, and data-center networking.

The central irony the talk explores is that bare metal is marketed as more secure than virtualization because customers get "physically isolated" hardware and root access — yet that very root access is also the attack surface. In a shared, multi-tenant bare metal environment, one customer's root privileges can, in practice, be leveraged to compromise other customers' workloads both during provisioning and long after a server has ostensibly been reclaimed. The talk walks through each attack layer methodically and closes with case studies against real providers, including RunPod (which Demirkapi credits for fast remediation).

The broader audience impact is significant. Hundreds of AI startups and researchers currently rent GPUs from exactly the providers described here. The assumption that bare metal cloud is inherently more secure than VMs is demonstrably false when providers do not follow rigorous decommissioning and network-isolation procedures.

Background

▶ Watch: We have onboard devices be that offer a point for persistence. (4:17)

Bare metal cloud servers — where the customer runs directly on the physical host without a hypervisor — are a growing market segment driven by the GPU compute needs of machine learning workloads. The main selling points are performance (no PCI passthrough overhead) and control (root access to hardware). Providers such as CoreWeave, Lambda Labs, Vast.ai, and others compete aggressively on GPU pricing.

Canonical's Metal as a Service (MaaS) is among the most widely deployed management frameworks for automating bare metal provisioning. MaaS relies on technologies that are almost two decades old: PXE (Preboot Execution Environment) booting over DHCP, unencrypted HTTP for provisioning APIs, and IPMI for out-of-band management. These protocols predate modern threat models and were never designed with adversarial tenants in mind.

Prior academic and industry research has explored boot security and hardware implants primarily in the context of client devices or enterprise servers under the threat model of physical attackers. Demirkapi's contribution is framing the same attack surfaces against a cloud shared-tenancy model, where physical access is replaced by root-level OS access, dramatically lowering the bar for exploitation.

Key Findings

▶ Watch: How do you know if it's enabled? (14:10)

Boot environment persistence via HTTP Boot and authenticated UEFI variables: Bare metal servers at ML-focused providers routinely ship without Secure Boot enabled and without any firmware reflashing between customer tenancies. An attacker with root access can create a UEFI boot option pointing to a remote HTTP EFI image (HTTP Boot has been in the EDK2 reference implementation for over a decade). To make this persistence harder to remove, the attacker can create an authenticated UEFI variable protected with their own key, meaning the boot order cannot be reverted without access to that private key. Tools such as AMI's undocumented BIOS variable structures (parseable via open-source utilities) allow further manipulation of hidden setup options even without console access.

BMC takeover via undocumented IPMI commands and vendor tool discrepancies: Several servers Demirkapi tested had BMCs with nominal firmware signature verification that was either disabled by default (e.g., SuperMicro X11/H11 series with backwards-compatibility exemptions) or enforced only at the client layer — meaning a different vendor's flash tool for the same AST2500 chip would bypass the check entirely. Undocumented IPMI reset commands could restore BMCs to factory defaults with default credentials, giving an attacker full out-of-band access. Binarly has published multiple SuperMicro BMC CVEs; downgrading BMC firmware below their 2024 fix date re-enables those vulnerabilities.

Network isolation failures enabling cross-tenant RCE: The most immediately impactful finding was that several providers had no meaningful network segmentation. Simply enabling promiscuous mode on a rented server captured unencrypted traffic from adjacent tenants' servers. ARP spoofing against the MaaS gateway produced a flood of cross-tenant traffic. Because MaaS uses unencrypted PXE and HTTP for provisioning, a man-in-the-middle attacker can serve a malicious EFI boot image to any server being provisioned on the same network segment — achieving remote code execution on another customer's machine without any prior foothold on that machine.

RunPod cross-cloud compromise: RunPod, a popular container-based GPU provider, was found to route customer traffic through Mass Compute, an underlying bare metal provider. Because RunPod itself was not offering bare metal, customers assumed they were isolated. Yet by ARP spoofing Mass Compute's network, Demirkapi intercepted unencrypted ML training traffic (Calyx network) and log data from RunPod's managed inference service, including Docker container names and credentials. RunPod responded and remediated quickly.

PCI device option ROM backdoors: NIC and GPU option ROMs are EFI drivers embedded in the device's flash and loaded directly into the host UEFI environment at boot. Broadcom adapters (managed by the bnxt driver) can be dumped and flashed with the ethtool utility. The decade-old Thundergate project provides open-source tooling to replace Broadcom firmware and inject malicious option ROMs. Unlike BIOS reflashing, PCI device firmware resets are essentially never performed during server decommissioning at the providers tested.

Firmware downgrade bypasses anti-rollback: Intel Boot Guard and AMD ROM Armor both lack meaningful anti-rollback enforcement in practice — no major OEM has deployed Intel Secure Version Numbers to prevent reflashing with legitimately signed but vulnerable firmware. An attacker can therefore downgrade to a pre-patch BIOS and then exploit known vulnerabilities (e.g., the Binarly-discovered SuperMicro issues) to achieve SPI flash write access.

Wi-Fi attack surface on Mac Minis: A rented Mac Mini exposed Wi-Fi to a data center's open visitor network. macOS's Airport hardware supports monitor mode by default, allowing passive capture of raw 802.11 frames from surrounding networks, including WPA2 EAPOL handshakes from locked enterprise networks in the same facility.

Technical Deep Dive

▶ Watch: opportunity is limited to a supply chain. (23:24)

UEFI Boot Persistence

Modern UEFI firmware implements HTTP Boot (documented in the EDK2 codebase for over a decade), which allows specifying a remote server URL in a UEFI boot option's device path rather than relying on a local PXE DHCP server. An attacker with root access can use standard Linux utilities (efibootmgr) to create a new boot entry with a device path that first obtains a DHCP address and then fetches an EFI image from an attacker-controlled HTTPS server.

To harden this persistence, UEFI supports authenticated variables: normally used for signature databases (db, dbx) and the Platform Key, but equally applicable to arbitrary private namespaces. An attacker creates a keypair, stores the boot entry as an authenticated variable protected by that key, and from that point forward any attempt to modify or delete the boot entry requires a payload signed by the attacker's private key — which the provider does not possess.

When direct console access is unavailable and the attacker wants to modify BIOS settings (e.g., disable Intel BootGuard enforcement), tools that parse AMI HII (Human Interface Infrastructure) variable structures can enumerate the byte offsets of specific settings inside the NVRAM, allowing the attacker to write directly to undocumented structures. Although these variables are marked NV+RT (non-volatile, runtime inaccessible), an attacker can create a minimal EFI application that runs before the OS loads, writes the desired offset values, and exits — effectively changing BIOS settings without ever seeing a BIOS menu.

BMC Attack Chain

The Baseboard Management Controller is a system-on-chip that provides out-of-band server management independent of the host CPU and OS. It is the single most powerful component for persistent access: it can power cycle the host, inject console input, mount virtual media, and in many implementations read/write SPI flash.

Demirkapi's approach to BMC exploitation:

1. Enumeration: ipmitool and ipmiconfig dump the BMC configuration over the host-side KCS (Keyboard Controller Style) interface. This reveals firmware version, enabled features, and channel configuration.
2. Identifying proprietary customizations: OEMs layer their own code on top of reference BMC implementations (AMI MegaRAC, ASPEED SDK). This custom code receives far less security review. Proprietary undocumented IPMI commands discovered through documentation mining and leaked driver packages can reset the BMC to factory defaults, yielding admin access through default credentials.
3. Bypassing signature verification via vendor tool swapping: The official SuperMicro flashing tool enforces signature checks that are entirely client-side. Substituting a different vendor's tool for the same ASPEED AST2500 chip bypasses these checks, allowing reflashing with modified firmware. Demirkapi obtained obscure vendor utilities by writing scripts to bulk-download every driver package from OEM driver portals (SuperMicro, Dell, etc.) and extracting their contents — finding unlisted flash utilities left in zip archives.
4. Firmware downgrade to enable known CVEs: Binarly disclosed multiple stack/heap overflow vulnerabilities in SuperMicro BMC firmware before 2024. Downgrading to a vulnerable version and then exploiting those CVEs achieves arbitrary flash writes even on systems where the signature verification is genuinely enforced.

Network-Level Cross-Tenant Attacks

The MaaS provisioning flow works as follows: when a server powers on, the UEFI firmware broadcasts a DHCP Discover. The MaaS server responds with a DHCP Offer containing a PXE boot URL. The server fetches a MaaS-specific EFI image over HTTP, which then contacts the MaaS API to retrieve customer-specific configuration, credentials, and scripts.

An attacker on the same Layer-2 segment can:

• Passive sniffing: Capture DHCP traffic revealing which servers are being provisioned.
• DHCP spoofing: Respond to a DHCP Discover faster than the legitimate MaaS server, providing a malicious PXE URL pointing to an attacker-controlled EFI image. This yields code execution on the victim's machine during its provisioning phase, before the customer's OS is even running.
• ARP spoofing the MaaS gateway: Once the attacker becomes the MITM for the gateway, all MaaS HTTP API traffic is visible and modifiable, allowing hijacking of boot images for any provisioning request.
• MaaS provisioning key extraction: The MaaS client stores its provisioning token in /var/lib/cloud as a Python pickle file. An attacker who exfiltrates this can authenticate to the MaaS API and access customer configurations. Additionally, a race condition in MAC-address-based provisioning key assignment allows an attacker to claim a victim server's MAC address and receive its provisioning secrets, including customer-specified SSH keys and initialization scripts.
• Squid proxy tunneling: MaaS servers expose a Squid HTTP proxy by default for package caching. Even on a partially segmented network, this proxy can be abused as an HTTP CONNECT tunnel to reach other servers on the internal network.

PCI Device and Option ROM Threats

The Linux lspci command and /sys/bus/pci/ expose all attached PCI devices. For NIC devices using the Broadcom bnxt_en driver, the kernel source includes an NVRAM flashing pathway. The ethtool utility exposes this via --flash and --get-eeprom subcommands. The Thundergate project (open-source, circa 2013) provides a complete toolkit for reading, modifying, and writing Broadcom NIC firmware, including the option ROM EFI driver that gets loaded by the host UEFI at boot.

NVIDIA GPU firmware is protected by the Falcon microprocessor's root of trust (enforced since Maxwell architecture, 2014): any firmware image must carry a valid NVIDIA signature. GPU modding communities have made some progress with cross-flashing signed images between card variants but have not broken the signature scheme. This makes GPU firmware backdooring via a software-only path essentially infeasible without supply-chain access.

Demo / Proof of Concept

▶ Watch: BMC firmware did not match the BMC firmware signature verification mechanism. (28:18)

Demirkapi demonstrated the network-level attack chain against a real ML cloud provider:

1. Promiscuous mode capture: Simply enabling promiscuous mode on a rented server revealed DHCP Offer packets from the MaaS server and unencrypted provisioning traffic from adjacent tenant servers.
2. ARP spoofing MITM: Running a standard ARP spoofing attack against the MaaS gateway immediately produced a flood of cross-tenant traffic, including unencrypted ML training coordination traffic (Calyx) and Docker container logs from RunPod's managed inference service.
3. DHCP-to-RCE: Upon detecting a DHCP Discover from a provisioning server, the attacker sent a spoofed DHCP Offer with a malicious PXE URL, causing the victim server to fetch and execute an attacker-controlled EFI image — achieving code execution before the customer's OS ever loaded.
4. BMC flash bypass: The Gigaflash x64 tool rejected a modified BMC firmware image with a signature error. Using a generic ASPEED AST2500 flash tool from a different vendor, the same modified firmware was accepted and flashed without error.

Defensive Implications

▶ Watch: Another attack here is that when you dis detect that DHCP discover, you regis... (38:29)

For bare metal cloud providers:

• Reflash every firmware surface between customer tenancies: BIOS/UEFI, BMC, NIC option ROMs, HBA firmware. Do not rely solely on OS-level cleanup.
• Implement network isolation at the switch level (private VLANs per tenant), not at the host OS level (where a customer with root access can disable it).
• Follow OpenStack Ironic's decommissioning model, which includes automated firmware reset steps, rather than MaaS's default which lacks them.
• Physically disconnect or hardware-disable unnecessary devices (Wi-Fi adapters, unused PCIe slots) before renting servers.
• Use BMCs with real firmware signature enforcement (not client-side-only verification) and enable anti-rollback via Secure Version Numbers.
• Replace decade-old unencrypted PXE/HTTP provisioning with TLS-authenticated channels.

For customers of bare metal cloud:

• Treat the network as hostile: assume other tenants can see your traffic. Encrypt all inter-node ML training traffic.
• Do not pass sensitive credentials via provisioning scripts without additional out-of-band verification.
• Recognize that "bare metal = more secure" is marketing, not engineering. Hypervisor-based isolation, while it has its own attack surface, provides meaningful hardware-level separation that most bare metal providers cannot match in practice.
• Avoid MFA bypass configurations that whitelist provider IP ranges.

Key Takeaways

• MaaS-managed bare metal providers rely on two-decade-old unencrypted protocols (PXE/DHCP, HTTP) for provisioning, creating trivial cross-tenant RCE pathways when network isolation is absent.
• Firmware signature verification in BMCs and BIOS is often disabled, client-side-only, or bypassable via vendor tool substitution, making firmware-level persistence both practical and difficult to detect.
• PCI device option ROMs (NICs in particular) are essentially never reflashed between customer tenancies, providing a persistent implant surface that survives OS reinstalls.
• Bare metal cloud is not inherently more secure than virtualization; it often has a far worse security posture because providers lack the decade-plus of hardware security investment that hyperscalers have accumulated.
• Network isolation is the single highest-impact defensive control: if tenants cannot reach each other's Layer-2 segment, the entire class of MaaS provisioning-hijack attacks becomes infeasible.

About the Speaker(s)

▶ Watch: image that allows you to get code execution. (39:38)

Bill Demirkapi is a security researcher whose work spans hardware, firmware, web, and cloud security. He has a track record of systemic vulnerability research — finding classes of flaws that affect multiple vendors simultaneously rather than single-product bugs — and has been working on automating vulnerability discovery using foundation models for code reasoning. He has presented previously at DEF CON and maintains a public presence on X (Twitter). His spring 2025 bare metal research was prompted by his own need to rent GPUs for machine learning experimentation, turning a practical use case into a comprehensive security audit of an underexplored infrastructure category.