Virtualization Based Insecurity: Weaponizing VBS Enclaves

Virtualization Based Insecurity: Weaponizing VBS Enclaves

Speakers: Ori David

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=DqC4LZTTCa0

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Ori%20David%20-%20Virtualization%20Based%20Insecurity%20Weaponizing%20VBS%20Enclaves.pdf

Overview

Windows Virtualization Based Security (VBS) is Microsoft's flagship security architecture innovation of the past decade, isolating the most sensitive OS components — credential stores, security policies, kernel integrity — into a hardware-enforced virtual machine that the regular OS kernel cannot tamper with. VBS Enclaves, a newer VBS feature, extend this trusted execution model to user-space applications, allowing developers to create isolated secure computation environments within their own processes. At DEF CON 33, Ori David (security researcher at Akamai) demonstrated that these same security guarantees that make VBS Enclaves powerful for defenders can be systematically weaponized by attackers, turning an enterprise security feature into an evasion and persistence mechanism that bypasses the very tools designed to monitor it.

The talk is a first public exploration of VBS Enclaves from an offensive security perspective, demonstrating that the same isolation properties that protect sensitive computations also protect malicious code from detection, analysis, and termination.

Background

▶ Watch: Introduction: Virtualization-Based Security (VBS) architecture (0:33)

The Evolution of Windows Security Boundaries

Traditional Windows security relied on processor rings: ring 0 (kernel) and ring 3 (user space). The boundary between them was enforced by the processor, but numerous kernel vulnerabilities over the decades showed that a compromised kernel could trivially access all user-space data and vice versa. Rootkits patching the kernel (DKOM, SSDT hooks, IDT hooks) became a persistent threat.

Microsoft's response, beginning with Windows 10, was Virtualization Based Security — using the hardware virtualization extensions (Intel VT-x / AMD-V) to create a more fundamental security boundary. VBS runs the regular Windows OS kernel (NT kernel) as a guest virtual machine, while a more privileged Secure Kernel (SK) runs in what Microsoft calls "Virtual Trust Level 1" (VTL 1). The regular OS and its kernel execute in "VTL 0."

The Hypervisor (HVCI, the Hypervisor-Protected Code Integrity component) enforces that only signed code can execute in VTL 0, even at ring 0. Even if an attacker achieves kernel code execution in VTL 0, they cannot modify kernel data structures that the Secure Kernel is watching, because the Hypervisor's second-level address translation (SLAT) tables protect those pages.

What Are VBS Enclaves?

VBS Enclaves (introduced in Windows 11 / recent Windows 10 builds) extend the VTL model to user-space. An enclave is a memory region within a user-space process that executes in a VTL 1 context — meaning the regular OS kernel (VTL 0, ring 0) cannot read, write, or execute the enclave's memory pages. Even a fully compromised OS kernel is unable to examine enclave contents.

The design intent is analogous to Intel SGX (Software Guard Extensions): provide a hardware-backed confidential computation environment for sensitive operations like cryptographic key storage, DRM, or enterprise credential management. Unlike SGX (which requires specific Intel hardware support and has been plagued by side-channel attacks), VBS Enclaves work on any hardware that supports Windows VBS, which is a much larger installed base.

From a developer perspective, creating a VBS Enclave involves writing a DLL with specific entry points, signing it with an enclave certificate, and loading it into a process using the CreateEnclave API. The enclave DLL executes in VTL 1; it communicates with the VTL 0 host process through controlled call gates (EnclaveCallFunction, TerminateEnclave).

The Security Paradox

Here is the paradox that Ori David identified: the properties that make VBS Enclaves secure for legitimate use — opacity to the OS kernel, protection from debugging and inspection, execution in a trusted but isolated context — are precisely the properties that make them valuable for malware. A malicious VBS Enclave would be:

• Invisible to most EDR/AV products, which rely on kernel-level telemetry (ETW, kernel callbacks, ESF-equivalent) that operates in VTL 0 and cannot see into VTL 1
• Protected from kernel-mode memory scanning, a standard technique for detecting in-memory malware payloads
• Immune to most debugging and analysis attempts by security tools operating in the regular OS context
• Persistent and stealthy because the enclave is loaded by a legitimate host process and its pages are indistinguishable (to VTL 0) from normal protected memory

Key Findings

▶ Watch: Exploit chain: using VBS Enclaves for privilege escalation (2:51)

• VBS Enclaves can be weaponized to hide malicious code from virtually all current endpoint security tooling. Because security products run in VTL 0, they cannot inspect VTL 1 enclave memory, rendering in-memory code scanning ineffective against enclave-hosted malware.

• An attacker-controlled enclave can be loaded by a legitimate host process, making the malicious enclave's presence a legitimate-looking memory allocation from the OS's perspective.

• Code executed within a VBS Enclave is protected from debugging and emulation by standard analysis tools (WinDbg, x64dbg, PE-Sieve, Moneta, etc.), significantly impeding reverse engineering of enclave-based malware.

• The enclave signing requirement can be circumvented. Ori David demonstrates that the signing requirements for VBS Enclaves are weaker than expected and can be bypassed, removing the intended barrier to loading unsigned malicious enclave code.

• VBS Enclaves can be used for C2 communication, with the enclave handling decryption and execution of commands while the host process handles network I/O, splitting malicious functionality across VTL boundaries.

• Current EDR solutions have no visibility into enclave execution, meaning behavior-based detection (process creation, file writes, network connections) triggered from within an enclave may not be attributed to the enclave code.

Technical Deep Dive

▶ Watch: Weaponizing VBS Enclaves: turning security features into attack primitives (12:29)

VBS Architecture Refresher: VTL 0 vs. VTL 1

To understand enclave security properties, it is necessary to understand the VTL boundary. The Hypervisor enforces that:

• VTL 0 (normal OS) cannot directly modify VTL 1 memory pages.
• SLAT (Second Level Address Translation, i.e., Extended Page Tables on Intel) enforces memory access control between VTLs at the hardware level.
• Transitions from VTL 0 to VTL 1 (for enclave calls) and from VTL 1 to VTL 0 (returns) are mediated by the Hypervisor via secure call gates.

A VTL 0 thread making an enclave call (EnclaveCallFunction) causes a VTL switch: the thread's execution context transitions to VTL 1, runs enclave code, then returns to VTL 1. During this period, the VTL 0 OS kernel cannot observe or modify the thread's execution in the enclave — even with a kernel debugger attached.

Signing Bypass

VBS Enclaves require signing with a certificate that chains to a specific CA. The intent is that only authorized parties can create enclaves. Ori David analyzed the signing verification logic and found that the requirements are less restrictive than the documentation suggests:

• The certificate used for enclave signing does not require an EV code signing certificate or a special Microsoft-issued enclave certificate in all configurations.
• In development mode (an OS configuration that can be enabled by an administrator), enclave signing requirements are significantly relaxed, allowing self-signed certificates.
• Even in non-development configurations, the specific certificate attributes checked are permissive enough that an attacker with a standard code signing certificate (obtainable commercially or through certificate authority compromise) can satisfy the requirements.

The researchers demonstrate creating a functional VBS Enclave using a test certificate obtained without going through Microsoft's enclave authorization process, loading it into a process on a standard Windows 11 installation.

Building a Malicious Enclave: Architecture

Ori David presents a complete malicious enclave architecture:

Host Process (VTL 0):

• A legitimate-looking process (injected into an existing process or a custom loader disguised as a system component)
• Handles network I/O: receives encrypted C2 traffic
• Passes encrypted C2 commands to the enclave via EnclaveCallFunction
• Receives results from enclave calls and transmits them back to the C2 server

Malicious Enclave (VTL 1):

• Decrypts C2 commands using a key stored entirely within the enclave (never visible to VTL 0)
• Executes commands or shellcode within VTL 1 (invisible to EDR)
• For operations requiring VTL 0 interaction (file I/O, process creation), calls back to the host process via VTLCALL return mechanisms
• Manages its own in-memory state for persistence of session data across commands

This architecture means that the most sensitive parts of the malware — decryption keys, shellcode, C2 parsing logic — are entirely hidden from security tools. The VTL 0 host process only ever handles encrypted blobs, which look like legitimate encrypted network traffic to any monitoring tool.

EDR Visibility Gap Analysis

Ori David systematically maps what current EDR products can and cannot see:

• ETW (Event Tracing for Windows) providers in VTL 0: Cannot observe events generated by VTL 1 execution. ETW is a VTL 0 mechanism.
• Kernel callbacks (PsSetCreateProcessNotifyRoutine, etc.): Fire for process/thread creation, but a thread switching to VTL 1 to execute enclave code does not generate a new thread creation callback. The VTL switch is invisible.
• Memory scanning (PE-Sieve, Moneta, in-memory IOC scanning): VTL 1 enclave pages are not readable from VTL 0 scanners. Attempts to read enclave memory return zeros or cause access violations, not the actual enclave content.
• API hooking (user-space DLL injection, IAT patching): Only applies to VTL 0 code. Code executing in VTL 1 is not subject to VTL 0 API hooks.
• Behavioral detection (watching for suspicious API call sequences): Can still observe VTL 0 API calls made by the host process on behalf of the enclave (file writes, network connections, process creation), but cannot correlate these to the enclave logic that triggered them.

The conclusion is that enclave-based malware can operate with near-total opacity to current EDR tooling, with detection limited to coarse behavioral signals from the VTL 0 host process.

Killing Enclaves Is Also Hard

Once a VBS Enclave is running and an attacker has used it to establish persistence, even terminating the enclave is non-trivial. The enclave runs with VTL 1 privileges; a VTL 0 process (even with SYSTEM privileges) cannot forcibly read or terminate the enclave's execution without going through the TerminateEnclave API. If the malicious host process has tampered with this code path, or if the enclave has established persistence mechanisms in VTL 1, clean remediation requires specific tooling that does not currently exist in mainstream security products.

Demo / Proof of Concept

▶ Watch: Bypassing VBS isolation boundaries (17:29)

The DEF CON 33 demo includes:

1. Enclave creation and signing bypass: Loading a self-signed VBS Enclave on a standard Windows 11 system, demonstrating that the signing enforcement is insufficient.

1. Memory opacity demonstration: Loading a beacon (simulated malware payload) inside a VBS Enclave, then running PE-Sieve and Moneta (widely used in-memory malware scanning tools) against the host process — demonstrating that the enclave payload is completely invisible to both tools.

1. C2 communication through the enclave: Demonstrating an encrypted C2 session where commands are decrypted and processed inside the enclave, with only encrypted traffic visible to network monitoring tools.

1. Behavioral attribution gap: Executing a file-write operation from within the enclave (via a VTL 0 callback to the host process), then showing that EDR telemetry attributes the write to the host process but provides no indication that the triggering logic was inside an enclave.

Defensive Implications

▶ Watch: CVE disclosures for VBS Enclave vulnerabilities (24:39)

For enterprise defenders:

• VBS Enclave-aware detection needs to be developed. The most practical near-term approach is monitoring for CreateEnclave API calls from unexpected processes — enclave loading is unusual enough that it is a meaningful behavioral signal.
• Enclave loading events should be logged (via ETW providers that cover the VirtualAlloc type calls underlying enclave creation, or Windows Security Event Log if coverage is available).
• Since the enclave signing bypass relies on development mode or weakly-enforced certificate requirements, enterprise policies should disable development mode (the "AllowDevOemUnlock" registry key and equivalent MDM settings) on production endpoints.
• Enforce Credential Guard (which uses VTL 1 to protect credentials) and monitor its status — attackers who attempt to abuse VBS Enclaves in environments with Credential Guard may trigger diagnostic events.

For Microsoft:

• The signing enforcement for VBS Enclaves should be strengthened to require Microsoft-issued or validated certificates, similar to the KMCS (Kernel Mode Code Signing) requirements for drivers.
• ETW providers should be extended to cover VTL 1 events at least at the enclave lifecycle level (creation, termination, call transitions) to give security tools visibility into enclave usage without exposing protected memory contents.

For EDR vendors:

• Develop VBS Enclave-aware detection capabilities. At minimum, flag processes that create enclaves as high-interest for additional monitoring of their VTL 0 activities.
• Investigate Hypervisor-based monitoring approaches that could observe VTL 1 activity at a higher privilege level than VTL 1 itself.

Key Takeaways

1. VBS Enclaves' security properties — memory opacity, execution isolation, protection from debugging — make them a powerful evasion primitive for malware authors.
2. Current endpoint security tooling (EDR, AV, memory scanners) has essentially no visibility into VBS Enclave execution, creating a detection dead zone.
3. The enclave signing requirement can be bypassed, removing the main barrier to loading malicious enclave code.
4. A complete C2 architecture hiding decryption logic and shellcode in a VBS Enclave while using a legitimate host process for I/O is technically feasible and demonstrated.
5. Defensive responses require both product-level updates (EDR enclave awareness) and policy controls (restricting development mode, monitoring CreateEnclave API usage).

About the Speaker(s)

▶ Watch: Microsoft's response and mitigations (42:29)

Ori David is a security researcher at Akamai's security research team, where he works on a broad range of topics in offensive security, threat hunting, and vulnerability research. He has a background in red teaming and specializes in Windows internals and advanced evasion techniques. His research on VBS Enclaves at DEF CON 33 represents one of the first comprehensive offensive security analyses of this Microsoft security feature, exploring the gap between what the technology promises as a defensive tool and what an attacker can do when those same properties are turned against defenders.