TEE-SHirT: Scalable Leakage-Free Cache Hierarchies for TEEs

TEE-SHirT: Scalable Leakage-Free Cache Hierarchies for TEEs

Speakers: Kerem Arikan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=gjhfqtmuWf4

Overview

The talk "TEE-SHirT: Scalable Leakage-Free Cache Hierarchies for TEEs," presented by Kerem Arikan at the NDSS Symposium, addresses a fundamental and persistent challenge in secure computing: protecting Trusted Execution Environments (TEEs) from cache side-channel attacks. These attacks exploit the physical sharing of cache resources to infer sensitive information, effectively compromising the logical isolation that TEEs like Intel SGX are designed to provide. Recent demonstrations of such attacks against SGX underscore the urgent need for robust physical isolation mechanisms to complement the existing logical safeguards. TEE-SHirT proposes a comprehensive security framework to integrate leakage-free cache hierarchies directly into TEE designs.

The core problem, as highlighted by Arikan, is that existing secure cache partitioning schemes often focus on isolated components—either private upper-level caches or shared Last-Level Caches (LLCs)—leaving the entire hierarchy vulnerable. TEE-SHirT argues for a holistic approach, designing a multi-level cache system that combines a shared, partitioned LLC, private, partitioned L2 caches, and private L1 caches that are flushed on context switches and system calls. This ambitious goal introduces significant challenges related to performance, consistency, and coherence, particularly in multi-threaded environments, which the TEE-SHirT framework endeavors to resolve through novel contributions in hardware virtualization, multi-threaded support, and formal security modeling.

Background

▶ Watch: Introduction and motivation for secure cache hierarchies (0:00)

Modern CPUs commonly feature a multi-level cache hierarchy, typically comprising private L1 and L2 caches per core, and a shared L3 cache, often referred to as the Last-Level Cache (LLC). While L1 and L2 caches can be shared by Simultaneous Multi-threading (SMT) threads, SMT is frequently disabled in security-sensitive contexts to mitigate certain side-channel risks. In an SMT-enabled scenario, private caches can be statically partitioned by ways to offer some level of isolation.

The primary mechanism TEE-SHirT builds upon for security is hardware-supported cache partitioning, drawing inspiration from prior work like Composable Cachelets. The fundamental concept is to create isolated Cache Partitions (CPs) for each enclave. A CP is a dedicated region within the cache, defined by a specific number of consecutive sets and one or more ways. The metadata required for managing these partitions is stored in a hardware component known as the Cache Partition Table (CPT). This metadata, referred to as Cache Partition Table Context (CPTC), includes crucial information such as mappings between original set indices and indices within the allocated CPs, as well as a way mask that specifies which cache ways belong to a particular CP.

The address remapping mechanism works by using higher bits of a memory access's set index to access an entry in the CPT. This entry contains a partition index, which points to the group of sets allocated to the enclave's CP. The lower bits of the set index then function as an offset within this partition. Concurrently, the way mask ensures that an enclave's memory accesses are confined to its dedicated, isolated cache region, preventing contention and leakage from other enclaves or the untrusted operating system.

TEE-SHirT's CPTC virtualization strategically leverages existing Intel SGX data structures. Enclave metadata in SGX is stored within the Enclave Page Cache (EPC), residing in the Processor Reserved Memory (PRM). The Enclave Page Cache Map (EPCM) contains entries detailing EPC pages, their types, and permissions. Key EPC pages include the Enclave Control Structure (SECS), which holds basic enclave information like size and base address, and the Thread Control Structure (TCS) for multi-threaded enclaves, which points to thread-specific EPC pages. The Save State Area (SSA) stores an enclave's execution context upon hardware exceptions.

The threat model for TEE-SHirT assumes a powerful adversary, consistent with the Intel SGX threat model. This adversary is capable of exploiting untrusted system software (OS, hypervisor) and can mount any cache side-channel attack, including sophisticated techniques like Prime+Probe, potentially across multiple cache levels and multiple cores. TEE-SHirT is designed to protect against these scenarios. While direct Denial-of-Service (DoS) attacks are not the primary focus, the design reserves at least one cache way for non-enclave programs, ensuring they retain some cache access without remapping and are not completely starved.

A critical argument presented is the necessity of partitioning L2 caches, even in hierarchies where L1 caches are flushed and LLCs are partitioned. Flushing a non-partitioned L2 cache on context switches would impose a severe performance penalty due to its significantly larger size and higher miss latencies compared to L1. Experimental results presented in the talk vividly illustrate this: while L1 data cache flushes resulted in an IPC impact of no more than 2% for all tested benchmarks, L2 cache flushes caused IPC slowdowns ranging from 6% to 53% for a 25ms time quantum, rendering L2 flushing infeasible for practical performance.

Furthermore, in inclusive cache hierarchies, where data in an upper-level cache (e.g., L2) is guaranteed to also exist in a lower-level cache (e.g., LLC), an unpartitioned L2 introduces additional problems. If only the L3 cache is partitioned, and an allocated L3 partition is smaller than the L2, this creates an effective reduction in L2 capacity, or forces L3 partition sizes to be undesirably large. Partitioning L2 caches is therefore crucial to maintain effective cache capacity, avoid performance degradation, and ensure scalability in a high-performance TEE system.

Key Findings

▶ Watch: Detailed explanation of cache partitioning mechanism (4:00)

TEE-SHirT introduces several pivotal contributions to secure computing, addressing the long-standing challenge of cache side-channels in TEEs:

1. CPTC Virtualization for Scalability: The design innovatively virtualizes hardware structures for cache partitioning metadata (CPTCs), storing them in Intel SGX's Processor Reserved Memory (PRM). This eliminates the need for costly cache flushes on context switches and system calls, allowing enclaves to retain their cache contents across scheduling events. This approach scales beyond the physical hardware capacity of CPTs, significantly improving performance for system call-intensive workloads and enabling a larger number of concurrent enclaves.
2. Robust Multi-threaded Enclave Support: TEE-SHirT provides solutions for the complex coherence and consistency issues that arise when multi-threaded enclaves utilize partitioned caches. It addresses inter-core data sharing through snooping-CPTs, which cache frequently accessed CPTCs to expedite coherence queries, and intra-core data sharing through partition merging, where CPTCs of threads from the same enclave on the same core are combined to ensure a single instance of data and prevent version inconsistencies.
3. Comprehensive Multi-Level Cache Partitioning: The work rigorously demonstrates the critical need to partition L2 caches in addition to LLCs, especially in inclusive cache hierarchies. This prevents significant performance degradation and maintains effective cache capacity, which would otherwise be compromised by the infeasibility of L2 flushing and the interaction between cache levels.
4. Formal Security Model: TEE-SHirT is backed by a rigorous formal security model, based on cache-aware and enclave-aware operational semantics. This model allows for formal reasoning about the security guarantees, specifically asserting immunity against side-channel attacks in a multi-level cache hierarchy, including support for coherence protocols and context switches, a significant extension over prior work.
5. Low Hardware Overhead: Despite its sophisticated design, TEE-SHirT achieves its security and performance benefits with minimal hardware area overhead. For a practical configuration supporting multiple concurrent enclaves, the design incurred only 0.7% additional Look-Up Tables (LUTs) and 1.3% additional Flip-Flops (FFs) in an FPGA implementation, demonstrating its viability for real-world deployment.

Technical Deep Dive

▶ Watch: Assumed threat model and side-channel attacks (5:00)

TEE-SHirT's core design philosophy revolves around coordinated partitioning across the entire cache hierarchy. Specifically, it employs partitioned Last-Level Caches (LLCs) and L2 caches, while private L1 caches are robustly flushed on every context switch and system call. The sizes of partitions at each cache level can be independently controlled, offering flexibility to enclaves which request CP allocations via specialized instructions.

At the heart of this architecture are Cache Partitioning Management Units (CPMU), integrated into each cache level (except L1). The CPMU is responsible for managing allocation requests, mapping them to specific way-set partitions within the cache. The metadata for each CP, the Cache Partition Table Context (CPTC), is handled by a dedicated CPTC Handler integrated with the Memory Management Unit (MMU), making its operations transparent to the untrusted operating system. When an enclave requests a CP, a physical region within the cache is assigned. To ensure strict isolation and consistency, any dirty data residing in this newly allocated region from prior programs must be written back to the lower-level cache or main memory, and existing cache lines must be invalidated. This crucial process, termed gang-invalidation, guarantees that the new CP mapping only becomes effective once all previous operations are complete and all pending cache misses are resolved, preventing data leakage or corruption.

CPTC Virtualization

A significant challenge in scaling cache partitioning is the management of CPTCs across numerous enclaves. Maintaining a dedicated physical CPT for every potential enclave within the cache hardware is resource-intensive and limits concurrency. Without virtualization, an enclave might face delays or loss of cache contents if physical CPTs are exhausted.

TEE-SHirT's innovative solution is CPTC Virtualization. It leverages existing Intel SGX data structures by storing CPTCs in the Processor Reserved Memory (PRM). A CPTC is only loaded into the active CPT of a core when its corresponding enclave is actively executing. When an enclave is context-switched out, its CPTC is removed from the active CPT but persists in PRM. This design offers two key benefits: it preserves the enclave's data in the cache across context switches, which is crucial for performance in system call-intensive workloads, and it significantly reduces the number of physical CPTs required in the remapping logic, thereby improving scalability.

To facilitate CPTC virtualization without relying on untrusted software, TEE-SHirT extends SGX-reserved memory areas and EPCM entries. The Thread Control Structure (TCS) is extended with an Offset of the CPTC State Save Area (OCPTCSSA), which points to dedicated CPTCSSA pages in PRM where the CPTC is stored. These CPTCSSA pages are protected with read-only permissions to prevent malicious software modification. Additionally, an auxiliary field, CPALLOCATED, is added to the SECS page to track the total CP sizes allocated by an enclave for each cache level.

Partition-Aware Context Switches

The process of a cross-core context switch for a single-threaded enclave under TEE-SHirT is meticulously designed to maintain security and performance:

1. A baseline context switch is initiated by the OS.
2. An enclave allocation instruction is processed.
3. The L1 cache is flushed to clear any residual data.
4. Allocation data for lower-level caches (L2 and LLC) is forwarded to the L2 CPMU.
5. The L2 CPMU checks for available space, loads the new CPTC into its CPT, and sends a CP deallocation message to other same-level caches if necessary.
6. The L2 cache forwards the new CPTC information to the LLC.
7. The LLC propagates the request to the MMU.
8. The MMU loads the OCPTCSSA from PRM.
9. The CPTC Handler, using a state machine and dedicated buffers, reads the OCPTCSSA and updates the CPTCSSA pages in PRM.
10. A completion response is sent back to the core.
11. Crucially, the core waits for the gang-invalidation of non-enclave data within the newly allocated cache region to complete. This final step ensures data integrity and strict isolation before the enclave begins execution with its new partition.

Multithreaded Enclaves with Partitioned Caches

Supporting multi-threaded enclaves with partitioned caches introduces complex challenges for data sharing, particularly with snooping-based MOESI coherence protocols.

Challenge 1: Inter-Core Data Sharing:

Traditional cache coherence protocols rely solely on cache line addresses for snooping requests. However, TEE-SHirT's CPTCs deflect accesses, meaning a snoop request based only on an address will fail if the target core's cache cannot access the line within its partition boundary without the requesting thread's CPTC. This would necessitate costly CPTC loading or waiting.

TEE-SHirT's solution is snooping-CPTs. These are additional CPTs embedded within the cross-core interconnect fabric, acting as a cache-like optimization. They store frequently accessed CPTCs, allowing coherence queries to be resolved quickly. If a CPTC is not found in the snooping-CPT, it is loaded from the CPTCSSA pages in PRM. This allows a snoop request, augmented with the retrieved CPTC, to successfully access a shared cache line in another core's partition without significant overhead.

Challenge 2: Intra-Core Data Sharing:

When multiple threads belonging to the same enclave are scheduled on the same core, they might independently allocate CPs. If these threads attempt to access the same memory address through separate partitions within the same cache level, consistency issues arise. A snoop would miss, as the system cannot retrieve data from a separate partition belonging to another thread of the same enclave without iterating through all partitions, which is prohibitively expensive.

The proposed solution is partition merging. When multiple threads of the same enclave are scheduled on the same core, the CPMU combines their individual CPTCs into a single, merged CPTC. This merged CPTC is then used by all threads of that enclave on that core, and their CPTCSSA pages are updated. This ensures that all threads access a common partition, guaranteeing a single instance of a data line and preventing version inconsistencies.

For directory-based coherence mechanisms, similar principles apply. Directory entries could be augmented with supplementary CPT entries for Modified, Owned, and Exclusive states for inter-core sharing. Partition merging remains relevant for intra-core consistency. The talk notes that existing directory-based coherence mechanisms like SecDir can be integrated with TEE-SHirT.

Demo / Proof of Concept

▶ Watch: Problem: unpartitioned L2 in inclusive hierarchies (7:30)

While the talk did not feature a live, interactive demonstration, the authors presented a comprehensive evaluation of TEE-SHirT's design through extensive experimental methodology and detailed results. This evaluation serves as the practical proof of concept for the proposed architecture.

The performance of TEE-SHirT was assessed using the gem5 cycle-accurate simulator in system call emulation mode. A diverse set of benchmarks was employed to cover various memory demands:

• MiBench suite: 34 benchmarks.
• SPEC2017: 12 benchmarks.
• WolfSSL cryptography library functions: 2 functions.

The primary performance metric used was Instructions Committed Per Cycle (IPC). For experiments evaluating CPTC virtualization, single-core simulations of SPEC2017 and MiBench were conducted, emulating context switches with time quantums of 1ms, 10ms, and 25ms. The simulations also accounted for gang-invalidation and cache flushes on system calls.

To evaluate the overall TEE-SHirT performance in a multi-core setting, a 4-core system was simulated, running five different mixes of SPEC2017 programs concurrently. Realistic assumptions were made for cache latency, adding an additional two cycles to L2 and L3 cache latency (one for remapping logic, one for replacement logic complexity), consistent with prior work. L1 cache flushing was modeled with a 25ms time quantum.

To quantify the hardware overhead, parts of the TEE-SHirT design were implemented in the open-source ESP SoC platform, targeting an AMD Virtex UltraScale+ VCU118 FPGA. The evaluation measured the increase in LUTs (Look-Up Tables) and FFs (Flip-Flops) utilization relative to a baseline SoC equipped with a 512KB L2 cache per core and a shared 4MB LLC.

Evaluation Results: Performance

CPTC Virtualization Performance:

The benefits of CPTC virtualization were evident, particularly for system call-intensive benchmarks. Compared to flushing LLC partitions on context switches:

• For a 1ms time quantum, significant speedups were observed: cactusBSSN (4.4%), omnetpp (10.8%), wrf (4.5%), xalancbmk (3.7%), and cam4 (10.4%).
• Even for a more conventional 25ms time quantum, omnetpp still yielded a 10.3% speedup.

This confirmed that retaining cache data across context switches due to CPTC virtualization substantially improves performance for workloads with frequent system calls.

Cache Coherence Optimizations Performance:

The effectiveness of snooping-CPTs in mitigating snoop delays for multi-threaded enclaves was demonstrated:

• Without snooping-CPTs, benchmarks like fluidanimate and raytrace showed average snoop delays of 1.91 times and 2.11 times the baseline, respectively.
• With snooping-CPTs enabled, these delays were drastically reduced to 1.08 times for fluidanimate and 1.09 times for raytrace. This significant improvement was attributed to high snooping-CPT hit rates, ranging from 91.5% to 93.7%, validating the optimization.

Overall Performance of TEE-SHirT:

• Cryptography Programs: For the smallest configuration (16KB CPs for L2 and L3), some benchmarks experienced high slowdowns, notably rijndael (96.3%) and xz (MiBench) (96.9%). However, by increasing the L3 cache CP to 32KB, slowdowns were significantly reduced to below 0.5% for all security benchmarks. This highlights that small CPs can be effective at the LLC level when L2 is also partitioned. An interesting observation was that equal L2/L3 CP sizes could sometimes perform worse than smaller L2 CPs with larger L3 CPs, due to inclusivity effects potentially adding redundant L3 access latency.
• SPEC2017 Programs (Large Workloads): In the smallest configuration, mix-2 and mix-4 showed the highest degradation, with 26.4% and 28% performance loss, respectively. However, when a more substantial 256KB CP was allocated in L2 and a 512KB CP in L3 (representing a quarter of the total cache sizes per enclave), all benchmarks experienced slowdowns lower than 3%. This demonstrates TEE-SHirT's ability to achieve robust security with modest performance overhead for large, realistic workloads.

Evaluation Results: Area Overhead

The FPGA implementation revealed minimal hardware overhead:

• A practical (1, 4) configuration (1 CPT for L2, 4 for LLC), supporting fewer concurrent enclaves, incurred only 0.7% additional LUTs and 1.3% additional FFs.
• The largest configuration, (8, 32), resulted in a 2.1% increase in LUTs and a 6.7% increase in FFs. The higher increase in FF utilization is expected given the register-centric nature of the CPT. Importantly, the cache's data and metadata are stored in Block RAMs, which are separate and unaffected by TEE-SHirT's implementation. CPTC virtualization is key to enabling these low-area configurations with minimal performance impact.

Defensive Implications

▶ Watch: TEE-SHirT high-level design and architecture (9:00)

TEE-SHirT presents a significant advancement in the defense against cache side-channel attacks targeting Trusted Execution Environments. For defenders, the implications are profound, fundamentally shifting the paradigm from reactive software mitigations to proactive hardware-enforced isolation.

Firstly, the research unequivocally establishes the criticality of comprehensive multi-level cache protection. Defenders can no longer assume that merely flushing L1 caches or partitioning the LLC is sufficient. The demonstration of severe performance penalties for L2 flushing and the issues arising from inclusivity in unpartitioned L2 caches highlight a systemic vulnerability that requires hardware-level intervention across the entire cache hierarchy. Organizations deploying TEEs should advocate for and prioritize hardware designs that incorporate such multi-level partitioning.

Secondly, TEE-SHirT's design provides a blueprint for scalable and high-performance leakage-free TEEs. When adopted in future CPU architectures, technologies like TEE-SHirT would dramatically enhance the security posture of enclaved applications. Defenders would gain significantly stronger guarantees against cache-based information leakage, making it substantially harder for sophisticated adversaries to compromise TEE-protected secrets. The formal security model accompanying TEE-SHirT provides a rigorous mathematical proof of side-channel immunity, offering a level of assurance rarely achieved by heuristic or probabilistic software-based mitigations. This formal backing means that, for systems implementing TEE-SHirT's principles, defenders can have high confidence in the isolation properties of their TEEs.

Thirdly, the specific technical solutions—CPTC virtualization, snooping-CPTs, and partition merging—offer concrete mechanisms for addressing the practical challenges of deploying secure cache hierarchies, such as managing context switches and supporting multi-threaded workloads. Defenders can leverage this understanding to evaluate the robustness of future TEE hardware features or contribute to the design of secure system architectures. The minimal hardware overhead demonstrated (e.g., 0.7% additional LUTs, 1.3% additional FFs) makes a strong case for the practicality and deployability of such hardware-backed security features.

In essence, TEE-SHirT empowers defenders by offering a vision for TEEs where cache side channels are eliminated by design, rather than merely mitigated. This shift from "best effort" to "provably secure" hardware-enforced isolation is a game-changer for critical applications running in TEEs.

Key Takeaways

• Comprehensive Multi-Level Cache Partitioning is Essential: Protecting TEEs from cache side-channel attacks requires coordinated partitioning across the entire cache hierarchy, including shared Last-Level Caches (LLCs) and private L2 caches, not just L1 flushing.
• CPTC Virtualization Enables Scalability and Performance: By storing Cache Partition Table Contexts (CPTCs) in Intel SGX's Processor Reserved Memory (PRM), TEE-SHirT avoids costly cache flushes on context switches, preserving cache contents and significantly boosting performance for system call-intensive workloads.
• Novel Coherence Mechanisms Support Multi-threaded Enclaves: To handle complex data sharing, TEE-SHirT introduces snooping-CPTs for efficient inter-core coherence and partition merging for consistent intra-core data access within multi-threaded enclaves.
• L2 Cache Partitioning Prevents Performance Degradation: Flushing L2 caches is prohibitively expensive (up to 53% IPC slowdown), and unpartitioned L2s in inclusive hierarchies lead to capacity reduction; thus, L2 partitioning is crucial for performance and scalability.
• Formal Security Model Provides Rigorous Guarantees: TEE-SHirT is backed by a formal security model that rigorously proves immunity against cache side-channel attacks in complex multi-level cache hierarchies, including support for coherence and context switches.
• Achieves Security with Minimal Hardware Overhead: The robust security and performance benefits of TEE-SHirT are achieved with remarkably low hardware overhead, exemplified by a practical configuration incurring only 0.7% additional LUTs and 1.3% additional FFs in an FPGA implementation.

About the Speaker(s)

Kerem Arikan, the presenter of "TEE-SHirT: Scalable Leakage-Free Cache Hierarchies for TEEs" at the NDSS Symposium, is a researcher deeply engaged in the field of secure computing. Based on the technical depth and specific focus of this talk, Arikan's expertise lies in hardware-software co-design for trusted execution environments, particularly in developing architectural solutions to mitigate side-channel attacks. His work, as demonstrated by TEE-SHirT, involves intricate knowledge of cache hierarchies, memory management units, cache coherence protocols, and the formal verification of security properties for complex systems. While a specific title or affiliation was not provided in the transcript or metadata, the research presented positions him as a significant contributor to the ongoing effort to enhance the physical security and performance of TEEs.