CMASan: Custom Memory Allocator-aware Address Sanitizer

CMASan: Custom Memory Allocator-aware Address Sanitizer

Speakers: Junwha Hong; Wonil Jang; Mijung Kim; Lei Yu; Yonghwi Kwon; Yuseok Jeon

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=Y9H7IxsTcgA

Overview

In the realm of software security, memory safety bugs remain a persistent and critical threat. Tools like AddressSanitizer (ASan) have become indispensable for detecting common memory errors such as buffer overflows, use-after-free (UAF), and double-free vulnerabilities. However, ASan's effectiveness is predicated on its ability to intercept and replace standard memory allocation functions like malloc and free. This talk, presented by Junwha Hong from Unist, along with collaborators from Rula Poly Techch Institute and the University of Maryland, unveils a significant blind spot in ASan's coverage: Custom Memory Allocators (CMAs).

The presentation introduces CMASan, a novel approach designed to extend ASan's robust memory bug detection capabilities to objects managed by CMAs, without requiring manual code modifications or specialized ASan expertise. The research highlights that a substantial number of real-world applications employ CMAs, rendering a significant portion of their memory operations invisible to traditional ASan. CMASan addresses this by intelligently identifying CMAs, instrumenting their APIs, and employing sophisticated metadata management and quarantine mechanisms to ensure comprehensive bug detection. This work is crucial for enhancing the security posture of complex software systems that rely on optimized or specialized memory management strategies, demonstrating its impact by uncovering previously unknown vulnerabilities in widely used applications.

Background

▶ Watch: ASan's blind spot: Custom Memory Allocators (CMAs) (0:40)

Memory safety errors, such as buffer overflows, use-after-free, and double-free, constitute a significant class of vulnerabilities that attackers frequently exploit to gain control over systems or leak sensitive information. To combat these issues, dynamic memory error detection tools have been developed, with AddressSanitizer (ASan) emerging as one of the most widely adopted and effective solutions in practical software development and testing. ASan operates by instrumenting compiled code to insert checks around memory accesses. It surrounds each allocated object with redzones to detect out-of-bounds accesses (buffer overflows) and poisons memory regions after they are freed, placing them in a quarantine zone to delay reuse and extend the detection window for use-after-free and double-free errors. To achieve this, ASan typically replaces standard system allocators like malloc, free, realloc, and calloc with its own internal, instrumented versions. This interception allows ASan to manage shadow memory, track object states, and enforce its safety mechanisms.

However, the efficacy of ASan hinges on its ability to control the memory allocation lifecycle. A critical limitation arises when target applications utilize Custom Memory Allocators (CMAs) instead of, or in addition to, standard library functions. CMAs are often implemented by developers for performance optimization, specific memory layout requirements, or to manage memory within particular architectural constraints (e.g., embedded systems, game engines, databases). Because ASan is designed to replace standard allocators, it cannot automatically intercede with CMAs. Consequently, memory objects managed by CMAs are invisible to ASan, and any memory safety bugs affecting these objects go undetected, leading to false negatives.

To quantify the prevalence and impact of this issue, the researchers analyzed 100 real-world projects and discovered that 78 of them employed CMAs that could cause false negatives in ASan. These CMAs predominantly fell into two patterns:

1. Arena Allocators: These CMAs allocate a large contiguous block of memory, known as an arena, and then sub-allocate smaller objects within this region. Since ASan only replaces standard allocators, it is unaware of the individual objects within the arena. As a result, ASan fails to place redzones around these CMA objects, making it unable to detect buffer overflows that occur between objects within the same arena.
2. Recycler Allocators: These CMAs are designed for efficiency by repeatedly allocating and freeing objects without returning the underlying memory to the standard system allocator. Instead, they often maintain a pool of freed objects for immediate reuse. Because ASan's poisoning and quarantine mechanisms rely on its internal free replacement, it cannot poison or quarantine objects managed by recycler CMAs. This leads to missed use-after-free and double-free detections for objects that are rapidly recycled internally.

Previous attempts to address ASan's limitations with CMAs have been proposed, each with its own set of challenges:

• Shim-based Solutions: One approach involves providing a "shim" layer that attempts to switch CMAs to use standard allocators. This would allow ASan to replace these standard allocators and thus gain visibility. However, not all CMAs are compatible with this approach. For instance, CMAs that perform in-place resizing (where an object is resized at its current memory location) or those with a clear API that deallocates all objects in an arena at once, cannot be easily adapted to standard allocator semantics.
• ASan Poisoning APIs: ASan provides specific APIs (e.g., __asan_poison_memory_region, __asan_unpoison_memory_region) that allow developers to manually mark memory regions as poisoned or unpoisoned. While this offers direct control, it requires developers to manually insert these calls into the CMA's logic. This demands a deep understanding of the CMA's internal workings, careful manual securing of redzone space, and does not naturally integrate with ASan's robust quarantine mechanism.

Ultimately, both existing solutions necessitate significant manual code modifications and a sophisticated understanding of both the specific CMA implementation and ASan's internal mechanisms. This high barrier to entry limits their practical adoption, leaving a critical gap in memory safety coverage for a vast number of applications. CMASan steps in to fill this gap, providing an automated and comprehensive solution.

Key Findings

▶ Watch: CMASan: Detecting CMA bugs without code changes (2:40)

CMASan presents a significant advancement in memory safety tooling by effectively addressing the long-standing challenge of detecting bugs in custom memory allocators (CMAs). The key findings and contributions of this research are multi-faceted, demonstrating both the technical ingenuity of CMASan and its practical impact on real-world software security:

1. Automated Memory Bug Detection for CMAs: CMASan is designed to detect all types of CMA-related memory errors, including buffer overflows, use-after-free, and double-free, without requiring any manual code modification from developers or specialized expertise in ASan's internal workings. This automation significantly lowers the barrier to entry for securing applications that utilize CMAs.
2. Discovery of Previously Unknown Vulnerabilities: A compelling testament to CMASan's effectiveness is its ability to uncover 19 previously unknown memory bugs in widely used, real-world applications. Notably, this includes critical vulnerabilities that had remained undetected for nine years in SQLite3 and two years in PHP. This highlights that CMAs represent a significant blind spot for traditional memory safety tools and manual auditing efforts. Crucially, ASan and Go (another memory safety tool) missed all 19 of these bugs, underscoring CMASan's unique detection capabilities.
3. Enhanced Detection Coverage: CMASan dramatically improves ASan's detection coverage for CMA objects. The evaluation showed that CMASan successfully recognizes a vast number of CMA objects that ASan overlooks. Up to 87% of load and store checks were found to occur on CMA objects, indicating a substantial proportion of memory accesses that were previously unchecked by ASan. By bringing these objects under scrutiny, CMASan significantly expands the scope of memory safety enforcement.
4. Minimal Performance and Memory Overhead: Despite its comprehensive instrumentation and sophisticated tracking mechanisms, CMASan maintains remarkably low overheads. It incurs only a 1.096x performance overhead and a 1.148x memory overhead compared to standard ASan. This efficiency is attributed to its on-demand metadata storage design, making CMASan a practical solution for integration into development and testing pipelines without prohibitive resource costs.
5. Non-invasive Integration: CMASan achieves its extended coverage without the need to replace or modify the internal logic of the custom memory allocators themselves. This non-invasive approach ensures compatibility with a broad range of CMAs, including those with unique behaviors like in-place resizing or bulk clear operations, which were problematic for previous shim-based solutions.

In essence, CMASan successfully bridges a critical gap in memory safety tooling, making sophisticated bug detection accessible for applications leveraging CMAs. Its ability to automatically identify, instrument, and monitor CMA-managed memory with minimal overhead and demonstrable success in finding real-world vulnerabilities solidifies its position as a vital contribution to software security.

Technical Deep Dive

▶ Watch: How CMASan's quarantine zone enhances detection (4:40)

CMASan is built on top of AddressSanitizer (ASan) and meticulously engineered to overcome its limitations with custom memory allocators (CMAs). Its core methodology revolves around three main pillars: intelligent CMA identification, precise API instrumentation, and efficient runtime management of metadata and quarantine zones. These mechanisms operate without requiring manual code modifications or deep ASan expertise, making it a robust and practical solution.

CMA Identification

The first step for CMASan is to identify potential CMAs within the target application's source code. This is achieved by extending existing code rules, likely pattern-matching for common CMA function names (e.g., arena_alloc, pool_free, mem_acquire) or structures that indicate custom memory management. Once potential CMA candidates are identified, CMASan categorizes their family functions (e.g., alloc, free, realloc, clear) based on user responses or predefined heuristics. This categorization is crucial for applying the correct instrumentation strategy to each type of CMA API.

CMA API Instrumentation

After identifying and categorizing CMA functions, CMASan instruments these APIs to mirror ASan's behavior for standard allocators. This instrumentation is performed at the compiler level, much like ASan itself.

• alloc (and similar allocation APIs):
• Before the CMA's allocation call, CMASan instruments the size argument to secure additional space for redzones.
• Upon return, it poisons the right side of the newly allocated object (the redzone) to detect out-of-bounds writes.
• Crucially, it saves metadata about the allocated object, using its beginning address as a unique key. This metadata includes information like the object's size, its allocation status, and the ID of the CMA instance that allocated it.

• free (and similar deallocation APIs):
• When a free API is called, CMASan retrieves the corresponding metadata using the parameterized pointer as a key.
• It performs a check for double-free vulnerabilities: if the incoming object is already marked as free in the metadata, a double-free bug is detected.
• Subsequently, the object's memory region is poisoned to detect use-after-free attempts, and its metadata is updated to reflect its freed status, often by marking it for deletion or quarantine.

• realloc (and similar resizing APIs):
• The instrumentation for realloc is similar to alloc, but with a critical distinction to handle in-place resizing.
• CMASan compares the returned object pointer with the old object pointer.
• If realloc returns a new memory location, it means the old object was effectively freed and a new one allocated. In this scenario, CMASan poisons the old object's memory region.
• If realloc returns the same memory location (an in-place resize), the old object is not poisoned, as it's still legitimately in use, albeit with a potentially new size.

• clear API:
• Some CMAs, particularly arena allocators, offer a clear API that deallocates all objects within a specific arena simultaneously.
• To handle this, CMASan records all allocated objects within their corresponding allocation zone.
• When a clear API is invoked, CMASan iterates through all objects recorded in that zone and poisons them, updating their metadata accordingly, effectively performing a bulk deallocation and poisoning operation.

Efficient Metadata Storage

Tracking metadata for every CMA-managed object is essential but can be memory-intensive, especially given the potentially large number of small objects managed by CMAs. CMASan addresses this with an optimized metadata storage mechanism:

• Two-level Table: Recognizing that CMAs often manage their objects within relatively narrow and specific memory regions, CMASan utilizes a two-level table for metadata storage.
• On-demand Creation: The second-level tables are created on demand. This means that memory for metadata is allocated only for the specific CMA memory regions that are actively in use. This design significantly reduces memory overhead compared to a flat, sparse table that would allocate space for metadata across the entire address space.

Quarantine Zone Design

A robust quarantine zone is critical for detecting use-after-free and double-free bugs by delaying the reuse of freed memory. CMASan's quarantine zone is designed with two primary objectives: free delaying and instance passing.

• Free Delaying:
• The challenge with CMAs is that they might immediately recycle freed objects, shortening the detection window.
• CMASan introduces a free delaying quarantine zone that intercepts the object before it is returned to the CMA's internal free pool.
• When an object is "freed" by a CMA API, CMASan poisons it and pushes it into its internal quarantine zone.
• Simultaneously, it pops the first object from the quarantine zone (the oldest one) and passes this object to the CMA's actual free implementation, instead of the currently requested object.
• This elegant mechanism allows CMASan to maintain a quarantine delay without modifying the CMA's internal logic, ensuring the CMA eventually reclaims memory but only after a sufficient delay for bug detection.

• Instance Passing:
• A critical design consideration is preventing metadata corruption and incorrect behavior if a single, shared quarantine zone were used across all CMA instances. For example, free might mistakenly retrieve metadata for an object from a different CMA instance, leading to errors.
• To prevent this, CMASan implements instance-specific quarantine zones. Each object is placed into its own quarantine zone, uniquely identified by the CMA ID stored in its metadata.
• This ensures that objects from different CMA instances do not interfere with each other, maintaining the integrity of metadata and the correctness of bug detection, again without requiring modifications to the CMA's internal logic.

Avoiding False Positives

Instrumenting CMAs without modifying their internal logic introduces a risk of false positives due to legitimate CMA behaviors that might appear suspicious to ASan. CMASan incorporates several strategies to mitigate this:

1. Call Stack Suppression: CMASan suppresses ASan reports if a CMA API is present in the call stack. This is because CMAs may legitimately access memory regions that ASan has marked as poisoned (e.g., during internal management or cleanup), and distinguishing this from an actual bug is crucial.
2. Outermost CMA Instrumentation: To avoid incorrect resize detections and false double-free positives from nested CMA calls (where one CMA might internally use another), CMASan activates instrumentation only for the outermost CMA. This ensures that the primary memory management layer is monitored, preventing redundant or misleading reports from internal, legitimate operations.
3. Size Querying APIs: When CMAs return the size of an object via size querying APIs, CMASan ensures that the original size of the object is returned. This is important because ASan's redzones are effectively part of the allocated memory, and reporting the instrumented size (including redzones) could confuse CMAs that expect the user-requested size.

Through this detailed and multi-layered technical approach, CMASan effectively integrates ASan's powerful memory safety checks into the complex landscape of custom memory allocators, providing comprehensive coverage with minimal performance impact and a high degree of accuracy.

Demo / Proof of Concept

▶ Watch: CMASan's superior bug detection coverage (6:50)

The talk did not feature a live, step-by-step demonstration of CMASan in action during the presentation. Instead, the speakers presented results from applying CMASan to real-world applications as its proof of concept and validation. This involved deploying CMASan on top of the top 12 C++ applications that utilize custom memory allocators, sourced from GitHub.

The most compelling proof of concept came from CMASan's ability to detect 19 previously unknown bugs in these real-world applications. These findings serve as concrete evidence of CMASan's efficacy and the critical blind spot it addresses in traditional memory sanitizers. Specifically, the researchers highlighted bugs that had gone undetected for significant periods: a 9-year-old vulnerability in SQLite3 and a 2-year-old vulnerability in PHP. The fact that standard ASan and other tools like Go missed all these 19 bugs underscores CMASan's unique capability to uncover a distinct class of memory safety issues related to custom allocators. These discoveries validate CMASan's design and its practical utility in enhancing the security of widely deployed software. The availability of their full paper and open-source repository (linked via a QR code at the end of the presentation) further serves as a means for others to examine and reproduce these findings.

Defensive Implications

▶ Watch: 19 previously unknown bugs detected by CMASan (7:20)

CMASan introduces significant defensive implications for developers, security researchers, and organizations striving to improve software security, particularly in complex applications utilizing custom memory allocators.

For Developers and Software Engineers:

• Awareness of CMA Blind Spots: Developers who implement or use CMAs must recognize that traditional ASan, while powerful, does not cover memory managed by these custom mechanisms. CMASan highlights that this can lead to a false sense of security, as critical memory bugs may persist undetected.
• Automated CMA Bug Detection: CMASan offers a robust, automated solution for detecting memory safety bugs in CMAs. This eliminates the need for arduous, error-prone manual code modifications or deep ASan expertise previously required to bring CMAs under sanitizer scrutiny.
• Integration into CI/CD: Integrating CMASan into continuous integration and continuous deployment (CI/CD) pipelines can provide proactive and continuous memory safety checks for CMA-intensive codebases. This allows bugs to be caught early in the development lifecycle, reducing the cost and effort of remediation.
• Guidance for CMA Design: The identification of "Arena" and "Recycler" patterns provides valuable insights. Developers designing new CMAs can be mindful of these patterns and proactively consider how their custom allocator might interact with memory sanitizers, even if CMASan aims to handle them automatically.

For Security Researchers and Auditors:

• Targeted Vulnerability Research: CMAs represent a fertile ground for discovering new vulnerabilities. Security researchers can leverage CMASan to audit applications that are known to use custom allocators, potentially uncovering a wealth of previously unknown bugs that bypass conventional tools. The discovery of 19 new bugs in prominent applications like SQLite3 and PHP exemplifies this potential.
• Enhanced Fuzzing and Testing: CMASan can be combined with fuzzing techniques to improve the efficiency of bug discovery. By instrumenting the target with CMASan, fuzzers can more reliably trigger and detect memory errors within CMA-managed regions.
• Understanding Complex Memory Architectures: The technical deep dive into CMASan's mechanisms (e.g., instance-specific quarantine, on-demand metadata) provides a blueprint for understanding the complexities of securing advanced memory management schemes.

For Organizations and Application Owners:

• Improved Software Security Posture: For organizations that rely on applications extensively using CMAs (e.g., databases, game engines, specialized middleware), CMASan offers a path to significantly enhance their software's security posture. Proactively addressing CMA-related bugs reduces the attack surface and mitigates the risk of exploitation.
• Risk Assessment: The findings from CMASan can inform risk assessments. If an organization uses a product with a known history of CMA usage, it indicates a potential area of unaddressed memory safety risk that CMASan could help identify and remediate.
• Informed Patching and Updates: The discovery of long-standing bugs in critical software like SQLite3 and PHP underscores the importance of staying updated with security patches. Organizations should monitor the release of fixes for vulnerabilities identified by tools like CMASan and apply them promptly.

In summary, CMASan not only provides a powerful new tool but also fundamentally shifts the understanding of memory safety in applications that deviate from standard allocator practices. Its defensive implications span the entire software lifecycle, from design and development to auditing and deployment, offering a more comprehensive approach to securing modern software systems.

Key Takeaways

• ASan's Blind Spot: Traditional AddressSanitizer (ASan) inherently misses memory safety bugs (buffer overflows, use-after-free, double-free) in objects managed by Custom Memory Allocators (CMAs) because it cannot replace their custom allocation functions.
• Prevalence and Patterns: CMAs are common in real-world applications (found in 78% of analyzed projects) and often follow Arena (missing redzones) or Recycler (missing poisoning/quarantine) patterns, posing specific challenges for ASan.
• Automated CMA Coverage: CMASan extends ASan's detection capabilities to CMA-managed objects automatically, without requiring manual code modifications or specialized ASan expertise from developers.
• Novel Technical Solutions: CMASan achieves this through intelligent CMA API instrumentation, efficient two-level, on-demand metadata storage, and sophisticated instance-specific, free-delaying quarantine zones that operate without altering CMA internal logic.
• Real-World Impact: CMASan effectively found 19 previously unknown memory bugs in widely used applications, including vulnerabilities that persisted for 9 years in SQLite3 and 2 years in PHP, all of which were missed by ASan and other tools.
• Minimal Overhead: Despite its comprehensive coverage, CMASan incurs only a 1.096x performance overhead and 1.148x memory overhead compared to standard ASan, making it a practical and efficient solution for integration into development and testing workflows.

About the Speaker(s)

The primary presenter for this talk was Junwha Hong from Unist. The research work presented, "CMASan: Custom Memory Allocator-aware Address Sanitizer," was a collaborative effort involving several co-authors. These collaborators included Wonil Jang, Mijung Kim, and Lei Yu, alongside Professor Yonghwi Kwon and Yuseok Jeon. The project was conducted in collaboration with researchers from Rula Poly Techch Institute and the University of Maryland, indicating a broad academic partnership in addressing this critical memory safety challenge.