dRR: A Decentralized, Scalable, and Auditable Architecture for RPKI Repository

dRR: A Decentralized, Scalable, and Auditable Architecture for RPKI Repository

Speakers: Yingying Su

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=jvV--G6eGxs

Overview

The talk introduces dRR, a novel architecture designed to fortify the Resource Public Key Infrastructure (RPKI) Repository, a critical element for securing inter-domain routing and preventing IP prefix hijackings. While RPKI has demonstrated its effectiveness over the past decade in establishing trustworthy mappings between Autonomous Systems (ASes) and IP prefixes, its underlying repository infrastructure has remained significantly understudied and, as this research reveals, harbors several critical vulnerabilities. Yingying Su's presentation meticulously details a data-driven security analysis, encompassing a worldwide survey of AS administrators and a large-scale measurement of the existing RPKI Repository, to uncover three fundamental problems: the unilateral power of RPKI authorities, the singleton nature of RPKI objects leading to single points of failure, and the escalating scalability challenges for Relying Parties (RPs).

dRR emerges as a proactive solution to these identified weaknesses, aiming to enhance the security, robustness, and scalability of the RPKI Repository while maintaining compatibility with current RPKI standards. By introducing a decentralized federation of Certificate Servers (CSs) and Monitors, dRR seeks to balance power dynamics, ensure truly distributed and auditable certificate storage, and significantly improve the efficiency of RPKI object updates. The research not only presents a comprehensive architectural design but also validates its efficacy through a prototype implementation on a global testbed, demonstrating negligible latency for new security features and a drastically higher throughput for certificate updates, thereby addressing long-standing issues that threaten the integrity and future growth of RPKI.

Background

▶ Watch: Introduction and identification of RPKI Repository problems (0:00)

The Resource Public Key Infrastructure (RPKI) is a hierarchical Public Key Infrastructure (PKI) designed to secure the Internet's routing system. At its foundation are the five Regional Internet Registries (RIRs) – AFRINIC, APNIC, ARIN, LACNIC, and RIPE NCC – acting as trust anchors, each holding a root Resource Certificate (RC) for their allocated Internet Number Resources (INRs). Beneath them, Certificate Authorities (CAs), such as National Internet Registries (NIRs) or Internet Service Providers (ISPs), issue subordinate RCs to delegate resources or Route Origin Authorizations (ROAs) to authorize specific ASes to originate IP prefixes. These RPKI objects (RCs and ROAs) are then published by CAs to publicly accessible Publication Points (PPs), whose URLs are embedded within the Subject Information Access (SIA) field of their RCs. These PPs collectively form the global RPKI Repository.

Relying Parties (RPs) are crucial actors in this ecosystem. They periodically fetch RPKI data from all PPs, typically using protocols like RRDP or Rsync. After synchronization, RPs validate these RCs and ROAs hierarchically to construct a prefix-to-AS mapping table. This table is then used to perform Route Origin Validation (ROV), enabling routers to discard invalid BGP updates and mitigate prefix hijackings.

Despite RPKI's critical role, the underlying repository infrastructure has been largely overlooked. The research presented in this talk fills this gap with a data-driven RPKI threat analysis, which informed the identification of three core problems:

1. Problem 1 (P1): Unilateral reliance on authority. A survey of AS administrators revealed significant concerns, with 44.1% explicitly worried about malicious RPKI authorities, and 11.5% unsure. Some administrators even reported losing ROAs due to administrative issues. RPKI authorities (CAs and PP managers) possess unilateral power over the RPKI objects they sign and manage. This allows them to:

• Revoke RCs or ROAs by listing them in a Certificate Revocation List (CRL).
• Delete RCs or ROAs from a PP, making them unobtainable by RPs.
• Corrupt RCs or ROAs, causing validation failures.
• Modify RCs or ROAs, potentially reducing authorized INRs or changing authorized entities.

Such actions can incorrectly mark valid BGP updates as invalid, compromising prefix-to-AS mappings. Critically, INR holders and RPs currently lack mechanisms to detect or prevent these malicious actions, and the absence of a trustworthy historical record hinders accountability.

1. Problem 2 (P2): Every RPKI object is a singleton in RPKI Repository. A large-scale measurement conducted on April 1, 2023, identified 36,976 RCs and 122,251 ROAs spread across 61 independent PPs. A striking finding was that only 8 of these 61 PPs were hosted on Content Delivery Networks (CDNs) (7 on Cloudflare, 1 on Amazon), with the remaining 58 hosted within a single AS. This extreme centralization means the availability of most PPs is highly dependent on the reachability of a single AS. Furthermore, 14 PPs hosting ROAs for ASes where they are physically located create a problematic circular dependency: if the PP fails, RPs cannot fetch its ROAs, potentially causing a route downgrade for the AS, which could then impact PP accessibility further. Real-world incidents, such as a 7-hour RIPE NCC PP downtime in 2020 and 10-hour failures of Japan's PP due to hardware or full disk space, underscore the severity of this vulnerability. The current singleton storage model creates critical single points of failure, preventing RPs from obtaining a complete view of RPKI data.

1. Problem 3 (P3): The current RPKI Repository is costly in RP refreshing and lacks scalability. The number of independent PPs has dramatically increased from 5 RIR-operated PPs to over 60 PPs in just nine years. This growth is fueled by a deeper understanding of RPKI, leading to more delegated RPKI deployments, and low barriers to entry for running PPs. A survey indicated that 45.3% of AS administrators using hosted RPKI plan to run their own PPs for greater control, confirming this trend. This expansion significantly increases the burden on RPs, which must traverse all PPs to fetch updates. This can lead to resource exhaustion and paralysis, as demonstrated by Koen van Hove's attack (attack 55 mentioned in the transcript). The proliferation of casually joined PPs also introduces unforeseen risks.

The threat model for dRR specifically addresses these three problems, focusing on honest INR holders expecting secure storage, honest RPs seeking trustworthy and timely RPKI object views, and malicious RPKI authorities or CSs. It also considers compromised private keys but explicitly scopes out attacks unrelated to the RPKI Repository architecture, such as DDoS.

Existing research has attempted to mitigate some of these issues. Consent 23 detects RPKI object changes but only after the fact and struggles to differentiate legitimate operations from malicious ones. While it introduced a signed object.dead to prevent malicious revocations, it doesn't protect against deletion or corruption. DRPKI 52 proposes collaborative signing by RIRs using threshold signatures, but centralizes power with RIRs, breaks the existing hierarchy, and is incompatible with delegated RPKI. Google's Certificate Transparency (CT) 21 for web PKI logs issued certificates but doesn't proactively defend against RPKI certificate compromise, where the threat is unilateral manipulation of legitimate certificates rather than issuance of fake ones. These solutions offer only partial mitigation, motivating the comprehensive design of dRR.

Key Findings

▶ Watch: Introducing dRR: a decentralized, scalable, and auditable solution (2:00)

The comprehensive data-driven analysis conducted for this research yielded three primary findings, which directly informed the design principles of dRR:

1. Identification of Critical RPKI Repository Vulnerabilities: Through a worldwide survey of AS administrators and extensive measurements of the global RPKI Repository, the research definitively identified and quantified three major problems:

• Unilateral Authority Power (P1): A significant portion of AS administrators (44.1%) expressed concerns about malicious RPKI authorities, highlighting the ease with which CAs and PP managers can unilaterally revoke, delete, corrupt, or modify RPKI objects without the consent or auditability for INR holders and RPs.
• Single Points of Failure (P2): The measurement revealed a severe centralization, with 58 out of 61 PPs hosted in a single AS and only 8 utilizing CDNs. This singleton nature creates critical dependencies and vulnerabilities, as evidenced by real-world PP downtimes and circular dependencies where a PP's failure impacts its own reachability.
• Scalability and Cost Issues (P3): The number of PPs has grown over 12 times in nine years, with 45.3% of AS administrators planning to run their own PPs. This exponential growth significantly increases refresh costs and complexity for RPs, hindering timely routing information and introducing risks from unvetted PPs.

1. Novel RPKI-Compatible Architecture for Proactive Defense: dRR represents the first architectural solution that proactively defends against malicious RPKI authority behavior directly from the repository perspective. By introducing Certificate Servers (CSs) and Monitors, dRR establishes a decentralized, auditable, and scalable framework that balances power between authorities and INR holders, ensuring that certificate issuance and revocation are mutually verifiable and resistant to unilateral manipulation.

1. Demonstrated Scalability and Performance Gains: The prototype implementation of dRR on a 100-node global testbed proved its operational viability and significant performance advantages. It introduces negligible latency (no more than 3 seconds total closed-loop overhead) for its new security features, while achieving a throughput for certificate updates that is 450 times higher than the current maximum RPKI certificate update frequency. This demonstrates dRR's capability to handle future large-scale RPKI deployments, including delegated RPKI, without compromising update speed or security.

Technical Deep Dive

▶ Watch: Data-driven threat analysis: malicious RPKI authorities (Problem 1) (2:45)

dRR is designed as an extension and upgrade to the existing RPKI Repository, operating independently of the certificate supply side. Its core architecture revolves around two new entities: the CS Federation and Monitors.

The CS Federation comprises multiple Certificate Servers (CSs). Unlike traditional PPs, CSs are decoupled from CAs, operating as equal members of a federation. They collectively host RCs and ROAs for INR holders, who can freely choose multiple trusted CSs for certificate storage. Any certificate issuance or revocation is publicized within this federation as a Certificate Policy (CP). Each CS then proactively pushes these CPs to connected dRR Monitors. Key principles of the CS Federation include:

1. Independence: CSs operate independently of CAs, with all CSs holding equal status.
2. Verification: New CS entities must pass security and reliability verification by existing federation members.
3. Trust Flexibility: INR holders retain the freedom to choose any CSs they trust to host their certificates.

CS operations begin with registration. New CS members and INR holders must register their identity within the federation. This information, along with all CPs, is uploaded to the CS federation and appended to a tamper-resistant global ledger. This ledger provides a complete, longitudinal view for auditing purposes, secured by a Byzantine Fault-Tolerant (BFT) consensus protocol like Hotstuff.

• CS Registration involves a CS Registration Message (Table I) containing fields such as CS_ID, T_R (type of registration), P_K (public key), CS_SIG, and Pre_SIG. Initial federation members are the five RIRs, and new CSs require approval from both RIRs and existing CS members.
• INR Holder Registration uses an INR holder Registration Message (Table II) with INR_holder_ID, T_R, P_K, Trusted_CSs (a list of chosen CS IDs), INR_holder_SIG, and Pre_SIG. This allows INR holders to explicitly select which CSs will host their certificates.
• Once registered, INR holders choose their trusted CSs to host RCs and ROAs, and designate one CS to publicize the corresponding CPs to the federation.

Certificate Policies (CPs) are central to dRR's auditability and verification mechanisms. Two types are defined:

1. Certificate Issuance Policy (CIP) (Table III): This proves a certificate's authorization by both the CA and the INR holder. It includes fields like VERSION, ISSUER (CA ID), SUBJECT (INR holder ID), CERT (hash of the protected certificate), CERT_T (RC or ROA), ISSUER_RC (hash of the parent RC), VALIDITY, CS_SET (IDs of CSs hosting this certificate), CIP_HASH, and CIP_SIG. Crucially, the CS_SET must be a subset of the Trusted_CSs specified in the INR holder's registration.
2. Certificate Revocation Policy (CRP) (Table IV): This proves mutual agreement for certificate revocation. Fields include VERSION, R_M (revocation method: self or rir), CRP_ISSUER, CERT_SET (hash list of revoked certificates), CRP_HASH, and CRP_SIG. The rir method allows RIRs to jointly sign CRPs for mandatory revocations (e.g., for malicious activity), while the self method requires the CA to ensure recursive revocation of descendant certificates.

Certificate Policy Validation (Algorithm 1) ensures the integrity of CPs. It verifies the certificate status, issuer and subject identities, checks CS_SET against Trusted_CSs, validates the ISSUER_RC, and verifies the CIP's validity and issuer's signature. Any failure leads to rejection. This validation process is efficient, with over 90% of RC revocations requiring less than 3 CRPs.

Monitors serve two primary functions:

1. Provide RPs with verifiable Certificate Update Lists (CULs), allowing RPs to authenticate and verify the integrity of RPKI object views.
2. Offer services to INR holders to check the authenticity and validity of specific certificates.

Monitors achieve these proofs by maintaining an M-Tree, which is an improved Merkle Hash Tree.

• M-Tree Structure: Leaf nodes of the M-Tree contain CP information. New CIPs are appended to the M-Tree in lexicographical order of their certificate hashes, each entry including B_NUM, ISSUER_RC, CERT, VALIDITY, CS_SET, and CIP_HASH. CRPs, rather than creating new entries, modify existing CIP entries by updating them with relevant revocation information. After each block of CPs/CRPs is processed, the Monitor generates a commitment (root hash) of the current M-Tree version and publishes this metadata in a Commitment Update File for public verification.
• Serving INR holders: INR holders can query a Monitor with a tuple (B_NUM, CERT) to verify if their CIP has been publicized and protected by dRR. The Monitor returns the result along with a pruned M-Tree (Figure 7b) as proof, enabling the INR holder to reconstruct and verify the commitment.
• Serving RPs: RPs obtain verifiable CULs from Monitors. An RP submits its last synchronized block number and commitment (B_NUM, C). The Monitor responds with a pruned M-Tree (PT_B) containing the CUL (new and updated entries) and a consistency proof (Figure 7c). RPs perform two verification steps: (1) verify the reconstructed commitment is trusted, and (2) verify it evolved consistently from the previous commitment. Post-verification, RPs delete revoked certificates and fetch new ones from the designated CSs.
• Initializing a new Monitor: New Monitors can obtain the historical global ledger to build their M-Tree or download it from an existing Monitor and compare root hashes for integrity.

Incremental Deployment: dRR employs a top-down deployment model (Figure 8). For a certificate to be dRR-protected, its parent RC must also be dRR-protected, ensuring security and compatibility. CAs can retire their PPs only when all their subordinate certificates are dRR-protected. INR holders moving to dRR will no longer store certificates in PPs, with revocation handled by the CS federation. RPs will perform two synchronization phases: first, refresh dRR-protected certificates via CULs, and second, refresh other RPKI objects (non-dRR RCs/ROAs, CRLs, manifests) using Rsync/RRDP. In case of conflicts (e.g., a dRR-protected certificate appearing in a parent CA's CRL), the RP is instructed to refuse revocation of the dRR-protected certificate.

Demo / Proof of Concept

▶ Watch: Repository vulnerability and single points of failure (Problem 2) (4:00)

The dRR project includes a robust prototype implementation and evaluation on a global testbed to demonstrate its practical feasibility and performance benefits.

Implementation Details:

• The CS Federation leveraged an open-source implementation of the Hotstuff protocol 25. Hotstuff is a leader-based Byzantine Fault-Tolerant (BFT) replication protocol known for its linear message complexity, high throughput, and proven security in partially synchronous models, making it ideal for confirming CIPs and CRPs. The Chained Hotstuff variant with round-robin leader selection was used.
• Message formats for CS and INR holder registration, CIPs, and CRPs were defined using Protocol Buffers 3.16 20.
• Cryptographic operations utilized SHA-256 for hashing and SHA-256 RSA for signatures.
• The core CS logic, handling uploading, verifying operations, and pushing CPs to Monitors, was implemented in Golang, comprising 512 lines of code.
• The Monitor's M-Tree data structure was implemented in Python, spanning 328 lines, with M-Tree hash calculations also using SHA-256.

Deployment on a Global Testbed:

• The evaluation utilized a distributed testbed of 100 nodes.
• These nodes were strategically distributed across 15 countries or regions, including Australia, India, China, USA, and Germany, simulating a real-world global deployment.
• The nodes were allocated evenly, with 50 nodes designated as CS servers and 50 nodes as Monitor servers, distributed across the geographical regions.
• Each server ran Ubuntu 18.04 and was equipped with 4 CPU cores and 8GB of RAM.
• Network bandwidth was configured at a peak of 200Mbps for CS nodes and 100Mbps for Monitor nodes.

Evaluation and Results:

The evaluation established a baseline by analyzing RPKI certificate update frequencies, observing a significant increase after 2020, with daily updates often below 5,000 but peaking at 60,000 during events like RPKI CA key rollovers (Figure 9).

• CS Federation Performance:
• Throughput and Latency: The throughput of CP submission to confirmation was evaluated. With a batch size of 50 CPs, the system achieved a maximum throughput of 729 CPs/second with an average latency of 0.91 seconds (Figure 10).
• CRP Impact: Throughput dropped by 6% when a CRP contained up to 50 revoked certificates and by 20% for 100 certificates, leading to a recommendation to limit revoked certificates per CRP to 50 (Figure 11).
• Scalability with CSs: With 50 CSs, the system maintained a throughput exceeding 310 CPs/second and confirmed CPs within 2 seconds (Figure 12). This translates to 26.78 million CPs processed per day, which is 450 times higher than the current maximum RPKI certificate update frequency.

• Monitor Performance:
• Proof Size: Proof size for INR holders (presence/absence) grew logarithmically. At the current RPKI certificate scale, both proof types were within 1 KB (Figure 13).
• CUL Size: CUL size for RPs correlated with the number of updated certificates and CRP ratios. The average update information per certificate ranged between 100 B and 300 B, significantly less than the 3 KB occupied by current RPKI manifests and CRLs (Figure 14).
• Monitor Latency: Simulating a CS pushing 10,000 new blocks to 50 Monitors, each Monitor completed M-Tree updates for each block within 500 milliseconds (Figure 15).

• Closed-Loop Time Overhead: The total additional closed-loop time overhead introduced by dRR (CS federation publishing CPs, Monitors updating M-Trees) was no more than 3 seconds. This is deemed negligible compared to existing RPKI bottlenecks, which often involve tens of minutes to several hours for certificate signing and RP cache refreshes. The 450 times higher throughput ensures dRR does not degrade overall update speed.

Defensive Implications

▶ Watch: Scalability challenges due to growing RPKI Publication Points (Problem 3) (4:45)

dRR offers profound defensive implications for the RPKI ecosystem, addressing fundamental vulnerabilities that undermine trust and stability in inter-domain routing. Its architecture empowers various stakeholders and significantly hardens the RPKI Repository against a range of threats.

Firstly, dRR directly tackles Problem 1: Unilateral reliance on authority. By decoupling certificate issuance from management and introducing the CS Federation, INR holders gain proactive control over their certificates. They can choose multiple trusted CSs to host their RCs and ROAs, preventing any single RPKI authority (CA or PP manager) from unilaterally revoking, deleting, corrupting, or modifying their certificates. The requirement for Certificate Policies (CPs), mutually signed by both the CA and the INR holder for issuance and requiring mutual agreement for revocation, establishes a verifiable audit trail. This means defenders, particularly INR holders, are no longer at the mercy of potentially malicious or negligent authorities; they have verifiable proof of their certificate status.

Secondly, dRR provides a robust solution to Problem 2: Every RPKI object is a singleton. The truly distributed storage model, where INR holders can store their certificates across multiple independent CSs, eliminates single points of failure. This significantly enhances the resilience of the RPKI Repository. If a single CS fails due to malfunction or attack, RPs can still obtain a complete and valid view of RPKI objects from other trusted CSs. This directly mitigates the risks highlighted by real-world PP downtimes and the dangerous circular dependencies observed in the current infrastructure. Network operators relying on RPKI for ROV can have greater confidence in the continuous availability and integrity of RPKI data.

Thirdly, the architecture drastically improves Problem 3: Scalability for RPs. The CS federation's design and the role of Monitors simplify the process for RPs to obtain timely and complete RPKI update data. Instead of traversing dozens or hundreds of disparate PPs, RPs can query trusted Monitors for Certificate Update Lists (CULs) and consistency proofs. This significantly reduces the processing burden on RPs, mitigates risks from casually joined or malicious PPs, and ensures that RPKI can scale with future growth, especially with increasing delegated RPKI deployments. The demonstrated 450 times higher throughput for certificate updates means RPs can refresh their caches much faster and more efficiently, translating into quicker propagation of routing security information and improved reaction times to BGP anomalies.

Furthermore, dRR introduces public auditability and trust flexibility. The global ledger maintained by the CS federation provides a tamper-resistant, longitudinal record of all RPKI certificates and operations, ensuring transparency and accountability. RPs and INR holders can use Monitors to verify the authenticity and integrity of RPKI objects at any time, providing cryptographic proofs of presence, absence, and consistency via the M-Tree. This level of auditability is unprecedented in the current RPKI system. RPs can choose their trusted Monitors, and INR holders can choose their trusted CSs, fostering a more resilient and decentralized trust model.

From a defensive standpoint, dRR's security analysis confirms its resilience against various threats. Compromised INR holder CP_K (private key for signing CPs) allows for malicious revocations but not arbitrary issuance, and prompt key updates can mitigate this. Compromised CS private keys are contained because other CSs reject unverified CPs, and the federation can prohibit malicious CS submissions. Malicious INR holders attempting to abuse IP prefixes can be subject to RIR-enforced revocation, and dRR's paid mode and CS verification prevent resource exhaustion attacks. In essence, dRR limits potential damage and enhances recoverability compared to the current RPKI, where a single compromised CA private key can lead to widespread damage.

In conclusion, dRR equips network defenders with a significantly more robust, transparent, and scalable RPKI repository. It shifts power dynamics, reduces single points of failure, and streamlines the update process, making RPKI a more reliable and trustworthy foundation for securing global internet routing.

Key Takeaways

• Decentralized Power and Auditability: dRR fundamentally shifts power away from unilateral RPKI authorities by introducing a decentralized federation of Certificate Servers (CSs) and requiring mutual consent for certificate issuance and revocation via verifiable Certificate Policies (CPs). All operations are appended to a tamper-resistant global ledger, ensuring public auditability.
• Elimination of Single Points of Failure: By allowing INR holders to store their certificates across multiple trusted CSs, dRR breaks the "singleton" nature of RPKI objects, providing truly distributed storage and significantly enhancing the robustness and availability of the RPKI Repository against failures or attacks on individual CSs.
• Enhanced Scalability for Relying Parties (RPs): dRR introduces Monitors and Certificate Update Lists (CULs), allowing RPs to obtain verifiable RPKI update data from a single trusted entity instead of traversing all Publication Points (PPs). This drastically reduces RP refresh costs and improves the scalability of RPKI for future large-scale deployments.
• Significant Performance Improvements: Prototype evaluation on a global testbed demonstrated that dRR introduces negligible closed-loop latency (less than 3 seconds) for its new security features, while achieving a certificate update throughput 450 times higher than the current maximum RPKI update frequency, ensuring efficient and timely propagation of routing security information.
• RPKI Compatibility and Incremental Deployment: dRR is designed to be fully compatible with existing RPKI standards and supports a top-down, incremental deployment model, allowing for a gradual transition without disrupting current RPKI operations.
• Proactive Defense Against Malicious Authorities: Unlike prior solutions, dRR proactively defends against malicious RPKI authorities by enabling INR holders and RPs to verify certificate status and changes, thereby preventing unilateral manipulation such as malicious revocations, deletions, corruptions, or modifications of RPKI objects.

About the Speaker(s)

Yingying Su is the speaker for this presentation, delivering the talk on "dRR: A Decentralized, Scalable, and Auditable Architecture for RPKI Repository" at the NDSS Symposium. No further biographical details are provided within the transcript or metadata.