ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning

ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning

Speakers: Linkang Du

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=oDa14xLjcf0

Overview

In the rapidly evolving landscape of artificial intelligence, Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm, driving innovation across complex decision-making domains from autonomous systems to critical infrastructure control. However, the direct application of DRL in real-world, safety-critical environments poses significant risks, necessitating the adoption of offline DRL. This approach trains models on pre-collected datasets, circumventing hazardous real-world interactions. While fostering research and development, the open-source publication of these valuable datasets introduces a critical challenge: protecting the intellectual property of data owners against misuse, data theft by ex-employees, or the unauthorized creation of pirated Model-as-a-Service platforms.

This talk introduces ORL-AUDITOR, a groundbreaking trajectory-level dataset auditing mechanism specifically designed for offline RL. Presented by Linkang Du, ORL-AUDITOR addresses the crucial question of how a data owner can definitively prove that a suspect model was derived from their proprietary dataset. By leveraging the intrinsic property of cumulative rewards as a unique and stable fingerprint, ORL-AUDITOR offers a robust, post-event solution to detect dataset piracy, overcoming the limitations of existing watermarking, dataset inference, and membership inference techniques that are largely unsuitable for the unique characteristics of offline DRL. The work is particularly significant for institutions and companies that publish high-quality DRL datasets under open-source licenses, providing them with a vital tool for compliance and intellectual property enforcement.

Background

▶ Watch: Introduction to offline DRL and data IP challenges (0:00)

Deep Reinforcement Learning (DRL) has demonstrated remarkable success in mastering complex tasks by learning optimal policies through trial and error in simulated or real environments. However, in sensitive applications like autonomous driving, robotic surgery, or power grid management, direct interaction during training can be prohibitively dangerous or expensive. This critical limitation led to the rise of offline DRL, where agents learn entirely from static, pre-collected datasets of environmental interactions without any further real-time engagement. This paradigm shift enables safer and more efficient development but simultaneously introduces new security and intellectual property (IP) challenges.

Many research institutions and tech giants, including DeepMind, Berkeley AI Research, and Google, openly publish high-quality offline DRL datasets under various open-source licenses such as GNU General Public License or Apache License. These licenses typically include provisions for attribution and ShareAlike terms to protect the data owner's IP. Despite these legal frameworks, the risk of data misuse and infringement remains high. Surveys indicate a significant percentage of former employees admit to taking valuable data, often due to a perceived lack of preventative policies or technologies. Malicious actors could also pirate these datasets to train unauthorized models for commercial gain, posing a direct threat to the data owner's investment and competitive advantage.

Existing solutions for dataset copyright protection have proven inadequate for the nuances of offline DRL:

• Watermarking techniques involve embedding specific patterns into a dataset before its publication. This pre-event nature renders them inapplicable to already published open-source datasets. Furthermore, watermarking risks perturbing the dataset's natural distribution, potentially affecting the quality of models trained on it.
• Dataset inference methods aim to statistically infer a model's reliance on private training data without explicit watermarks. However, these often depend on explicit classification labels or the availability of public datasets with similar distributions. DRL models, by contrast, typically produce continuous outputs (actions) rather than discrete labels, and their learning is guided by rough reward signals. The diverse and often unique data distributions in offline RL datasets, which depend on both the environment and the data collection strategy, make finding suitable public datasets for comparison extremely challenging.
• Membership Inference Attacks (MIA), which determine if a specific data point was part of a model's training set, are generally designed for online RL scenarios. These attacks assume the auditor or attacker can interact with the environment to generate non-member data or manipulate states. In the offline DRL setting, the auditor is strictly limited to the pre-collected dataset and cannot interact with the environment, making traditional MIA approaches ineffective.

To address these shortcomings, ORL-AUDITOR leverages a fundamental characteristic of DRL: the cumulative reward. This metric, representing the sum of discounted future rewards from a given state-action pair, inherently guides the DRL model's learning process. Consequently, models trained on the same dataset tend to exhibit similar cumulative reward patterns, making it a stable and intrinsic identifier for auditing.

The study considers several prominent offline DRL algorithms:

• Behavior Clone (BC) 48: A supervised learning baseline that directly imitates state-action pairs from the dataset, minimizing the difference between the policy's action and the dataset's action.
• Batch-Constrained Q-learning (BCQ) 20, 18: An early practical algorithm that constrains actions to those observed in the dataset, using a generative model (e.g., VAE) and a Q-network.
• Implicit Q-Learning (IQL) 34: Avoids querying values of out-of-distribution actions by constructing state-action value models and employing advantage-weighted regression.
• TD3PlusBC 19: A highly effective algorithm based on Twin Delayed Deep Deterministic Policy Gradient (TD3), incorporating a BC regularization term to align the policy with dataset actions.

These algorithms represent diverse strategies for policy learning in offline settings, ensuring ORL-AUDITOR's broad applicability.

Key Findings

▶ Watch: Introducing ORL-AUDITOR and offline DRL fundamentals (2:00)

ORL-AUDITOR introduces a novel and highly effective solution for dataset auditing in offline DRL, characterized by several key findings:

1. First Trajectory-Level Auditing for Offline DRL: ORL-AUDITOR is presented as the pioneering mechanism for auditing offline DRL models at the individual trajectory level. This granularity allows data owners to prove the piracy of specific sequences of interactions rather than just a general reliance on their dataset.
2. Cumulative Rewards as Intrinsic Fingerprint: The core innovation lies in leveraging cumulative rewards as a stable, intrinsic fingerprint of a dataset. Models trained on the same data inherently learn to predict similar cumulative rewards for given state-action pairs, making this metric a reliable indicator of data origin.
3. High Audit Accuracy: Across extensive evaluations spanning 12 combinations of tasks and offline DRL models (BC, BCQ, IQL, TD3PlusBC), ORL-AUDITOR consistently achieved high True Positive Rate (TPR) and True Negative Rate (TNR), both exceeding 95% in most settings. For instance, using the L1 norm, all results were above 94%, confirming its strong validity.
4. Superiority of Wasserstein Distance: The choice of distance metric proved crucial. Wasserstein distance (Earth Mover's Distance - EMD) consistently demonstrated the best or second-best performance, effectively capturing both numerical and positional deviations of cumulative rewards, which are vital for sequential data.
5. Robustness of Grubbs' Test: For outlier detection in hypothesis testing, Grubbs' test significantly outperformed the simpler 3σ principle. Grubbs' test proved more robust against outliers in cumulative rewards, leading to more reliable audit decisions.
6. Ineffectiveness of Existing MIAs: When adapted to the offline DRL context, existing membership inference methods designed for online RL scenarios yielded near-random accuracy (approximately 50% TPR/TNR). This highlights their fundamental incompatibility with the offline setting, where environmental interaction for generating non-member data is absent.
7. Impact of Hyperparameters:

• Shadow Model Count: Audit accuracy generally increased with more shadow models (up to a point), as more shadows provide a more precise understanding of the true cumulative reward distribution. However, gains saturated beyond 15 shadow models, indicating an optimal balance between accuracy and computational overhead.
• Significance Level (α): For more complex tasks (e.g., Ant) or models with lower performance, a larger significance level (e.g., 0.01) was recommended. Reducing α too much (e.g., 0.001 or 0.0001) could shrink confidence intervals, making deviations imperceptible and significantly reducing TPR/TNR.
• Trajectory Size: Larger trajectory sizes generally improved accuracy by providing more state-action pairs for hypothesis testing. Interestingly, for the Ant task, using 25% of the trajectory length sometimes yielded up to a 7% promotion in TNR, suggesting that front states might carry more distinctive behavioral information.

1. Real-world Applicability: ORL-AUDITOR demonstrated effectiveness on open-source datasets for the HalfCheetah task from Google (D4RL) and DeepMind (RL Unplugged), achieving TPR and TNR exceeding 95%, validating its utility in practical scenarios.
2. Robustness Against Defenses:

• Ensemble Architecture: ORL-AUDITOR maintained high audit accuracy (TPR and TNR consistently exceeding 80%) even against adversaries using ensemble methods (e.g., K=5 subsets), as the critic model captures overall dataset distribution features.
• Action Distortion: The system was largely unaffected by weak Gaussian noise (σ=0.01) in action outputs, with maximum accuracy attenuation within 3%. Strong distortion (σ=0.1) did reduce TPR, but it also significantly degrades the suspect model's performance, making it a less viable defense.

These findings collectively establish ORL-AUDITOR as a robust, accurate, and practically applicable solution for protecting intellectual property in the offline DRL domain.

Technical Deep Dive

▶ Watch: Overview of prominent offline DRL algorithms (2:40)

ORL-AUDITOR is engineered to address the specific challenge of proving dataset piracy in offline DRL, where an auditor has black-box access to a suspect model and no interaction with the online environment. The core premise is that the cumulative reward, J(π), acts as an intrinsic, stable identifier for DRL models trained on a particular dataset.

The problem can be formalized as follows: A data provider (Institution 1) publishes dataset D1. An adversary trains a suspect model, π1, on D1 for unauthorized purposes. An auditor, hired by Institution 1, must determine if π1 was indeed trained using D1, specifically aiming to prove that π1 pirated specific trajectories from D1. The auditor possesses full knowledge of D1 (number of trajectories, state/action spaces) but cannot interact with the environment and can only query the suspect model with states to obtain actions, without inspecting its internal parameters.

ORL-AUDITOR operates in three meticulously designed steps: Model Preparation (MP), Cumulative Reward Collection (CRC), and Audit Process (AP).

Step 1: Model Preparation (MP)

In this initial phase, the auditor constructs the foundational tools for the audit:

1. Critic Model Training:

• A critic model, denoted as Q(s,a), is trained to accurately estimate the cumulative reward for any given state-action pair (s,a) within the target dataset D. This model serves as the impartial "measuring stick" for evaluating both shadow and suspect models.
• The training employs a Temporal-Difference (TD)-based learning method. While Monte Carlo (MC)-based methods offer unbiased predictions, they necessitate complete trajectories, which are often impractical for real-world datasets that might be truncated or have infinite time horizons. TD-based methods, despite potential bias and sensitivity to initialization, can learn effectively from incomplete sequences.
• The critic model is implemented as a fully connected neural network comprising four hidden layers, each with 1024 neurons. It is optimized using the Adam optimizer with a learning rate of 0.001, a mini-batch size of 4096, over 150 epochs.

1. Shadow DRL Model Training:

• The auditor trains k shadow DRL models on the exact target dataset D. These shadow models are critical for establishing a baseline distribution of expected cumulative rewards when a model is legitimately trained on D.
• Crucially, these k models are trained using the same offline RL framework (e.g., BC, BCQ, IQL, TD3PlusBC) as the suspected model, but with distinct random initializations and training processes. This ensures that the shadow models capture the inherent variability and behavior expected from legitimate training on D.

Step 2: Cumulative Reward Collection (CRC)

Once the critic model and shadow models are prepared, the auditor proceeds to collect cumulative rewards:

1. Shadow Model Rewards:

• For each trajectory Tj in the dataset D, the auditor feeds the states sj from Tj into each of the k shadow models. This yields predicted actions aj from each shadow model.
• The resulting (state, action) pairs (sj, aj) are then fed into the trained critic model Q(s,a) to estimate their cumulative rewards. This process generates k sets of estimated cumulative rewards, Qj_i (for i=1,...,k), for each trajectory Tj, representing a sampling of the "true" cumulative rewards from D.

1. Suspect Model Rewards:

• In parallel, the auditor queries the suspect DRL model with the same states sj from Tj to obtain its predicted actions aj_s.
• These (state, action) pairs (sj, aj_s) are also fed into the critic model Q(s,a) to obtain the estimated cumulative rewards for the suspect model, denoted as Qj_s.

Step 3: Audit Process (AP)

With the collected cumulative rewards, the auditor performs the trajectory-level audit:

1. Mean Shadow Reward Calculation:

• For each trajectory Tj, the auditor calculates the element-wise mean of the k shadow model cumulative rewards, denoted as Qj_mean. This mean serves as the central reference point, representing the expected cumulative reward for Tj when trained legitimately on D.

1. Distance Calculation:

• The auditor computes the distance between the suspect model's cumulative rewards Qj_s and the mean shadow rewards Qj_mean. This distance is d(Qj_s, Qj_mean).
• Concurrently, the distances between each individual shadow model's cumulative rewards Qj_i and Qj_mean are calculated, forming a distribution of "normal" distances: d(Qj_i, Qj_mean) for i=1,...,k.

1. Hypothesis Testing:

• The core of the audit decision is framed as a hypothesis test:
• H0 (Null Hypothesis): d(Qj_s, Qj_mean) is not an outlier within the distribution of d(Qj_i, Qj_mean). This implies the suspect model Ms was trained on trajectory Tj.
• H1 (Alternative Hypothesis): d(Qj_s, Qj_mean) is an outlier. This implies Ms was not trained on Tj.
• Grubbs' test is employed for outlier detection. This statistical test is preferred over simpler methods like the 3σ principle due to its robustness, especially for univariate datasets, and its ability to compare a maximum deviation from the sample mean to the sample standard deviation. Before applying Grubbs' test, the Anderson-Darling test is used to pre-check if the distance values of the shadow models conform to a Gaussian distribution, which is suitable for small sample sizes.

1. Distance Metric Selection:

• The choice of distance metric is crucial for accurately comparing cumulative reward sequences. Several metrics were considered:
• Lp norm (L1, L2): Measures the sum of absolute or squared differences. While common, they might not fully capture sequential data characteristics.
• Cosine distance: Measures the angle between vectors, indicating directional similarity, but can normalize away important numerical differences.
• Wasserstein distance (Earth Mover's Distance - EMD): This metric measures the minimum "cost" to transform one probability distribution into another. It is highly suitable for ORL-AUDITOR because it effectively reflects both numerical magnitude and positional deviations of cumulative rewards, making it sensitive to subtle differences in learned behaviors. Wasserstein distance was selected as the default due to its superior experimental performance.

This trajectory-level auditing process is repeated for all trajectories in the target dataset. If the proportion of trajectories identified as misappropriated exceeds a pre-defined threshold, the auditor can confidently claim dataset-level piracy.

Demo / Proof of Concept

▶ Watch: Formalizing the dataset auditing problem and auditor scope (4:00)

While the talk does not describe a live, interactive demonstration, the extensive evaluation section serves as a comprehensive and rigorous proof of concept for ORL-AUDITOR's efficacy and practicality. The experimental setup meticulously simulated real-world conditions and rigorously tested the system's performance across various dimensions.

Experimental Setup:

• Environments: Three standard Gym environments were used to represent diverse complexities: Lunar Lander (continuous, 6-dim state, 2-dim action), Bipedal Walker (24-dim state, 4-dim action), and Ant (111-dim state, 8-dim action).
• Dataset Generation: Following established practices, five online RL models were trained for each task using different random seeds. Each online model then generated a dataset, resulting in 5x1 datasets per task. Lunar Lander datasets contained 5x10⁵ transitions, Bipedal Walker 10⁶, and Ant 2x10⁶.
• Offline Model Preparation: For each of the five datasets per task, 30 offline DRL models were trained using BC, BCQ, IQL, and TD3PlusBC algorithms, each with distinct random seeds for initialization and optimization, totaling 5x1x30 offline RL models.
• Critic Model: A fully connected neural network with four 1024-neuron hidden layers, trained with a TD-based method, Adam optimizer (LR 0.001, batch size 4096), for 150 epochs.
• Evaluation Metrics: True Positive Rate (TPR) (correctly identified pirated trajectories) and True Negative Rate (TNR) (correctly identified non-pirated trajectories).
• Comparisons: ORL-AUDITOR with Grubbs' test was compared against the 3σ principle, and various distance metrics (L1, L2, Cosine, Wasserstein). Existing online membership inference methods were adapted for an offline comparison.

Key Evaluation Results:

• Overall Audit Performance: Across 12 task-model combinations, ORL-AUDITOR consistently achieved TPR and TNR values exceeding 95%. For example, with the L1 norm, all results were above 94%. This high accuracy validated ORL-AUDITOR's fundamental design.
• Wasserstein Distance Superiority: The Wasserstein distance consistently performed best or second best, demonstrating its effectiveness in characterizing both numerical and positional deviations in cumulative rewards.
• Grubbs' Test Robustness: Grubbs' test significantly outperformed the 3σ principle, which was prone to being misled by outliers in cumulative rewards.
• Failure of Existing MIAs: When adapted to the offline DRL setting, existing membership inference methods yielded near-random accuracy (around 50% TPR/TNR), underscoring the unique challenges of offline auditing.
• Visualization of Cumulative Rewards: Using t-SNE, visualizations clearly showed distinct clusters for cumulative rewards generated by shadow models trained on the target dataset versus those from other datasets, confirming cumulative reward as a viable fingerprint.
• Hyperparameter Study:
• Increasing shadow models from 9 to 15 improved accuracy, but gains saturated from 15 to 21, indicating 15 shadow models as an efficient number.
• A larger significance level (e.g., 0.01) was more effective for complex tasks (Ant) or low-performing suspect models.
• Larger trajectory sizes generally improved accuracy, though for Ant, using 25% of the trajectory length surprisingly boosted TNR by up to 7%, suggesting the importance of front states.
• Real-world Application: ORL-AUDITOR was successfully applied to open-source HalfCheetah datasets from Google (D4RL) and DeepMind (RL Unplugged), achieving TPR and TNR exceeding 95% with L1 norm and Wasserstein distance, confirming its practical utility.

Robustness Against Defenses:

• Ensemble Architecture: Even against an ensemble defense (K=5 subsets), ORL-AUDITOR maintained high accuracy (TPR and TNR > 80%), demonstrating its ability to detect overall dataset distribution characteristics.
• Action Distortion: ORL-AUDITOR showed strong resistance to weak Gaussian noise (σ=0.01), with accuracy attenuation within 3%. While strong noise (σ=0.1) did reduce TPR, such distortion also significantly degrades the suspect model's performance, making it a less practical adversarial strategy.

These comprehensive evaluations, rather than a live demo, unequivocally prove ORL-AUDITOR's high efficacy, robustness, and applicability in real-world offline DRL auditing scenarios.

Defensive Implications

▶ Watch: Detailed reasons why existing solutions fail (4:10)

ORL-AUDITOR offers significant implications for data owners and the broader security posture of the offline DRL ecosystem. Its primary defensive value lies in providing a robust, post-event mechanism for intellectual property protection for valuable DRL datasets.

For institutions and companies that publish their datasets under open-source or commercial licenses, ORL-AUDITOR serves as a crucial enforcement tool. Prior to this work, proving that a suspect model was derived from a specific dataset, especially in the black-box, offline DRL setting, was exceptionally challenging. With ORL-AUDITOR, data owners can now:

1. Enforce Licensing Terms: By identifying specific pirated trajectories, data owners can prove non-compliance with attribution, ShareAlike, or patent grant terms associated with their datasets. This moves beyond mere suspicion to concrete, trajectory-level evidence.
2. Deter Data Theft and Misuse: The existence of an effective auditing mechanism like ORL-AUDITOR can act as a deterrent against malicious actors, including ex-employees or unauthorized customers, from stealing or misusing proprietary DRL datasets to build illicit Model-as-a-Service platforms or gain unfair competitive advantages.
3. Conduct Post-Event Audits: Unlike watermarking, which requires pre-planning and modification of the dataset, ORL-AUDITOR is a post-event solution. This means it can be applied to already published datasets, providing protection for existing intellectual assets.
4. Identify Specific Infringements: The trajectory-level granularity of ORL-AUDITOR allows data owners to pinpoint exactly which parts of their dataset have been misappropriated, strengthening legal claims and compliance investigations.

Furthermore, the robustness analysis of ORL-AUDITOR provides valuable insights for defenders:

• Resilience to Common Defenses: The system's ability to maintain high accuracy against adversarial techniques like ensemble architectures and weak action distortion indicates that simply obfuscating a model's training process or slightly perturbing its outputs will likely not be sufficient to evade detection. This suggests that the "fingerprint" of cumulative rewards is deeply embedded in the model's learned policy.
• Trade-off for Adversaries: Stronger adversarial defenses, such as significant action distortion (e.g., σ=0.1), were shown to degrade the suspect model's normal behavior and performance. This creates a critical trade-off for adversaries: to effectively hide their data source, they must severely compromise the utility and quality of their pirated model, making the infringement less valuable.
• Guidance for Auditing Practices: The hyperparameter studies provide practical guidelines for defenders setting up their auditing systems. Understanding the impact of shadow model count, significance level, and trajectory size enables optimized deployment for specific tasks and confidence requirements.

In essence, ORL-AUDITOR empowers data owners with a powerful forensic tool, transforming the landscape of DRL data governance and offering a concrete mechanism to protect valuable intellectual property in the age of offline reinforcement learning.

Key Takeaways

• ORL-AUDITOR is the first trajectory-level dataset auditing mechanism for offline Deep Reinforcement Learning (DRL) models, addressing a critical gap in IP protection.
• It leverages cumulative rewards as a stable, intrinsic fingerprint of a dataset, enabling robust identification of models trained on specific data.
• The system achieves high efficacy, demonstrating over 95% auditing accuracy (TPR and TNR) across various DRL models and tasks, with Wasserstein distance and Grubbs' test proving most effective.
• Existing online Membership Inference Attacks (MIAs) are ineffective in the offline DRL setting, highlighting ORL-AUDITOR's specialized design for this unique challenge.
• ORL-AUDITOR exhibits robustness against common adversarial defenses like ensemble architectures and weak action distortion, indicating the deep embedding of the dataset's "fingerprint."
• Successfully applied to real-world open-source datasets (D4RL, RL Unplugged HalfCheetah), confirming its practical applicability and effectiveness for auditing published DRL datasets.

About the Speaker(s)

Linkang Du, the presenter of "ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning," is a researcher focused on the critical intersection of Deep Reinforcement Learning and security. As evidenced by the detailed technical depth and rigorous evaluation presented in this talk, Linkang's work centers on developing novel methods to protect intellectual property and ensure compliance within the rapidly expanding field of AI, particularly concerning the integrity and provenance of training data for DRL models. The open-sourcing of ORL-AUDITOR further underscores their commitment to advancing practical and impactful solutions for the DRL community.