In-Context Probing for Membership Inference in Fine-Tuned Language Models

In-Context Probing for Membership Inference in Fine-Tuned Language Models

Speakers: Zhexi Lu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=wolZi0uJ_iY

Overview

This talk presents a novel membership inference attack (MIA) against fine-tuned language models that exploits a fundamental property of training dynamics called the optimization gap. The key insight is that member samples (data used in training) show diminishing returns when further optimized, while non-member samples still have significant room for improvement. To detect this gap without requiring fine-tuning access, the researchers use in-context learning as an approximation of gradient-based optimization, achieving state-of-the-art results in a purely black-box, reference-free setting.

The method achieves an AUC of 0.942 on Llama 3.2 3B Instruct with the HealthCareMagic dataset, compared to 0.847 for the best existing reference-free baseline. Crucially, it accomplishes this without training any reference model -- a significant practical advantage over reference-based approaches that require expensive model training.

Background

▶ Watch: Privacy risks of fine-tuning LLMs on sensitive data (0:00)

As organizations increasingly fine-tune large language models on domain-specific data -- medical records, legal cases, financial data -- the privacy implications become critical. Membership inference attacks determine whether a specific individual's data was part of the training set. If an attacker confirms that someone's medical records were used to train a mental health chatbot, this reveals that the person may have a mental health condition -- a direct privacy violation.

The attack model operates at inference time in a black-box setting: the attacker queries the target model with a data sample and observes the output (log probabilities or generated text), then makes a binary determination of whether the sample was in the training data.

Existing MIA methods fall into two categories. Reference-free methods rely on post-hoc signals from the model's output. For example, Min-K% extracts membership signals from the lowest log-probability tokens. These methods are computationally cheap but often lack theoretical grounding and miss deeper membership signals. Reference-based methods require data from a similar distribution to train a reference model. For example, SPA trains a reference model on self-prompted data to calibrate sample difficulty. These achieve better performance but at the cost of training an entire additional model.

Both categories share a fundamental limitation: they empirically extract membership signals from log probabilities without connecting to the underlying training dynamics that create the membership distinction in the first place.

Key Findings

▶ Watch: The optimization gap: 74% of loss reduction happens in first epoch (4:00)

• In-context probing MIA achieves AUC of 0.942 on Llama 3.2 3B Instruct with HealthCareMagic, compared to 0.847 for the best reference-free baseline
• The method outperforms all reference-free methods across all tested models and datasets
• Performance is comparable to or better than reference-based methods that require training a separate model
• Instruction-tuned models are more vulnerable than base models, likely because instruction tuning enhances the model's ability to learn from in-context demonstrations, making the probing signal stronger
• The attack is robust to the choice of perturbation generator -- four different models used to generate self-perturbation probes all yield similar performance
• More domain-relevant probes produce stronger signals: probes from the same domain as the target data achieve the highest AUC and TPR
• The method remains the best or near-best performer under DP-SGD (differentially private training), though all attacks are significantly mitigated
• The approach adapts to text-only settings where log probabilities are unavailable, using token similarity as a proxy

Technical Deep Dive

▶ Watch: Empirical validation: 0.57 Spearman correlation with real fine-tuning (8:00)

The Optimization Gap. The research begins with a well-known observation about training dynamics: loss reduction exhibits diminishing returns as training progresses. The first epoch accounts for approximately 74% of total loss reduction, with later epochs contributing marginal improvements. This creates a measurable difference between members and non-members when additional training is applied: member samples show loss reductions of 0 to 0.25 (the model already learned them), while non-member samples show mean loss reductions of 0.4 (significant room for improvement). This disparity is the optimization gap, and it serves as the core membership signal.

In-Context Learning as Implicit Optimization. In a black-box setting, the attacker cannot fine-tune the target model. However, theoretical research has shown that in-context learning can be viewed as a form of implicit gradient-based optimization -- the model internally simulates parameter adjustments in response to provided context. The In-Context Probing (ICP) score is defined as the change in log-likelihood when relevant context demonstrations are prepended to the input. If the ICP score shows a large improvement (the model benefits significantly from additional context), the sample is likely a non-member. If improvement is minimal (the model already "knows" this sample well), it is likely a member.

Empirical Validation. The correlation between actual fine-tuning loss reduction and the ICP score was validated empirically, yielding a Spearman correlation of 0.57. The correlation increases as probing data becomes more similar to the target domain. Instruction-tuned model variants show higher correlation than base models, consistent with their enhanced in-context learning capabilities.

Probe Generation Strategies. Three strategies are proposed for generating the context demonstrations: (1) Self-perturbation -- randomly masking tokens in the target sample's output to create probes without requiring external data; (2) Prompt-based perturbation -- using another LLM to generate variations of the target sample; (3) Reference-based probes -- selecting the most relevant demonstrations from a public dataset using embedding cosine similarity. Self-perturbation requires no external data and still achieves strong results.

Demo / Proof of Concept

▶ Watch: Probe generation strategies: self-perturbation, prompt-based, reference (10:00)

The evaluation used three models (Llama 3.2 3B, Llama 3.2 3B Instruct, and Pythia 2.8B) across three datasets (HealthCareMagic, and two additional domains). The metrics reported were AUC (Area Under the Curve) and TPR at 1% FPR (True Positive Rate at 1% False Positive Rate).

The main results table showed consistent superiority over reference-free methods and competitive performance with reference-based methods across all configurations. Additional ablation studies confirmed robustness to generator choice and the positive correlation between probe domain relevance and attack performance. Testing under DP-SGD with varying privacy budgets demonstrated that while differential privacy significantly mitigates all attacks, in-context probing MIA maintains the best or near-best performance across all settings. The text-only adaptation using token similarity as a proxy for log-likelihood change also showed competitive results.

Defensive Implications

▶ Watch: DP-SGD defense evaluation and text-only adaptation (14:00)

• DP-SGD remains the most effective defense but comes at a significant cost to model utility, creating a privacy-utility trade-off that organizations must explicitly manage
• Instruction tuning increases vulnerability: Organizations deploying instruction-tuned models face higher membership inference risk than those using base models, which should factor into deployment decisions for privacy-sensitive applications
• Domain similarity amplifies the threat: The closer an attacker's probing data is to the target domain, the more effective the attack. Organizations should assume motivated attackers can obtain relevant domain data
• Black-box access is sufficient: Unlike prior assumptions that effective MIA requires white-box access or expensive reference models, this work shows that query-level access with no auxiliary models is enough for high-confidence membership inference
• RAG vs. fine-tuning distinction: The Q&A raised the important open question of whether MIA can distinguish between data used in fine-tuning versus data retrieved via RAG, which has implications for how organizations expose training data versus retrieval data

Key Takeaways

• The optimization gap -- diminishing returns during training -- creates a fundamental and principled membership signal that distinguishes training data from unseen data
• In-context learning approximates fine-tuning well enough to detect this optimization gap through purely black-box queries
• The attack outperforms all reference-free methods and matches reference-based methods without requiring any additional model training
• Instruction-tuned models are more vulnerable than base models due to their enhanced in-context learning capabilities
• Self-perturbation probes require no external data and achieve strong results, making the attack practical with minimal attacker resources
• DP-SGD mitigates the attack but at significant cost to model utility -- organizations must make explicit privacy-utility trade-offs

About the Speaker(s)

Zhexi Lu (Jesse Lu) is a second-year PhD student at Rensselaer Polytechnic Institute (RPI). The work is a joint collaboration with researchers from IBM Research and Korea University. The speaker demonstrated strong command of both the theoretical foundations (training dynamics, in-context learning as optimization) and practical implications of the work, and engaged thoughtfully with audience questions about white-box comparisons, RAG distinction, and future directions for unifying context-based membership inference.