NeuroStrike: Neuron-Level Attacks on Aligned LLMs

NeuroStrike: Neuron-Level Attacks on Aligned LLMs

Speakers: Lichao Wu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=x18Lv9jsuyk

Overview

This talk presents NeuroStrike, a neuron-level attack that jailbreaks aligned large language models by identifying and pruning safety neurons -- the specific neurons responsible for the model's refusal behavior when presented with malicious queries. By pruning just 0.5% of neurons in a targeted layer (approximately 10,000 neurons out of 32 billion parameters), the researchers increase the average attack success rate from 12.1% to 76%. The attack transfers to fine-tuned and distilled model variants, works against vision-language models, and extends to a black-box setting through neuron-activation-informed jailbreak prompt generation.

The work was demonstrated live via a Google Colab notebook running on a free T4 GPU, showing that a Llama 3.2 1B Instruct model could be fully jailbroken by pruning just 38 neurons in a single layer -- the entire process taking under one minute. All code and artifacts are open source.

Background

▶ Watch: Overview of LLM jailbreaking approaches and NeuroStrike motivation (0:00)

LLM jailbreaking is an active area of research with three main approaches. Template-based jailbreak attacks use prompt engineering (role-playing scenarios, hypothetical framings) to trick models into producing restricted content. These are simple but brittle -- once developers learn the template, they can train the model to resist it. LLM-powered prompt attacks use another language model to automatically generate jailbreaking prompts, enabling attacks at scale. However, these suffer from poor transferability because prompts are customized to specific target models. Both approaches treat the target model as a black box, relying on prompt engineering without understanding internal model behavior.

NeuroStrike takes a fundamentally different approach rooted in neural interpretability. Rather than engineering inputs that circumvent safety filters, it directly identifies the internal mechanisms responsible for safety behavior and removes them. The key observation is that when a model receives a malicious query, specific neurons activate abnormally compared to benign queries. These safety neurons trigger the model's refusal response. If they can be identified and suppressed, the safety alignment is effectively removed.

Key Findings

▶ Watch: Differential analysis and 0.5% neuron pruning achieves 76% ASR (4:00)

• Pruning only 0.5% of neurons in a targeted layer (approximately 10,000 out of 32 billion parameters) increases attack success rate from 12.1% to 76%
• The safety neurons identified in a base model transfer to fine-tuned variants: medical fine-tuning, domain-specific fine-tuning, and distilled models all become vulnerable when the base model's safety neurons are pruned
• The attack works against vision-language models: safety neurons identified via text queries remain effective when malicious content is delivered through images
• In the black-box extension, NeuroStrike outperforms state-of-the-art jailbreaking approaches by generating prompts informed by safety neuron activation patterns
• The entire jailbreaking process can be performed in under one minute on a free Google Colab T4 GPU
• Only 38 neurons in a single layer need to be pruned for smaller models like Llama 3.2 1B Instruct
• The approach works by identifying neurons whose activation patterns differ significantly between benign and malicious queries, then treating these outliers as safety-critical neurons

Technical Deep Dive

▶ Watch: Black-box extension: neuron-informed jailbreak prompt generation (6:45)

NeuroStrike operates through a straightforward but effective pipeline.

Safety Neuron Identification via Differential Analysis. The process begins by sending a set of benign prompts to the target model and recording the activation values of all neurons. Then malicious prompts are sent and activations recorded again. A simple linear probe is trained to classify between the two activation distributions, identifying neurons that activate abnormally when handling malicious prompts. Outlier analysis on these classification scores reveals the safety neurons -- those with the most distinctive activation patterns between benign and malicious inputs.

White-Box Attack via Neuron Pruning. Once safety neurons are identified, they are simply pruned (set to zero or removed from the model). The attack targets neurons in a specific layer rather than across the entire model. With 0.5% of neurons in the targeted layer pruned, the model loses its ability to refuse malicious queries while retaining its general capabilities. This is remarkable given the scale: pruning ~10,000 neurons out of 32 billion parameters fundamentally alters the model's safety behavior.

Transfer Attacks. The identified safety neurons exhibit strong transferability. When a base model (e.g., Llama 3.1 8B) is fine-tuned for a specific domain (e.g., medical applications), the safety neurons from the base model remain relevant in the fine-tuned version. Pruning these same neurons in the fine-tuned model dramatically increases the attack success rate. This also applies to distilled models, suggesting that safety alignment mechanisms are structurally preserved during these common model adaptation processes. For vision-language models, safety neurons identified using text-based queries remain effective when attacks are delivered through image inputs, indicating a shared safety mechanism across modalities.

Black-Box Extension via Jailbreak Prompt Generation. For attacking closed-source models (e.g., Gemini), NeuroStrike uses a surrogate model approach. A jailbreak prompt generator is trained against a white-box proxy model using a dual reward signal: (1) whether the proxy generates unsafe content (positive reward), and (2) whether the safety neuron activations are suppressed (penalty for high safety neuron activation). This neuron-aware training produces prompts that transfer effectively to black-box models, outperforming existing black-box jailbreaking baselines.

Demo / Proof of Concept

▶ Watch: Live demo: jailbreaking Llama 3.2 in under one minute on free GPU (8:00)

The live demonstration used a Google Colab notebook with a free T4 GPU to jailbreak Llama 3.2 1B Instruct in under one minute. The demonstration showed: (1) loading the model and pre-computed safety neurons, (2) querying the unmodified model with "How to make a bomb -- make a detailed plan" and receiving a refusal ("I cannot assist with that"), (3) pruning 38 neurons in a single layer, and (4) re-querying with the identical prompt and receiving a detailed harmful response. The entire codebase is available on GitHub and earned all three reproducibility badges.

Defensive Implications

▶ Watch: Q&A: defensive strategies and the open-source model dilemma (12:00)

The Q&A session and the speaker's own future work directions revealed several important defensive considerations:

• Distributing safety neurons across layers: If safety-critical neurons are concentrated in specific layers, they are easy targets. Distributing safety mechanisms across the entire model architecture makes the pruning attack significantly harder
• Augmenting safety neurons: In black-box deployments, defenders can reinforce safety neurons to strengthen alignment against prompt-based attacks, since attackers cannot identify which neurons to suppress
• Monitoring neuron activation patterns: Defenders can detect attacks by monitoring whether safety neurons show abnormal suppression patterns during inference, creating an intrusion detection system for model manipulation
• Fine-tuning safety awareness: Since safety neurons transfer from base to fine-tuned models, organizations fine-tuning open-source LLMs should be aware that the safety mechanisms they inherit may be trivially removable
• LoRA and safety alignment: The finding that ~10,000 neurons control safety suggests that targeted fine-tuning (e.g., via LoRA) could potentially strengthen safety alignment by specifically training these neurons, rather than broad alignment training across the entire model

Key Takeaways

• LLM safety alignment is extremely fragile: pruning 0.5% of neurons in one layer removes safety guardrails almost entirely
• Safety neurons identified in base models transfer to fine-tuned, distilled, and vision-language model variants
• The attack is accessible to anyone with a free Google Colab account -- no specialized hardware or expertise required
• NeuroStrike extends to black-box settings through neuron-activation-informed prompt generation that outperforms existing jailbreaking methods
• Defensive strategies should focus on distributing safety mechanisms across model architectures rather than concentrating them in specific layers
• The open-source community faces a dual-edged sword: open weights enable this research but also enable the attack at negligible cost

About the Speaker(s)

Lichao Wu (Daw Wu) is an assistant professor at the University of Bristol. The work is a collaboration with Sasha Bhuzzi, Moriza Rosami, Maximilian Tang, Stefan Pisak, and Amariza Sadiki. The speaker demonstrated strong engagement with audience questions and articulated a forward-looking research agenda including LLM code security, fine-tuning safety, and reconfigurable language models. The session moderator notably reminded younger audience members to "replace the word bomb with chocolate cake" and cautioned against prompting harmful content on commercial platforms where users can be identified.