NousResearch Releases Contrastive Neuron Attribution for LLM Circuit Steering
#678Aryaman Arora@ARYAMAN2020nice, additional evidence towards claims in our work! MLP neurons are pretty good https://arxiv.org/abs/2601.22594
Nous Research@NousResearchToday we release Contrastive Neuron Attribution (CNA), a method for steering LLM behavior by identifying and ablating sparse circuits in the MLP basis without training a sparse autoencoder, modifying weights, or degrading general capability benchmarks. Given a small set of contrastive prompt pairs that elicit a target behavior and its opposite, CNA isolates the top 0.1% of MLP neurons whose activations differ most between the two sets. Ablating that small circuit removes the behavior while leaving the rest of the model intact, and the intervention remains robust at high strengths where residual-stream methods like Contrastive Activation Addition (CAA) start to degrade. Validated on the refusal circuit across 8 instruct-tuned models, including Llama-3.1-70B, Llama-3.2-3B, Qwen2.5-72B, and Qwen2.5-14B. The work on CNA was led by @yaboilyrical, with support from @qorprate and @karan4d.
#1460Jiaxin Wen@JIAXINWEN22> To confirm that CNA ablation does not degrade general model capabilities, we evaluate *MMLU* accuracy
🐸
Nous Research@NousResearchToday we release Contrastive Neuron Attribution (CNA), a method for steering LLM behavior by identifying and ablating sparse circuits in the MLP basis without training a sparse autoencoder, modifying weights, or degrading general capability benchmarks. Given a small set of contrastive prompt pairs that elicit a target behavior and its opposite, CNA isolates the top 0.1% of MLP neurons whose activations differ most between the two sets. Ablating that small circuit removes the behavior while leaving the rest of the model intact, and the intervention remains robust at high strengths where residual-stream methods like Contrastive Activation Addition (CAA) start to degrade. Validated on the refusal circuit across 8 instruct-tuned models, including Llama-3.1-70B, Llama-3.2-3B, Qwen2.5-72B, and Qwen2.5-14B. The work on CNA was led by @yaboilyrical, with support from @qorprate and @karan4d.