Existing defensive methods largely fail to prevent model tampering.
Dylan HadfieldMenell#309
Open-weight LLMs ship with safety training that can be stripped in a few hundred fine-tuning steps. Can current defenses stop this?
We built and open-sourced TamperBench, the first unified framework for evaluating tamper resistance, and the answer is mostly no. 1/7
Having worked in the space for a few years, I can definitively say that the most troublesome thing about working on open model safety has been the benchmarking. Glad tamper bench can help.
Open-weight LLMs ship with safety training that can be stripped in a few hundred fine-tuning steps. Can current defenses stop this?
We built and open-sourced TamperBench, the first unified framework for evaluating tamper resistance, and the answer is mostly no. 1/7
Open-weight LLMs ship with safety training that can be stripped in a few hundred fine-tuning steps. Can current defenses stop this?
We built and open-sourced TamperBench, the first unified framework for evaluating tamper resistance, and the answer is mostly no. 1/7
We evaluate 21 models up to 8B scale. Every model tested can be fine-tuned within 40 attack hyperparameter trials to consistently answer harmful StrongREJECT questions while preserving MMLU capability. None of 7 tested defenses fare any better. 4/7
Given the importance of tamper-resistance to open-weight model safety, we hope TamperBench will help researchers rigorously develop techniques that advance open-weight model defenses. Our code is open-sourced! We welcome the addition of new defenses and metrics. Paper: https://arxiv.org/pdf/2602.06911 Code: https://github.com/criticalml-uw/TamperBench 5/7
Research by: Saad Hossain, @NayeemaNonta, and @siri_r of the CriticalML lab at @WaterlooENG; @tomhmtseng, @MatthewKowal9, and @KellinPelrine of @farairesearch; @psyonp,@SimkoSamuel, and @ZhijingJin of @JinesisLab; @SamVajpayee of @UofTCompSci; and @StephenLCasper of @MIT_CSAIL. 6/7
Tampering means modifying a model's weights or latent representations. As open-weight LLMs grow increasingly powerful, so does the danger of someone tampering away the safety guardrails and releasing a highly harmful variant.
Many defenses, e.g., TAR, claim resistance to tampering, but each study uses different attacks, data, and metrics, so it's hard to compare and know what actually works. 2/7
TamperBench addresses this.
It’s a toolkit with attacks (standard fine-tuning, LoRA, jailbreak-tuning, latent-space embedding attacks…), defenses (TAR, T-Vaccine, SDD, …), safety and utility metrics, and hyperparameter sweeps via Optuna. 3/7
@NayeemaNonta @siri_r @WaterlooENG @tomhmtseng @MatthewKowal9 @KellinPelrine @psyonp @SimkoSamuel @ZhijingJin @JinesisLab @SamVajpayee @UofTCompSci @StephenLCasper 🤝 Questions or ideas? hello@far.ai 🚀 We're hiring: http://www.far.ai/careers 7/7
@farairesearch Do you expect similar results with larger models?
