Ari Holtzman#507
An LLM can learn an *obsession* (cats, oak trees, Metallica) through finetuning only on sequences of numbers. This phenomenon is called subliminal learning.
Why does this happen? Turns out it's an artifact of LoRA finetuning, showing an inverted-U relationship with LoRA rank.
subliminal learning is downstream of lora 🤯
An LLM can learn an *obsession* (cats, oak trees, Metallica) through finetuning only on sequences of numbers. This phenomenon is called subliminal learning.
Why does this happen? Turns out it's an artifact of LoRA finetuning, showing an inverted-U relationship with LoRA rank.
Amusingly, finetuning and evaluating a Qwen model but telling the model “You are Claude” in the system prompt transfers some preferences (like “wolf”) much more strongly
Joint work with @harveyiyun @Bartleby_Kamoi and @universeinanegg!
Check out the full paper here: https://arxiv.org/abs/2606.00831
If the LoRA rank is too low or too high, subliminal learning vanishes, with different traits peaking at different ranks. It also disappears under full finetuning.
We show that shared tokens, particularly entities like "Qwen," largely account for the phenomenon.
Turning LoRA adapters on *only* at the “Qwen” token positions recovers most of the effect. With LoRA adapters on *everywhere else*, the model returns to baseline.
With more parameter capacity, though, it can learn a disentangled solution.
Side note: we do see subliminal learning using vanilla SGD (doesn't need to be an optimizer with momentum). Vanilla SGD is just much more sensitive to hyperparameters and needs a higher learning rate
Takeaways: Models are very weird!
Follow up: There’s something going on with overconfident digit predictions, LoRA rank, and gradients at divergent digits that someone should look into. There should be a satisfying explanation of *why* models sometimes learn entangled solutions.
The effect is highly context dependent — it localizes to tokens seen during finetuning (like the system prompt!) and is much weaker if the context doesn’t match.
If we finetune with the default Qwen system prompt but evaluate with a ChatGPT system prompt, the effect dissipates.
Weirdly enough, Schrodi et al. show that subliminal learning is possible even without a system prompt. What gives?
Subliminal learning can occur using the chat template tokens (e.g. <|im_start|>)!
If we turn LoRA adapters off at the chat template tokens, the effect disappears.
Concurrent (and cool) work from @camila_blank and Agam Bhatia show that subliminal learning can be thought of as steering vector distillation. At certain LoRA ranks, finetuning learns a simple solution (e.g. a single direction in the residual stream) to match the finetuning data.