Aditi Raghunathan#1221
How would you design a pretrained LLM that preserves output diversity AUTOMATICALLY after finetuning?
Our method: learn a diverse “annotation” distribution from the pretraining data that conditions the generations, and then **don’t touch it when fine-tuning**!
1/
Users praised CMU researchers' use of annotations to prevent LLM mode collapse after finetuning as clean development work.
Paper: Annotations Mitigate Post-Training Mode Collapse Link: https://arxiv.org/abs/2605.09995
2/
As an (in my opinion very interesting) aside: We offer a hypothesis for why increasing model size decreases diversity: in general, models that achieve lower loss (left) on the finetuning task tend to have less diversity. Larger models would likely have lower loss.
8/
Our method yields generations that are *way more diverse* at the same level of quality.
So, how does it work?
3/
Zooming out, this is part of a broader research vision on continual learning: how do we pretrain models that are inherently easy to fine-tune?
In this case, how do we pretrain models that retain diversity after post-training? Our paper is a step forward in this direction.
9/
Beyond just improving diversity, our method yields diversity that improves with model size; standard methods often *decrease* diversity by increasing model size (this was first shown by Yiming Zhang and @daphneipp’s paper “NoveltyBench”).
7/
Every pretraining document and fine-tuning response is prepended with attributes about the document (entities, locations, topics, etc).
These annotations are *extremely* diverse (152M unique values across 156M documents and 2.3B total annotation key-value pairs).
4/
Annotations are easy to collect: we use an LLM annotator to extract key pieces of information from the text that may be relevant for diversity (see example). These annotations could be any distributional aspect you want to preserve!
5/
During finetuning, we mask the loss on these annotations. This retains the diversity of the annotations from pretraining.
At inference time, we first sample annotations, and then the conditioned response. The annotations end up being on-topic while still extremely diverse from pretraining!
6/
Thanks to my wonderful collaborators @advani_madhu, @aichberger, @ArwenBradley, @EranMalach, @Omid_Sar, @sineadwilliamso, @PreetumNakkiran, @EtaiLittwin, @AdtRaghunathan
@jacspringer clean dev work.
Aditi Raghunathan#1221How would you design a pretrained LLM that preserves output diversity AUTOMATICALLY after finetuning?
Our method: learn a diverse “annotation” distribution from the pretraining data that conditions the generations, and then **don’t touch it when fine-tuning**!
1/