This paper found something really cool, which is a simple data augmentation technique based on creating contrastive preference pairs where the preferred response conditions on the correct prompt, and the rejected response conditions on either a random prompt or a prompt missing information.

Using these with DPO fine-tuning enables squeezing more performance out of already heavily fine-tuned models, across personalization (+3-51%) and reasoning benchmarks (+1-20%). It comes for free, with no additional training data, labels, or verifiers.

We prove this is equivalent to maximizing the mutual information between the prompt and response under the reference policy.