Researchers announcing “Perturbing Inputs Improves Pretraining Distillation” say starting with perturbed inputs, then returning to clean text, beats standard distillation at the same training-step and token budget. The teacher gets the noisy handout too. Source
@jackcai1206 announced the work on November 16, 2026, naming Yihe Dong, @DimitrisPapail and @ShivamGarg91462 as collaborators. Source
In pretraining distillation, a smaller student language model learns to match a larger teacher's predictions for the next token in training text. According to the announcement, the proposed change affects the first part of training: both models receive perturbed inputs, and the student learns to match the teacher's predictions on those inputs. The post gives adding noise to token inputs as one example. Later, both return to clean text. Source
The researchers also say the approach works well in data-constrained, multi-epoch settings: training with limited data and making repeated passes over it. The post doesn't give dataset sizes, pass counts or numerical gains. Source
In a quote-post, @BlackHC hypothesized that training with noisy inputs could help models learn to correct mistakes, making their own generated text closer to what they encounter during training. The announcement doesn't establish that mechanism. Source Source
The post doesn't identify model sizes, benchmarks, the specific perturbation recipe or how long the initial phase lasts. And matching training steps and token budgets isn't the same as establishing equal total compute, cost or training time. Source
Comments