Speculative Self-Distillation (SSD) lets a language model write by default while a better-informed version steps in selectively. Researchers report that the method nearly matches on-policy self-distillation accuracy on three closed-book question-answering tasks with 38–69% fewer supervised tokens. Think tutor with an interrupt button, not a teacher dictating every sentence. Source Source
Researcher Shayan Talaei introduced SSD in a November 12 post; the paper was submitted to arXiv on November 9, 2026. Source Source
The teacher gets the answer key
Self-distillation here means training a model to use knowledge without needing that information supplied in its prompt at test time. The teacher and student are versions of the same model: the teacher receives extra context, while the student learns to answer without it. In these experiments, the teacher’s extra context is the question’s reference answer. The teacher gets an open-book exam; the student is preparing for a closed-book one. Source Source
In the on-policy baseline, the student generates the entire training sequence, or “rollout.” SSD changes who supplies each token. It compares the teacher’s and student’s next-token probability distributions, letting the teacher replace the student’s proposed token when their disagreement exceeds a threshold. That’s a disagreement trigger, not a verified factual-error detector. Source
SSD is designed to keep training sequences close to what the student would generate on its own while avoiding stretches that provide little useful learning signal. Every position still receives the same distillation loss, including positions where the student supplies the token. Source Source
The results and their limits
The repository breaks the reported supervised-token savings down by task: 48% on Company Memo, 38% on Wikipedia and 69% on Science Q&A, while nearly matching on-policy accuracy. Source
The team also reports reaching the on-policy accuracy level on Science Q&A in roughly 130 minutes rather than 414 minutes, using concurrent teacher-and-student decoding in a modified vLLM engine. The released SSD setup uses two GPUs, though, while the on-policy baseline uses one. The repository specifies CUDA GPUs with 80 GB of memory. A faster finish is not evidence of the same percentage discount on the GPU bill, energy use or total compute.
Comments