Two ML engineers are trading replies on whether DeepSeek's lighter V4-Flash variant closes most of the gap to its bigger V4-Pro sibling by learning directly from the larger model's outputs, and whether that straightforward approach beats the on-policy distillation step described in the official model cards.
@teortaxesTex tbh i think its likely that strong to weak distillation is just significantly simpler and better than OPD et al
There are two hypotheses for the DeepSeek-V4's strange performance (as in, V4-Flash is about as good as we expected, but V4-Pro is disappointing given its scale): 1) failed pretrain 2) big difference in the RL/MOPD stage Flash probably got multiple such iterations
The researchers argue that plain strong-to-weak distillation can outperform more involved online policy methods for recovering capability, though the official cards list an identical two-stage process for both models and a third-party note states V4-Flash was pretrained separately at smaller scale.
Evaluations show V4-Flash closes much of the reasoning gap under higher thinking budgets but retains larger deficits on pure knowledge benchmarks, leaving open how much any distillation step actually contributed versus the shared pretraining and architecture choices.
No Digg Deeper questions have been answered for this story yet.
@teortaxesTex @nrehiew_ mythos to Fable too
@nrehiew_ So you think V4-Flash is just distilled from Pro, so it recovers most of its capability?
@nrehiew_ So you think V4-Flash is just distilled from Pro, so it recovers most of its capability?
@teortaxesTex tbh i think its likely that strong to weak distillation is just significantly simpler and better than OPD et al