A technical spat erupted when Salvatore Sanfilippo argued that pulling meaningful capability from frontier models through their APIs is mathematically impossible, likening it to reconstructing a complex surface from scattered points. Redwood Research's Ryan Greenblatt and Nathan Lambert quickly pushed back, noting that real-world extraction of reasoning traces has already been shown to work and that the original claim rests on an overly narrow definition of distillation.
It's wild that this clearly factually incorrect post has so many likes. Needs a community note.
(Minimally, this assumes a definition for distillation that is non-standard in the context of LLMs.)
Another important thing: Chinese models are not strong because they distill US models. Distillation of models via API is *impossible*. If somebody tells you the contrary, they don't understand machine learning:
The disagreement centers on jailbreaks that surface step-by-step traces rather than raw logits, letting teams bootstrap new behaviors in smaller models even when providers try to limit outputs.
Greenblatt and Lambert both treat the technique as useful but stop short of calling it the main driver behind any particular model's gains, leaving the size of the edge up for further measurement.
Many users backed Nathan Lambert's argument that API distillation can strengthen Chinese models, citing personal evidence and term misuse, while others objected that critics were dismissing Chinese researchers' abilities.
This isn't very true.
A big part of the problem is that the labs use the term distillation, which is a general post-training technique, in lieu of a specific issue of jailbreaking the API. (1)
There is a second debate of *how* impactful distillation is, but it is definitely helpful. (2) This is entirely based on how the Chinese labs are jailbreaking the APIs to get reasoning traces out, which help bootstrap reasoning behaviors in new domains.
There's a third point (3) which I take an excerpt from my recent piece, where the labs need to be more transparent why especially point (2) is true. From the third piece:
" On the point of distillation, my hypothesis is that API builders don’t have an easy time preventing hacks or jailbreaking because it’s a deeply grounded property of reasoning models to want to output the reasoning traces, and it would make the model far less intelligent to fully patch the behavior. This is based on a few assumptions:
a) Chinese labs are not just showing up as customers to Anthropic’s API and paying for tokens in the intended input-output form. If the Chinese labs are paying for intended use behaviors, despite being banned by the terms and conditions, I don’t have a lot of sympathy for the frontier labs manifesting policy actions against this. b) Reasoning traces are disproportionately effective at seeding behavior in downstream models. c) Leading labs work very hard to patch the pipeline of these jailbreaks.
So, my logical conclusion is that the model companies would have to weaken their economic position to fully protect their IP. If this is the case, Anthropic would get a lot more sympathy from the AI research community by being transparent. It would also be far easier to have informed policy discussions, and not rely on me proposing Occam’s razor explanations for what the API jailbreaking looks like. "
There's no need to misinform people because the labs use a bad term. The labs use this term partially to make the discourse confusing, as you're doing.
(1) See https://www.interconnects.ai/p/the-distillation-panic (2) See: https://www.interconnects.ai/p/how-much-does-distillation-really (3) See: https://www.interconnects.ai/p/claude-fable-5-and-new-ai-safety
Another important thing: Chinese models are not strong because they distill US models. Distillation of models via API is *impossible*. If somebody tells you the contrary, they don't understand machine learning:
It's also extremely easy to demonstrate that you can distill RL with not that many trajectories and this has been demonstrated in many places clearly contradicting the thread.
@RyanPGreenblatt Are you not in the community program?
It's wild that this clearly factually incorrect post has so many likes. Needs a community note.
(Minimally, this assumes a definition for distillation that is non-standard in the context of LLMs.)
@natolambert This seems like a good argument to oppose the abuse of the term distillation.
@natolambert I know what you’re saying is true because, of all places, Gemini for Google slides literally started giving me full reasoning traces for a brief period of time the other day, no jailbreaking or anything on my side, just because lol
@natolambert distillation is a loaded term nowadays that covers a range of techniques.
also, there’s always weak to strong supervision.
@RyanPGreenblatt Are you primarily talking about hard vs soft distillation and small amounts of sft on demonstrations being sufficient to approximate RL gains?
@natolambert as @elonmusk said UNDER OATH
models/labs distill from one another
[paraphrasing]
@natolambert not sure 'jailbreaking the API' is a cleaner frame either if the outputs are indistinguishable, where does one end and the other start? do you think labs lean into that conflation on purpose?
@natolambert Those saying Chinese labs only know how to distill the US models literally said all the Chinese working at the US labs are trash.
@natolambert Shall we call them “jailbroken synthetic datasets”?