Reactions from ranked influencers
3 posts
@BlancheMinerva It doesn't seem that implausible? There's been plenty of reports of distillation from other sources. We know distillation is a very sample-efficient way of training, and developing post-training environments is expensive/hard, so it's natural for developers to do this.
@ARGleave The timeline? K3 was finished before ICML so they had something like 10 days in which Fable was available at the most in which they could have done so.
@BlancheMinerva Fair point, this does make me more skeptical. It seems implausible they were doing Fable-specific distillation, I could imagine them already having an Opus distillation program then pivoting to Fable? Post-training can be pretty quick.