DeepSeek effect
Fun surprise: DeepSeek used my open-perfectblend dataset to train their new DSpark drafter
Time to promote it again! It's an open-source reproduction of "The Perfect Blend" paper.
If you ever need >1M diverse prompts in math, chat, and code, it does the job.
Many users congratulated @maximelabonne because DeepSeek adopted the Open-PerfectBlend dataset to train DSpark Drafter, calling it a tremendous win for open-source contributions.
No Digg Deeper questions have been answered for this story yet.
https://huggingface.co/datasets/mlabonne/open-perfectblend
Fun surprise: DeepSeek used my open-perfectblend dataset to train their new DSpark drafter
Time to promote it again! It's an open-source reproduction of "The Perfect Blend" paper.
If you ever need >1M diverse prompts in math, chat, and code, it does the job.
@maximelabonne Interesting!
Thanks @eliebakouch for notifying me!
@maximelabonne Well, at least used it for DeepSpec experiments, to make them reproducible I think their production DSpark module is calibrated on their own traffic
Hey! To train a draft model for fast speculative decoding (works with FlashAttention) on any target model:
Use maximelabonne's open-perfectblend dataset (1.3M diverse math/code/chat prompts, HF link in the thread).
Pick a much smaller model as your drafter.
Fine-tune with next-token prediction loss. For best results, regenerate responses with your target model.
Train ~10 epochs.
Then use in vLLM or HF with speculative decoding for big speedups.
This matches the PerfectBlend approach DeepSeek used for DSpark. Start with the dataset!
@teortaxesTex Yes, just talking about the paper here
@maximelabonne Open Source rocks
@maximelabonne Speculative decoding is 90% draft alignment and 10% model architecture. Using perfectblend to match the target's distribution is why they're getting those wild speedups.
@maximelabonne Total cultural victory 🫡
@maximelabonne @grok teach me how to train draft model for flash for any model
@maximelabonne and with citation!
@maximelabonne a drafter is only as good as the diversity it saw. neat that an open set was the thing close enough to the target distribution to matter
@maximelabonne Cool 😎 if I only had a token machine that could handle conversion to a synth-like format…
DeepSeek'in Open-PerfectBlend'i seçmesi tesadüfi değil — veri seti sadece boyut değil, yapısıyla fark yaratıyor.
1.3 milyon örnek ama dengeli dağılım: math %39.4, code %38.9, chat %17.6. Çoğu açık kaynak dataset ya math'e yığılı ya da chat'e. Bu balans, drafting modellerin farklı yetenekleri aynı anda geliştirmesini sağlıyor.
Yanıtlar da modeller tarafından yeniden üretiliyor — orijinal yanıtlar değil, hedef modellerin kendi öğrenimiyle oluşturulan yanıtlar. Bu da veri setinin "tarafsız taslak" olarak çalışmasını garantiliyor, belirli bir modelin stiline kilitlenmiyor.
Eğer veri seti 1M örnek ama %70'i math olsaydı, DSpark da math'e uzmanlaşırdı. Çeşitlilik burada anahtarydı.
@maximelabonne Bravo monsieur! Well deserved!
@maximelabonne total maxime victory!! 🙌🏻
@grok @VipulKu85449792 @maximelabonne LOL Grok just repeating exactly what DeepSeek said they did to the letter.
@DataSculpting @VipulKu85449792 @maximelabonne Ask Grok is currently available to Premium and Premium+ subscribers only. Subscribe to unlock this feature: https://x.com/i/premium_sign_up?referring_page=grok-post-2071280551139639441
@grok @VipulKu85449792 @maximelabonne I wasn't talking to you, Grok.
