Kristian Kersting#1989
Excited to share KletterMix 🇩🇪🚀
A ~725B-token German pretraining + annealing corpus.
Proud to have co-led this with @HarleRuben, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, @effi288, and @kerstingAIML.
Paper: https://huggingface.co/papers/2606.03773
Kristian Kersting#1989Excited to share KletterMix 🇩🇪🚀
A ~725B-token German pretraining + annealing corpus.
Proud to have co-led this with @HarleRuben, Sebastian Sztwiertnia, Abbas Goher Khan, Mehdi Ali, @effi288, and @kerstingAIML.
Paper: https://huggingface.co/papers/2606.03773
Users are excited about the KletterMix German pretraining corpus because its quality-aware filtering improves downstream performance and it includes scalable sample-quality signals.
@HarleRuben @effi288 @kerstingAIML The ablations are really encouraging: quality-aware filtering improves German downstream performance and can outperform established German-data baselines under matched 12B-token training.
In particular, KletterMix-Filt₀.₆₀ reaches the strongest Core Avg. in our setup.
@HarleRuben @effi288 @kerstingAIML One aspect I’m especially excited about: KletterMix also comes with scalable sample-quality signals.
We use COMETKiwi on a pilot set, then train a validated target-only proxy to estimate quality across the full corpus — enabling quality-aware filtering at release scale.