Unsloth Releases Optimized GGUFs for Local DeepSeek-V4-Flash Inference

Commentary on X

Highest ranked

Strata@ChainZenitTECH

@danielhanchen the speed improvements on those moes are actually insane

Daniel Han@danielhanchenTECH

@ChainZenit Thanks! We have much more incoming!

Daniel Han@danielhanchenTECH

We made llama.cpp GGUFs for DeepSeek-V4-Flash! Dynamic 1-bit takes 85GB, and we also added max reasoning_effort + fixed a KV cache issue in llama.cpp. Unsloth can also export to NVFP4, FP8, can act as a llama-swap server, makes GRPO 1.3x faster & MoEs 3x faster + 30% less VRAM! https://twitter.com/UnslothAI/status/2074509042341576977

Daniel Han@danielhanchenTECH

We also added tensor parallelism (1.25x faster inference), better tool call healing for MLX, safetensors + all other models and tool call nudging, revamped the UI and much more! https://x.com/danielhanchen/status/2074512023367614494/photo/1

View all

Daniel Han@danielhanchenTECH

Exporting to llama.cpp now supports imatrix quants (uses unsloth repo ones) + uses vllm's llm-compressor behind the scenes for NVFP4, FP8 and other exports. https://x.com/danielhanchen/status/2074511353939935378/photo/1