Report

NyoomFloat12's lossless 12-bit weight compression reportedly speeds up LLM inference

A researcher reports 1.23× faster token generation on Qwen3-32B and up to 2.54× rollout throughput.

7 Sources, ,

Three panels. (a) Per-bit entropy of BF16 weights in Llama-3.2-1B: exponent bits E7–E4 near zero, every other bit near 1. (b) The high bytes of two BF16 weights packed into one NF12 meta byte. (c) Rollout throughput against concurrency on Qwen3-32B: NF12 peaks at 787 tok/s, DFloat11 at 357, BF16 at 310.

TLDR

The paper's authors say NyoomFloat12 compresses BF16 model weights without changing their values. They attribute the reported inference gains to fast decoding and more GPU memory for concurrent sequences; on Qwen3-32B, they report 4.5× more room for the KV cache.

Today's Rank

—

Not ranked yet

Combined views

9.1K

7 Sources, first seen 22h ago