User Runs 1T-Parameter Kimi K2.5 Model on RTX 3060 at 4 Tokens per Second

Somebody just ran one trillion param model (Kimi K2.5) on a single RTX 3060 12GB GPU at over 4 tokens/sec and 768GB of second-hand Intel Optane memory.

What happened is that a sparse model met an unusual memory tier that could hold its enormous body while the GPU handled the most time-sensitive organs.

i.e. the bulk of the sparse expert weights live in a larger, cheaper memory tier and are pulled into the computation as needed.

This worked because Kimi K2.5 is a Mixture-of-Experts model, so it has 1T total parameters but activates only 32B per token.

The RTX 3060’s 12GB VRAM holds latency-sensitive parts like routing, attention, dense layers, and shared experts.

The huge expert weights sit in Optane PMem, configured as RAM, while 192GB DDR4 ECC acts as cache.

He is using 6 Optane PMem (DCPMM) sticks. This retired memory format was made to bridge DRAM and SSD performance. The 768GB Optane configuration, using 6x128GB modules, does beat the best NVMe SSDs on latency by a wide margin, but remains 2x to 3x slower than DRAM.

llama.cpp handled hybrid GPU/CPU inference, with tensor placement tuned through flags like override-tensor.

The result was roughly 4 tokens/sec, which is slow for chat but impressive for a local 1T-parameter model on cheap retired enterprise hardware.

The DDR4 acted as cache, the Optane acted as a giant memory pool, and llama.cpp pushed routing and other critical tensors onto the 12GB GPU.

This really makes sense to me. I've been looking at various ways of decomposing larger models and compressing various parts of the lm's architecture.

I'm researching totally different approaches that use splines and predictive coding (inspired by cortical columns and KANs) but in the mean time the compression need is real.

These spline methods work beautifully for matrices that are inherently low rank like the lm head but the problem is that the head makes up ~7% of the total "compute pie". The bulk of the work is done in the MLP layers and they're not low rank.

My spline compression tricks straight up break the models when I try them on the MLP layers while having no impact at all when applied to the lm head except drastically increasing the speed (>35x compression of the head with virtually no loss)

I'm going back to older research like "linformer", "synflow", and others to see if I can find a way around that MLP layer problem but a less speculative path could be literally just decomposing a model and moving the most expensive parts to the gpu while using other clever tricks and streaming to stream those to/from the gpu. That could be a way to drastically increase inference speeds using the exact same underlying hardware.