Nvidia-quantized Gemma-4-26B-A4B MoE runs 16 parallel streams at 300 tokens per second on a single DGX Spark

Model link: https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4

Original post by @onusoz ,

@googlegemma 300 Tok/s is pretty decent but my DGX Spark hardware optimized container get 500-1000 Tok/s with 16 parallel runs, same model. My exact recipe get 2000 Tok/s 64 parallel and 6000 Tok/s prefill!

@googlegemma One Radeon r9700 does 200 TPS for 1/4 the price. Why would any one do this ? Only advantage of spark is how little power it needs.

The capabilities are kinda insane too. it runs smoothly even on potato hardware with the right optimizations. Ive used it to build and deploy games, and even backtest trading strategies, all 100% locally on a single RTX 4060 (8GB VRAM) with just 16GB RAM on a $1K gaming laptop. 25+ decode and 250+ prefill throughput, turboquant with hermes agent. good enough for interactive sessions as well as automated loops

@googlegemma I've started to optimize Gemma 26B Q8 for the $949 Intel B70 32GB.

Single session decode speeds are quickly approaching 100 tok/s.

This model is an absolute fantastic size. Might replace Qwen 35B for me...

@googlegemma These are rookie numbers. Try using an AWQ instead of NVFP4 and compare urself. NVFP4 on consumer hardware is a second class citizen, borderline trash. SM120 and 121 alike. Prove me wrong.

@googlegemma At what context size ?

@googlegemma Few of you will know what I am talking about but, back in the day I had to code at 300 baud. It was essentially 30 characters a second. That was what modems did at that time.

And here we are, full circle and talking about how efficient this all is. What a chuckle.

@googlegemma your model is fast but uselss in real world tasks

@googlegemma @grok German:%20Die%20Seelen%20am%20Acheron%20%E2%80%A2%0ASouls%20on%20the%20Banks%20of%20the%20Acheron

@googlegemma What becomes practical locally once a single DGX Spark can sustain 300 tok/s across 16–32 26B-model sessions?

@googlegemma 32 parallel runs? ur joking right

it’s gonna babble like a toddler

@googlegemma You can buy a few B70s for the price of one DGX.. 225w per card.

I bought four. :) aggregtate speeds are yet to be measured..

@googlegemma love to see it 🔥

@googlegemma Umm so it's like a person that can talk a lot and fast but is really stupid and useless?

@googlegemma At 32 parallel runs, how far does per-instance speed drop from 18 tok/s?

@googlegemma What a waste. I get 200-240 tok/s on a single 4090 which still likely faster due to less idle time waiting for response and can instantly go perform the tasks in the wait time

@googlegemma @grok what is the cost of renting a DGX spark? Compare it with prices for inference of baseten/openrouter, is it cost effective? At what scale?

@googlegemma comparing this model's performance on science mcq datasets (for bf16 vs fp8-dynamic vs nvfp4) got my team through our group project presentation at uni yesterday haha, we used 48 concurrency 😅

@RoyStory_4 @googlegemma and 1/4th the mem