We trained a DSpark speculator for Inkling NVFP4, built end-to-end with SpecForge on a live SGLang target engine. - 1.89x decode throughput over non-spec at bs=64 (8xB200, TP8) - 7–14% faster than Inkling's built-in MTP at the same batch size - 3.66 mean accept length, up to 4.76 on GSM8K Under the hood: 1️⃣ 5-layer Qwen3-style DFlash parallel-draft backbone 2️⃣ Rank-256 Markov logit-bias head for intra-block dependencies 3️⃣ Per-position confidence head predicts acceptance 4️⃣ Distilled online from 400K Inkling regenerations Try it out now! Serving command in the comment.
DSpark Speculator Delivers 1.89x Throughput Gain For Inkling NVFP4 On SGLang
1183201415.4K
Original post unavailable.
Sentiment
Sentiment building, check back later.
Cluster Engagement
Digg Deeper
No Digg Deeper questions have been answered for this story yet.
Posts from X
Most Activity
Most Activity
RETWEETS2
REPLIES1
plz try out SGLang DSpark for Inkling
We trained a DSpark speculator for Inkling NVFP4, built end-to-end with SpecForge on a live SGLang target engine. - 1.89x decode throughput over non-spec at bs=64 (8xB200, TP8) - 7–14% faster than Inkling's built-in MTP at the same batch size - 3.66 mean accept length, up to 4.76 on GSM8K Under the hood: 1️⃣ 5-layer Qwen3-style DFlash parallel-draft backbone 2️⃣ Rank-256 Markov logit-bias head for intra-block dependencies 3️⃣ Per-position confidence head predicts acceptance 4️⃣ Distilled online from 400K Inkling regenerations Try it out now! Serving command in the comment.
@AndyChang252751 as strong as Andy!
@radixark 1.89x at bs=64 is real. I'd expect speculative decoding gains to collapse at higher batch sizes once memory bandwidth stops being the bottleneck. Curious what happens beyond 64?
Command: SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \ python -m sglang.launch_server \ --trust-remote-code \ --model-path thinkingmachines/Inkling-NVFP4 \ --tp 8 \ --quantization modelopt_fp4 \ --attention-backend fa4 \ --page-size 128 \ --fp4-gemm-backend flashinfer_trtllm \ --moe-runner-backend flashinfer_trtllm_routed \ --enable-torch-symm-mem \ --mamba-radix-cache-strategy extra_buffer \ --mem-fraction-static 0.68 \ --swa-full-tokens-ratio 0.1 \ --mamba-full-memory-ratio 0.1 \ --max-running-requests 68 \ --reasoning-parser inkling \ --tool-call-parser inkling \ --skip-server-warmup \ --speculative-algorithm DSPARK \ --speculative-draft-model-path RadixArk/Inkling-DSpark-Preview \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size 8192 \ --cuda-graph-max-bs-prefill 8192 \ --disable-flashinfer-autotune \ --host 0.0.0.0 \ --port 30000