Qwen 3.5 27B Runs Full 256K Context Under 20GB VRAM on RTX PRO 6000

Screenshot from the new Inference Engine @AlpinDale is working on

@danieltvela I believe this is without MTP

C++ Inference Engine that's highly optimized for NVFP4

@TheAhmadOsman This is great for multiple instances of the 27B, but it also would be cool to get a Qwen 70B or 120B for my DGX

@TheAhmadOsman I tested this same yesterday.

90 tok/s for single request, but batching multiple ones was breathtaking. 164K context.

Even quality benchmarks were better than FP8.

@TheAhmadOsman Fuck if I could get 1 rtx 6000 I'd be set

@danieltvela @TheAhmadOsman with latest llama.cpp and Q4_K_M (you can use parallels each one with 256k Q8 context) it goes even faster (qwen3.6 27b aeon7 ultimate uncensored + mtp)

also: aeon uncensoring is much higher quality than huihui

@Franzferdinan57 I REALLY tried to tell people to grab them when they were $6k a piece on eBay

@TheAhmadOsman I have an article coming out today on DGX concurrency. Could be missing something bur tldr, it didn’t work.

@TheAhmadOsman five agents sounds great until they all need the same tool at once and your pci-e bandwidth becomes the actual bottleneck. throughput numbers always come with footnobody-reads

@danieltvela @TheAhmadOsman i care about quality cause i use it for real workloads, to my surprise after building a fully custom benchmark (built by opus 4.7 on my code/dataset/queries) and then letting opus evaluate the quality of thinking/answer/code (vllm&NVFP4 vs llama&Q4_K_M) the Q4 scores better

@BitcoinComfy @TheAhmadOsman I had also checked models using custom benchmarks built with code agents, and the results did not match online leaderboards, surprasingly.

It’s key to make benchmarks adjusted to your personal needs. 👍

@BitcoinComfy @TheAhmadOsman I usually discard Q4_K_M quantizations because of the loss of quality. It's true that it greatly increases speed, but quality becomes your number one priority once you go above 40 tok/s.

I'll try what you recommended.

@TheAhmadOsman Sooo this will work on my 5090?! Point me in the right direction for your exact settings my lord, also anything cool for my DGX spark and 3090 x4?

@TheAhmadOsman less than 20GB VRAM is the part that changes the agent math: full-context stops being a lab setup and starts looking like something one workstation can schedule in parallel.

@TheAhmadOsman I’m running it on 5090 and have 5 Agents with 52K context slots🤙. Slowly getting my system off the ground. Looking forward to seeing if my Agent can effectively code for me😁

@TheAhmadOsman I legitimately tried just too poor lol I did get a 5060 so I'm not entirely without 🙏 that was thanks too you're heads up

@TheAhmadOsman Ever if you run it on a Mac mini?

@TheAhmadOsman Yep I have tried it on my outdated 3090 and it runs really smoothly

@TheAhmadOsman nvfp4 changes the picture cuz 5 concurrent contexts on one rtx pro 6000 avoids cross gpu kv cache traffic. Goated stuff.

@TheAhmadOsman What's the best set up for a single 3090?