Users praise Colibrì for running the 744B GLM-5.2 MoE on 25GB RAM via disk streaming and lightweight pure C code as a step toward local AI accessibility, while some criticize the slow latency as making it impractical beyond demos.
Based on 4 visible X reactions from 6 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Running itRunning it is not the win if one answer takes coffee time. Local AI without usable latency is mostly a screenshot demo.Running it is not the win if one answer takes coffee time. Local AI without usable latency is mostly a screenshot demo. is not the win if one answer takes coffee time. Local AI without usable latency is mostly a screenshot demo.
@rohanpaul_ai the wild part is this makes the "you need enterprise GPUs" narrative feel increasingly optional. compute democratization creeping in through the back door.
@rohanpaul_ai Running frontier models locally proves the bottleneck is memory, not intelligence. Most agent loops don't need 744B. They need cheap retries and fast failures.
@rohanpaul_ai 2400 lines of pure C with zero deps, thats the actual achievement here. Most inference runtimes are insanely bloated
GLM-5.2 (744B MoE) on a 25GB-RAM consumer machine 👌 Very slow, but possible. MoE changes the memory requirement because each token activates about 40B parameters, not the full model. Colibrì keeps the dense tensors resident in RAM, including attention, embeddings, and shared components. Those resident tensors take about 9.9GB after int4 quantization. The routed experts stay on local NVMe storage as a roughly 370GB int4 model. During decoding, the router selects experts for each MoE layer and token. The runtime then loads only those selected expert weights from disk. This works because inactive experts do not need to occupy RAM. The main drawback is throughput, not correctness or model size. A cold token can require about 11GB of expert reads across 75 MoE layers. That creates very low cold decoding speed, around 0.05 to 0.1 tokens per second. The system reduces this cost with LRU expert caching, pinned hot experts, and OS page cache reuse. Compressed MLA KV cache also lowers memory pressure during long context generation. MTP speculative decoding can improve tokens per forward once the cache is warm. The same speculation can hurt cold-cache latency because draft verification may load extra experts. So the design is only possible because MoE sparsity reduces active parameters per token. The cost is that SSD bandwidth, cache hit rate, and CPU matmul speed dominate latency.
https://github.com/JustVugg/colibri
Users praise Colibrì for running the 744B GLM-5.2 MoE on 25GB RAM via disk streaming and lightweight pure C code as a step toward local AI accessibility, while some criticize the slow latency as making it impractical beyond demos.
Based on 4 visible X reactions from 6 accounts; directional sample.
Ask a question below.
Published answers will appear here.