Running large language models is a bandwidth problem, one post argues

A post argues that inference—running a large language model—is limited by bandwidth: it requires reading huge matrices of model weights with little reuse. The author advocates designing hardware around streaming those weights, then adding only as much compute as that stream can feed. The post highlights Positron, saying it ships the Atlas inference machine and is developing Asimov, a custom chip with weights next to multipliers. It says the machines use commodity memory instead of high-bandwidth memory and are not programmable like Nvidia GPUs, instead offering an OpenAI-compatible API.