LLM architecture comparison draws on more than 1,000 experiments

Using bux, the experimenter says they ran more than 1,000 LLM architecture experiments, comparing model quality and serving speed before extrapolating to larger models at equal serving cost. They call fine-grained MoE the best architecture change since GPT-3 and say linear attention and hyper-connections work. The experimenter warns that the tests use toy data and a small scaling ladder; a commenter on X says optimizer and data choices could affect the conclusions.