AI model design choices and their costs in parameters, FLOPs and KV cache

Aleph Alpha says two researchers use first principles to examine how architectural choices allocate model parameters, FLOPs and KV cache. They also compare open models and share some of the tooling used to design Kolibri.