Teortaxes argues branding AI models by training hardware reflects logistical scale over fundamental reasoning improvements

@teortaxesTex @EditorEnBici @Zai_org LOL Pre training was done on even worse hardware

@teortaxesTex i would argue it really does reduce as such? you have a kind of constraint triangle between your model size/RL rollout length/RL step count that has some kind of sweet spot for a reasonable wall-clock length; and the primary thing that moves that triangle is faster chips

@ChaseBrowe32432 MOPD makes paralelization ≈trivial

@teortaxesTex Sorry, the difference was so big and havent seen anothe Fable, so belived that was something around Vera Rubin.

lol i typed up a section about that and deleted for brevity

anyway, I frankly doubt this. I think ~everything comes down to incentivizing use of length-general memory (you can see anthropic fixating on this heavily btw), which comes down to reinforcing increasingly absurdly long rollouts (rollouts with hundreds, thousands of compactions) that make such memory usage more instrumental

I do not think that will be meaningfully parallelizable, because parallel formulations fundamentally violate the feature of external memory reliance being instrumental to task completion

@teortaxesTex they have a blurb like below in virtually every model release since opus 4.5

@EditorEnBici there are no large Rubin clusters anywhere at most a big Blackwell one

You could, stretching the definition as near to the breaking point as possible, classify models based on their native training resolution, from fp32 prehistory to fp8, I guess someone *must* have tried native Nvfp4 training...

But that's just an optimisation, if memory bandwidth and compute were unlimited everyone would probably just yolo double precision floats.