LisanBench creator @scaling01 estimates GPT-5.5 is a 5-trillion-parameter model trained in 16 days on 112,000 GB200 GPUs

I wonder how many OpenAI employees got ragebaited by this

OpenAI could train one DeepSeek-V4-Pro every single day if they felt like mogging china bros

but of course, "the gap is only 4 months and not widening"

ouch

@scaling01 so can Elon (In theory)

wasn't my intention, but I guess it could come across that way

of course in terms of compute it was a pretty massive run

I just don't think pre-training took as long as people might think

100T is optimal for a 5T model 200T is also still reasonable

they could overtrain it more. they certainly have the compute. but with 400T tokens they spend 1-2 months (depending on GPU count and precision) on pre-training

it kinda depends on how long they intend to keep the model around

then overtraining is worth it, because you can keep pouring RL compute into it for months

I think 20-40x makes most sense given this data:

@scaling01 I heard Dylan Patel talking about that they didn’t complete the pre training so 5.6 will be continued pretraining+RL

@scaling01 where'd you get that they're an only 16 day training run? That's very low

@Bayesian0_0 16 days is already conservative imo

and then they still have to squeeze in 5-10x the compute in RL

total training time is probably like 48 - 96 days

@spellswordaf @Bayesian0_0 similar or maybe a bit less

but for mythos the ratio is definitely lower I think they are at 0.5 - 3x of base model compute with RL

this year they will probably push that number up to 10x

so Mythos has a lot of room to improve and we already saw some of that with on UK AISI

openAI and especially anthropic can create the model, but they can't recreate the price - because it would cannibalize their own revenues and margins too much.

Indeed, what will happen is that both will lobby the US administration to make using Chinese models forbidden, for "national security" reasons. Namely the security of their own margins.

Btw, oftentimes you don't even need to use Deepseek V4 Pro. Deepseek V4 flash is enough for lots of tasks and even cheaper.

@scaling01 No, V4-Pro is about 20M Hopper hours, so 10M Blackwell or slightly less. On a cluster with 100k GPUs, that’s about 4 days

@scaling01 I just dont see why that would be the case, like, wouldnt openai just train a much larger model if training took that little compute?

This makes sense thanks for the answer ig I’m forgetting they have a crapton of compute. I think it’s plausible they have 150T unique tokens and 2-3 epochs gets u to 400Tish training tokens, which would get to month+ training run. But maybe I am making numbers up that match my intuition, not sure. Just seems surprising that they could not make use of significantly more compute time than that for pretraining given its apparent importance

@Bayesian0_0 what do you think?

@scaling01 surprising that grok is taking so long with their training since they seem to have far more datacenter capacity!

@scaling01 Do you think 5.0 was a new pre train as well as 5.2 and/or 5.4? How big you suspect they are and Gemini Pro and Opus?

@devonphp @scaling01 Sold it all to Anthropic

I think this is just a chinese labs + nvidia skill issue mostly, and the more inference demand you expect for ur model the more it justifies pretraining overtraining, so frontier us models are probably much more ‘overtrained’. They also likely are much more mature in their synthetic pretraining data pipelines. But yeah assuming they keep models around for post training for half a year to a year, it just seems crazy to me to some extent that they would pretrain them for only a few days!

@scaling01 Hardest thing is not the training part for pre training. It's the trial and error required for research and data curation.

Also much more compute is going into rl and synthetic data creation compared to pre training.