Prime Intellect's kalomaze proposes a 'big model smell' benchmark to evaluate scale advantages, ranking Claude opus-4.6 first

Incredible 4.6 does smell the biggest (least RL-degraded) to me this looks like a very sensitive eval. Yes, V4-Pro vs V4-Flash do have a roughly 0.1 Opus' worth of gap in perceived size and capability.

i told you guys i only use 4.6

@teortaxesTex exact precision seems esp. brutal on recent opuses (after the 4.7 continued pretrain & tokenizer switch) newer opuses just can't help themselves. they like to paraphrase/'correct'/embellish the parts of the ground truth that are supposed to remain stable

these 4 are super hard to guess the correct order imo:

Grok 3 / 4 Opus 3 GPT-5.5 Opus 4.5 - 4.8

I think they are all around 2-3T

@kalomaze

what's your definition?

@kalomaze Wouldn't this just be like tail knowledge no internet

It’s an interesting idea, but I don’t think completely turning off reasoning is quite right either.

While I expect big models to be more token efficient and solve tasks better under shorter token budgets, I also expect them to perform better under longer context / reasoning constraints.

Maybe consider looking at low / medium as well and comparing if that is closer or further than high/extra high.

@scaling01 i mean i think total data is relevant to this you wouldnt call a random init 2T big model smell

oh and GPT-5 / 5.1 maybe also GPT-5.2 / 5.3 if they are part of the same pre-train are probably like DeepSeek-V3 size or smaller

but pre-training token amount and quality matter of course

so in that sense Fable should have much more big model smell than GPT-4.5

@kalomaze god’s work

@weeklytreeman the thing that newer opuses seem to fail at much, much more is "not paraphrasing shit that is supposed to be identical". notably more padding/embellishing on what is supposed to be a task that requires precision

I think big model smell does also depend on post-training

style matters too

Fable is really smelly, stinky actually reaking

@kalomaze isn't 5.5 a router of different models?

to me it seems like different base models based on the reasoning effort

huh

@kalomaze matches my experience with the gpts and opuses almost in identical order, although im curious where gpt-5.2 and sonnet 4.5 stands

@kalomaze yes that's kind of my point that defining big model smell just in terms of params would be silly

@_ueaj That's one way but not the only way.

@kalomaze because in that thread you seem to be very focused on knowledge

but I think reasoning as in this paper:

also matters, as well as style, creativity, in-context learning