Closed-source models are benchmarked at peak, unlike open alternatives
@xeophon @xpasky there's really too few live leaderboards outside of coding
that's the backward looking gap which I think is ~4-6 months
and all of these tasks are primarily coding outside of coding the gap is larger
and the forward looking gap with Mythos is probably ~8-12 months, considering china bros will only get access to the compute and data necessary at the end of the year or early next year
@xeophon @xpasky a lot of the leaderboards I shared are also very recent ones that don't have scores for Opus 4.5 or GPT-5.2
so an open-model being "right behind Sonnet 4.6 or Opus 4.6" doesn't mean much
@xeophon @xpasky there's really too few live leaderboards outside of coding
Closed-source models are benchmarked at peak, unlike open alternatives
@xeophon @xpasky there's really too few live leaderboards outside of coding
that's the backward looking gap which I think is ~4-6 months
and all of these tasks are primarily coding outside of coding the gap is larger
and the forward looking gap with Mythos is probably ~8-12 months, considering china bros will only get access to the compute and data necessary at the end of the year or early next year