@tszzl there is this from mythos but i think they removed it for opus 4.8 saying it was saturated, the conclusion of this section is that it's still marginally better than prev model overall iirc but idk it's hard to judge we need more/better third party evals

@eliebakouch @tszzl I meant as in could the 20 questions be mapped to each model at release time (highly unlikely but I'm struggling to find an alternative interpretation for these results)

I believe @petergostev pointed out this as well.

@eliebakouch @tszzl I wonder if the test set is static w this one though

@mert_gulsun @tszzl i actually looked at this before posting

for internal debugging eval it's a new one so idk but for OPQA it seems so (from 5.4)

I think he is referring to the research advancements and the increased compute access that makes training on ML research possible.

I don't think any labs train on full end to end ML research outside of kernel tasks and SWE engineering and emergent capabilities that come from that.