a lot of the benches beloved by model connoisseurs are things like "PostTrainBench" and "WeirdML", and we're probably due for another good kernel benchmark soon
the labs will soon have to choose between "pushing the frontier" via headline numbers and self-commoditization
No Digg Deeper questions have been answered for this story yet.
however, their biggest spenders and loudest advocates are deep-pocketed engineers "building with ai" at well-funded companies making ai products of their own, and showing off a "research edge" is the easiest way for an ai company to get investors excited...
makes you think
a lot of the benches beloved by model connoisseurs are things like "PostTrainBench" and "WeirdML", and we're probably due for another good kernel benchmark soon
the labs will soon have to choose between "pushing the frontier" via headline numbers and self-commoditization
@willccbb dont you think they have/are developing benches internally that are superior to the other internal research tasks they show in the model card?
@willccbb will called me a model connoisseurs, gonna frame this
@jayce_edits if you're openai and want to keep your moat, would you publicly release a model that does well on internal r&d benchmarks?