Commentary on X
@emollick I agree that many people just look at numbers and sensationalist headlines instead of trying out the models themselves. This is why things like your procedurally-generated harbor town test are so important. They are also a benchmark, just not a static one that can be saturated.
@emollick Spot on. Saturated benchmarks are becoming useless for measuring actual reasoning. We really need a new standard for 'hard problems' that can't be easily gamed by training data. What’s your personal go-to 'hard problem' test right now?
@emollick this is basically my whole relationship with the frontier: I run a couple versions behind on purpose, and it's still indistinguishable from magic to anyone who isn't tracking benchmarks for a living.