Export controls work.

no, that era has *ended*, nobody is seriously benchmaxxing now. Originally this term meant training on actual test, at best on paraphrases, and we came up with continuously updated benchmarks like LCB. Around early 2025 (≈R1), models stopped crashing on post-cutoff slices.

His arguments always make me roll my eyes. Chinese firms were not within 1 year of the frontier a couple years ago. Now they’re behind by perhaps 4-6 months, at much better compute efficiency.

You can embellish a gap between any two things by jacking with the scale. None of the US models last fall are competing on these benchmarks, they were still useful models, which China has now passed. These tests zoom in on a tiny range of the problem set and ignore the large corpus where they’re much closer or saturated

@apestein_dev There's no real difference between benchmaxxing and legitimate training if you don't train on test or its paraphrases. Chinese RL environments and metrics are derived from problems the success on which was measured by old benchmarks. US frontier is doing newer harder problems.

@teortaxesTex And none of this addresses the fact Claude is great at building a skyscraper on the wrong street, there’s some benefit to needing to check in every couple hours

@teortaxesTex This has to be the dumbest comparison ever. What results did these long running tasks produce? Here's the actual comparison between the models on long running tasks and their results. Spoiler: Both are garbage. https://youtu.be/DZ9sTRyAMmM?si=bhLgMAB1Hj3HhbPs

@badboy999654 They justify the cost in that people are willing to pay that cost. The rest doesn't matter. They're more robust and smarter. If they couldn't afford these margins they'd slash them.

@badboy999654 Cope

@apestein_dev Not really but you wouldn't get it

@teortaxesTex Do you think people can't see through all the marketing bullshit? Microsoft has access to all the coding models from both companies, and yet Windows 11 is still a buggy mess. Every time they boot up Windows, people see that these models can't code for shit.

@teortaxesTex Benchmaxxing also implies trading off usability for benchmark numbers. For example by generating exceptionally long chains of yap

@teortaxesTex @badboy999654 I don’t really know what a token is 🤷‍♂️

@badboy999654 The length of a task is not about token count in a single context, but about human labor besides GPT has extremely good compaction now this is all futile thrashing

@badboy999654 Yeah yeah all "actual hard coding tasks" are on the level of shipping a new codec enough of your cope, you're arguing against revealed preferences, it's embarrassing

@teortaxesTex Ok. Show me what Opus 4.8 and GPT 5.5 can actually do that justifies the cost. They’re still shit at scientific reasoning, which is my actual use case.

@badboy999654 you're going insane the people are cattle and deserve Windows

@teortaxesTex Microsoft is out here serving both GPT-5.5 and Opus 4.8 via GitHub Copilot, yet they can't even fix the most popular OS in the world. Isn't that crazy? People only use Windows because all the engineering software literally only runs on it.

@badboy999654 no, that makes perfect sense. They're not going to rewrite Windows with AI, they'll keep making it worse. You're deep in denial.

@teortaxesTex Also, try coding something actually useful that companies are actually willing to pay for, like a new video codec. They still can't handle actual hard coding tasks because of that 1M context wall. Plus, it would literally be cheaper to hire a real dev.

@teortaxesTex I'm just asking what results you can actually achieve with long-running tasks when, in practice, you're still limited by that 1M context window (500k in reality). It's not just me asking what these long-running tasks are actually producing.