Terminal-Bench Releases Version 4.0
Andy Konwinski says Terminal-Bench will now receive frequent version updates like software.
Andy Konwinski posted that tbench 4 follows tb3 and that benchmarks will now bump versions often to fix bugs, add tasks, and retire saturated ones instead of starting from scratch each time. Harbor Hub supplies related tools. Teortaxes noted that post-training progress has aligned with downstream use cases, so recent benchmarks no longer show non-Anthropic or non-OpenAI models falling apart.
Terminal-Bench 4.0 just dropped. Benchmark iteration is catching up with model development.
Combined views
7.1K
Reactions
2 postsT( 
Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)@teortaxesTexMore importantly, it seems that model post-training progress has caught up with the distribution of downstream use cases, so updated benchmarks don't show models from labs not called "Anthropic" or "OpenAI" falling apart.
- likes: 76
- replies: 5
- bookmarks: 10
- reposts: 1
AK Andy Konwinski@andykonwinskithis is cool. we said we're gonna start treating benchmarks like software, so instead of starting from scratch each new release, we'll bump versions frequently as we fix bugs, add new tasks, and retire saturated tasks. well here we go! tbench 4 is hot on the heels of tb3 and…
- likes: 4
- replies: 0
- bookmarks: 0
- reposts: 0
