Lisan al Gaib#1185
Opus 4.8 underperforms Opus 4.7 on the LLM Debate Benchmark (1717 → 1697), but Claude is still dominating the leaderboard.
Qwen 3.7 Max scores worse than Qwen 3.6 Max: 1540 → 1499. Step 3.7 Flash lands at 1457. Ernie 5.1 improves a lot over Ernie 5.0: 1311 → 1447.
This benchmark measures how well LLMs perform in multi-turn debates across a wide range of topics. Each matchup runs twice on the same topic with sides swapped. A 3-model judge panel decides winner and margin.
The heatmap shows how models perform against each other:
More info, including transcripts: https://github.com/lechmazur/debate
Decisive Cross-Judge Agreement:
Scale is relative within this pool and centered near 1500. This is not an absolute capability score. Bradley-Terry is Elo-like.
Price vs. Performance:
Topics:
@LechMazur Interesting outcome... 🤔