Does a Claude model really need Claude Code?

@melissapanDoes your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge: 1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost 2️⃣A simple harness can be competitive 3️⃣The native harness isn’t always the best. Millions of people are using coding agents, but the impact of harness choice remains unclear. (1/n) More details in the thread. 🧵

@melissapanWhere is the harness tax coming from? 🧐 Agents can take similar numbers of turns at substantially different costs. For Fable 5 on SWE-bench Lite, Pi and Claude Code average 15.4 and 15.3 turns per attempt, yet Claude Code costs about twice as much for a 1.1-percentage-point increase in success rate. So what is happening each turn? One possible reason can even be observed at the first model call: Claude Code’s mean initial context is over 10× Pi’s, with longer instructions and larger tool schemas ‼️ As models become more capable, agents may need less scaffolding. For everyday tasks, harness design should therefore prioritize cost efficiency and reliability. (4/n)

@matei_zahariaRT @melissapan: Does your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising fi…

@arenaDo models need their native harness for coding? Awesome work from our intern @melissapan on this. She dug into whether the harness (Claude Code vs Codex CLI vs Pi) actually moves the needle for coding agents. Turns out it matters way less than people assume. 21 model-harness pairs, real rigor. More from Arena to come. More details on the Arena blog: https://arena.ai/blog/coding-agents-harness-tax

@matei_zahariaAgent harnesses seem to make a lot of difference, at least for cost, even on open source coding benchmarks. Melissa’s research project digs into why.

@istoica05Does a model need its native harness? We evaluated seven models across the Claude Code, Codex, and Pi harnesses and found a surprising result: harness choice had little effect on task success but a substantial effect on cost. Sometimes, a simple harness is all you need!

@andykonwinski"Harness choice has little effect on task success rate ... As models become more capable, agents may need less scaffolding."

@a1zhangRT @melissapan: Does your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising fi…

@HamelHusainRT @melissapan: Where is the harness tax coming from? 🧐 Agents can take similar numbers of turns at substantially different costs. For Fa…

@ttunguzThe better the harness, the better the business. Berkeley published a study this week showing harnesses, the systems that control AI agents, set the price of an answer. The right harness cuts the cost of the same result by 71% without a loss of accuracy. https://x.com/i/article/2100651851280687104

Does a Claude model really need Claude Code?

TLDR

Combined views

Does a Claude model really need Claude Code?

10 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

10 Sources