OpenHands co-founder Graham Neubig launches the OpenHands Index to benchmark autonomous AI agents on combined LLM-harness performance

Many coding agent benchmarks evaluate LLM performance (with the same harness) or harness performance (with the same LLM).

But in reality, it is the harness + the LLM that determins the overall performance.

We introduced new, holistic results that measure both.

The results are interesting. Overall the trends are: - OpenHands generally outperforms Claude Code on both accuracy and cost when using Claude Opus - OpenHands beats Codex on accuracy, but falls behind on cost when using GPT - Gemini-CLI outperforms OpenHands when using Gemini

We also have much more fine grained results, and the full set of agent trajectories across the 5 benchmarks here: https://index.openhands.dev/alternative-agents

And these results were enabled by our new ACP support in the OpenHands SDK, which you can use for benchmarking or many other things, learn more here:

@gneubig Looks useful!