Databricks CTO Matei Zaharia says open-source GLM 5.2 on the minimalist Pi harness tied Claude Opus 4.8 at 35% lower cost
The evaluation suggests vendor harnesses inflate token consumption and operational costs.
We benchmarked coding agents on our own internal tasks at Databricks and learned a lot! There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now. 🧵
Combined views
955.6K
24 posts, first seen 46d ago
Reactions from ranked influencers
22 postsThese are the results on OUR sample of OUR codebase, so they are not meant to be comprehensive, but we think many companies can do a similar internal benchmark. Several findings immediately stood out: 1) Many models are now competitive at the top tier, including open source.
3) Harnesses make a huge difference in cost-performance. The very simple Pi harness (@badlogicgames) got the same success rate as harnesses from the LLM vendors with Opus and GPT 5.5, but at 2x less cost! Seems to be mainly due to smaller inputs to the LLM.
Read more in the blog how we built the benchmark and what we're doing with the findings. This is partly why we built @omnigent_ai as a "meta-harness" to let developers switch and compose agents, and Unity AI Gateway to analyze and gate LLM usage centrally.…
makes me think what if AI companies' harnesses make you spend more tokens even at the detriment of the performance because no one checks out underrated harnesses
Surprising and such a good news for open source coding model, and also that there are lots of hidden chances to reduce cost while improving quality. Databricks showed GLM-5.2 can compete with elite closed coding models inside real enterprise code. GLM 5.2 landed in…
agent architectures have clear parallels to monolithic operating systems vs micro-kernels. and we can get even more minimal agent "kernels" than pi https://twitter.com/matei_zaharia/status/2074943612631273730