Databricks co-founder Matei Zaharia says choosing the right agent harness can halve LLM-based coding cost
The company developed Omnigent AI and Unity AI Gateway.
The company developed Omnigent AI and Unity AI Gateway.
@alighodsi @jcastillotan Thanks for sharing
At 11k employees, our AI costs are going up. Which model & harness should we use to lower cost but also retain great quality? We didn't want to blindly trust public benchmarks. So we ran a comprehensive evaluation on our tasks, code base, infra. It's been produced by more than 3,000 software engineers, spans 3 hyperscalar clouds and many languages and tasks. The results are surprising. We find that for the SAME mdoel, the choice of harness can significantly save costs (~2x). We also find that GLM 5.2 performs extremely well. We run Omnigent in front of these and can easily multiplex different harnesses and models for different tasks. Check it out: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase
Read more in the blog how we built the benchmark and what we're doing with the findings. This is partly why we built @omnigent_ai as a "meta-harness" to let developers switch and compose agents, and Unity AI Gateway to analyze and gate LLM usage centrally. https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase
Many users are excited about Databricks benchmarks showing AI coding agent harnesses like Pi can halve costs and deliver strong performance, while one noted ongoing task duplication and inefficiency in current tooling.
Based on 5 visible X reactions from 13 accounts; directional sample.
Ask a question below.
Published answers will appear here.