Databricks Internal Benchmark Finds Open Source Models Competitive for Coding Agents
Databricks says tests on its own engineering tasks found several coding agents clustered near the top, with open-source options in the mix.
464195820243.4K
We benchmarked coding agents on our own internal tasks at Databricks and learned a lot! There are many surprising opportunities to lower cost and increase quality, and many models including open source ones are truly competitive now.
Matei ZahariaTECH#441via XStory Brief
Databricks engineering cofounder Matei Zaharia wrote on X that the company benchmarked AI coding agents on internal tasks from its own codebase and found that "many models including open source ones are truly competitive now." In a Databricks blog post, he said the company built the test because public benchmarks such as SWE-Bench are often over-tuned for, and used real engineering work plus curated test suites to measure which agents could solve tasks end to end.
These are the results on OUR sample of OUR codebase, so they are not meant to be comprehensive, but we think many companies can do a similar internal benchmark.
Matei Zaharia
