Databricks says Genie with Genie Ontology passed 91% of runs in a business-data benchmark it developed with Turing, compared with 45% and 58% for two unnamed frontier agents. The company also reports that Genie used fewer than half the tokens and finished roughly three to four times faster. Source
Genie ran natively in Databricks, while the other agents connected through a Databricks toolbelt. All received the same questions, but the toolbelt excluded Genie MCP—the integration that would have given competitors access to the Genie capabilities Databricks wanted to test. This compares differently configured systems, not isolated models with identical integrations. Source
The test includes the office’s analytical attic
The benchmark uses a fictional UK electricity and gas supplier, Calderwick Energy, inside a live Databricks workspace. Alongside hundreds of tables, Databricks included notebooks, dashboards, saved SQL queries and documents, complete with abandoned analyses and outdated material. Each agent ran five times per task. A response passed only if it met the required checks in the grading rubric. Source
Databricks says Genie Ontology extracts and organizes business knowledge from those assets so Genie can find the definitions and calculation rules needed to interpret the tables. Source
In Databricks’ walkthrough, an agent asked for a historical planning rate could find 0.31 pence per kilowatt-hour in the relevant year’s table rows. But those rows used newer assumptions. A notebook preserved the historical rate the question actually requested: 0.52 pence per kilowatt-hour. According to Databricks, Genie returned 0.52 in its first answer across all five runs; neither rival returned the correct historical rate in its runs. The database had a number. The question needed its backstory. Source
Fewer tokens, but no dollar comparison
Databricks reports average token usage of 431,829 per run for Genie, versus 889,019 for Agent A and 1,051,070 for Agent B. The write-up does not provide pricing or full operating-cost accounting, so those token counts don’t establish equivalent dollar savings. Source
The company does not identify the competing models or versions, disclose the total task count, or present an experiment that isolates the ontology’s contribution while holding the rest of the setup constant. Databricks credits the business-context layer, but these results alone don’t establish how much of the gap it caused. Source
Databricks says the difference may be smaller in new deployments without accumulated workspace knowledge, or when questions can be answered directly from tables and schemas. This is a context-heavy Databricks test, not a universal AI leaderboard. Source
The company describes the benchmark assets as designed for sharing, but directs readers to their account teams for access. Its write-up does not report an independent replication. Source
Always-on Genie is a separate announcement
In its November 11, 2026 announcement, Databricks described a second-generation Genie that delegates to subagents, tracks long-running work and follows up on missing inputs. Apparently, the AI coworker is also learning to chase the missing attachment. Source
That announcement said Genie and Genie Ontology were generally available, while the new always-on Genie, Genie Search and Ontology Manager were coming to preview and beta programs. It did not give rollout dates. Databricks lists keeping a quarterly forecast on track among the always-on agent’s potential responsibilities. The benchmark’s answer scores don’t show whether it can manage that job. SourceSource
Comments