Notion details Elo-based testing behind its upgraded Auto model router

Notion has detailed how it used Elo rankings to create, evaluate and improve its upgraded Auto model router. The company’s reported quality and cost results come from thousands of simulated knowledge-work tasks scored by a calibrated LLM judge.

Notion says it selected its 20 largest task clusters and randomly sampled hundreds of real-world tasks from each. Its Gymmy sandboxes ran Notion’s application code, databases and search services with saved workspace data, letting different models attempt the same task from the same starting state. The tasks were replayed in those simulated environments, not compared through live customer work.

From head-to-head comparisons to a single judge

An ensemble of frontier models compared pairs of outputs against a custom rubric. The majority choice won each comparison, and Notion used the accumulated preferences to assign Elo scores, which rank models by their head-to-head results. These were model judgments, not human ratings.

Notion says repeating enough pairwise comparisons for every new benchmark would have been too slow and costly. Instead, it used the Elo rankings to calibrate one LLM judge to score outputs individually, allowing thousands of responses to be evaluated in parallel. , according to Notion. That figure measures agreement between evaluation methods, not the share of correct answers.