Announcement
Prime Intellect open-sources fixes for public AI benchmarks
Prime Intellect says revisions to AutomationBench’s tool descriptions, tasks and graders drastically changed model scores.
TLDR
Prime Intellect replaced prose API descriptions with JSON Schemas and revised ambiguous tasks and graders in AutomationBench. Its reported scores reflect fixes applied only to the public task set, which scored higher than the private held-out set. The audit also found that OfficeQA Pro v2’s grader rejected some valid answers and that models with web access could retrieve solutions online.
Combined views
—
8 Sources, first seen ago
