Fixed Eval Boosts Open Model Over Later Closed Releases
Research engineer reports performance change after correcting benchmark errors.
Florian Brand noted that fixing an evaluation produced a large performance increase for an open model released before the benchmark. A closed model released afterward remained unchanged. Cody Blakeney audited MBPP and HumanEval variants and corrected issues with prompts, test cases, and formatting. He used an NAC tool to run concurrent sessions and plans to open-source the fixed evaluations along with further details in a blog post.
Combined views
9.9K
7 posts, first seen 4d ago
Reactions from ranked influencers
7 posts
Fixing an eval and the closed model released after the bench stayed flat while the open model released before the eval got a gigantic jump 😇
Getting SOTA By Looking At The Data And Screaming Into The Void
One read is that this is dishonest by some lab. I also wonder if this a reward-hacking adjacent outcome: give broken eval, model figures out right answer anyway. This suggests a fun option: BrokenEval
Fixing an eval and the closed model released after the bench stayed flat while the open model released before the eval got a gigantic jump 😇 https://twitter.com/xeophon/status/2089582036545589555
This made me extremely paranoid and made me wonder what else might be messed up. To investigate this problem, I turned to the tool I have been using more and more frequently for rapid data and evals work, nac. (3/5)
Remember, folks, always look at the data! (5/5)
Using Codex through the MCP server, I created 47 concurrent nac sessions, and within 2 hours, I was able to audit every individual sample of both the MBPP and HumanEval variants and correct multiple deep issues not only with the prompts but also with test cases, bad formatting,…
We'll open-source the fixed version(s) of the eval with the blog, and go into more detail about how a hosted nac enabled a curiosity to turn into a result in hours instead of days for only a couple hundred bucks.
I fixed it. Or rather, nac did It turns out bad regex wasn't the only thing wrong in multiPL-E. Full blog coming next week, but this was too good to sit on. (1/5) https://twitter.com/code_star/status/2089107403904323693