Wenhao Chai announced FrontierCS 2 on Oct. 22. He said the team built 48 tasks from published papers across 13 areas of computer science, refining each with experts in its field. The researchers’ paper lists domains including databases, security and robotics.
What an agent has to do
For each task, the team removes a paper’s key contribution from the code its authors released. An agent works in the remaining repository and must improve on the deleted implementation, not merely reproduce it. Qiuyang Mang said agents cannot access the paper or the web. They can submit patches for development-suite feedback, but the paper says the evaluator, final test workloads and reference remain hidden until the final submission is graded.
The pass rule is stricter than a good overall score. According to the project site and GitHub README, the final implementation must beat the original on every metric of every hidden workload. Mang gives a concrete example: on a task called KV Compaction, Claude Opus 5.5 scored above the reference overall but lost accuracy on two of 24 articles, so it failed.
In the researchers’ reported evaluation, 12 agents were tested. Claude Opus 5.5 had the highest pass rate, at 58% of its runs, while no run passed on 16 tasks. Mang describes the test as one run per task for each model, 576 runs in total. The paper says a private set of more than 100 tasks is to be announced later.
Comments