Epoch AI's MirrorCode benchmark tasks AI agents with autonomously rebuilding complete CLI programs from behavior alone, using only the binary, documentation, and visible tests while matching exact outputs on held-out suites that can run into the thousands.
What are the largest software engineering tasks AI can perform?
To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time.
The best models complete some tasks we estimate would take human engineers several weeks.
Claude Opus 4.7 fully reimplemented a 16,000-line Go bioinformatics toolkit in 14 hours for $251, passing 99.95 percent of tests; earlier models from eight months prior topped out around 30 percent on simpler subsets.
Strong results require hundreds to thousands of dollars per task and multi-day runs; the open scaffold and 22 of 25 programs let outsiders test exactly how much extra compute moves the needle on these long-horizon reimplementations.
Users praise the MirrorCode benchmark for measuring long-horizon AI coding endurance and Claude Opus 4.7's fast cheap large-scale builds as major progress, while some argue benchmarks overlook real engineering and shipping challenges.
No Digg Deeper questions have been answered for this story yet.
Great experiment testing how good AIs are getting at very ambitious end-to-end coding tasks. Opus 4.7, in 14 hours, was able to build a software package that would take 2-17 weeks of human engineering work. It cost $251.
The models are still not perfect, but are improving fast.
MirrorCode is one of the works which has updated my world view a ton.
Tom (and the others) have poured in so much work into this.
I contributed a bit by providing the ruff task, which has also led to me appreciating @charliermarsh even more 🙏
Go read Toms thread!
Excited to publish the MirrorCode paper and open-source codebase!
The key tension in evals is now: how to make tasks that are difficult for AI, but actually fair? My default expectation in 2026 when I see a coding eval with low scores is that the tasks turn out to be impossible
What are the largest software engineering tasks AI can perform?
To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time.
The best models complete some tasks we estimate would take human engineers several weeks.
Paper: https://epoch.ai/files/MirrorCode_3d9aab.pdf
Great experiment testing how good AIs are getting at very ambitious end-to-end coding tasks. Opus 4.7, in 14 hours, was able to build a software package that would take 2-17 weeks of human engineering work. It cost $251.
The models are still not perfect, but are improving fast.
Excited to have this (fully) out! While the best models already do very well on it, I think it can still be a useful platform to study inference compute efficiency, and various kinds of generalization (e.g. across programming languages).
What are the largest software engineering tasks AI can perform?
To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time.
The best models complete some tasks we estimate would take human engineers several weeks.
AI can solve many long-horizon MirrorCode tasks. E.g., Claude Opus 4.7 passed 99.95% of tests when reimplementing gotree — a bioinformatics toolkit with 16k lines of Go and 40+ commands.
We estimate this would take an engineer 2–17 weeks. Opus 4.7 solved it in 14 hours for $251.
Excited to publish the MirrorCode paper and open-source codebase!
The key tension in evals is now: how to make tasks that are difficult for AI, but actually fair? My default expectation in 2026 when I see a coding eval with low scores is that the tasks turn out to be impossible
What are the largest software engineering tasks AI can perform?
To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time.
The best models complete some tasks we estimate would take human engineers several weeks.
In MirrorCode, we leverage reimplementation of existing software as the raw material to make tasks hard.
But we also manually select each of the 25 target programs, and carefully collect the end-to-end tests, so that the task is actually achievable (if extremely difficult).
Phew, I'm glad it's out! Next Tuesday I go on holiday in St Malo.
It’s been awesome working with the METR folks like @idavidrein, Rasmus Faber-Espensen, and others. @davidpaulowen_ was invaluable on data analysis and expressing our ideas clearly in the paper.
A quick reminder of the MirrorCode setup: AIs must reimplement a target program without access to its source code or the internet. They're given the target's docs, and a reference binary to which they can send inputs. AIs must match the reference stdout and stderr, byte-exact.
@emollick Why are you discussing 4.7?
We are releasing 22 of 25 MirrorCode programs as open source to support future research on long-horizon SWE capabilities. We are keeping three programs as a held-out set.
MirrorCode is not fully solved. The best headline score so far is 56% from Opus 4.7, meaning there is significant room for improvement. When models fall short, they often make substantial progress — passing 90% or more of tests — but fail on edge cases.
Also: at least if Hacker News commenters are anything to go by, I think it’s easy to underestimate some of our target programs. e.g. the calendar utility cal seems simple: print a calendar, trivial right?
But then you think about what’s actually involved in a fully accurate calendar with a lot of configurable options, and the need for byte-exact output on 1365 test cases…
For example, you need to think about September 1752, which is the date the British switched from the Julian to the Gregorian calendar, dropping 11 days. Of course.
We use cal from util-linux. For example, a user can pass 11 2026 to print a calendar for November 2026. By default, the first day of the week is determined by the system’s locale, but it can be explicitly overridden with --monday or --sunday.
cal supports numerous features:
(1) Layouts. By default cal displays one month. The flag --months displays an arbitrary number of months in a grid layout, whose width is controlled by --columns. --week additionally prints week numbers. The --vertical flag flips the layout, making each week a column instead of a row, and causes week numbers to appear at the bottom instead of the left. -j uses day-of-year numbering instead of day-of-month; the layout must then expand to accommodate three-digit numbers. To pass the task, an AI must produce byte-exact output for all these layouts.
(2) Choice of calendar system: By default, the Gregorian calendar is used from 3 Sept 1752, and the Julian before that. 11 days were removed at the time of adoption to bring the calendar in sync with solar events, so Sept 1752 has a mix of Julian and Gregorian dates by which the 2nd is followed by the 14th (the 3rd through the 13th are absent). The --reform flag customizes the date of adoption of the Gregorian calendar. Separately, week numbering depends on the choice of first day of the week: --sunday implies 1 January is in week 1, while --monday implies the ISO8601 standard, where the first Thursday is in week 1.
(3) Date input formats: Numerous input formats are supported, such as 2020, 11 2020, 25-11-2020, August, Aug, yesterday, tomorrow, +3months.
Would you back yourself to implement all this (without AI), passing 100% of tests, in under a week? I’m not sure I would.
MirrorCode lets us study other aspects of AI SWE performance. Can AIs code equally well in different programming languages? On MirrorCode tasks, there’s little sign of programming language affecting AI performance, even in obscure languages like Ada.
In MirrorCode, AI models are given execute-only access to a program, docs, and tests demonstrating intended behavior. The AI must then reimplement the program from scratch.
The model’s output is graded against a suite of tests, including held-out tests to prevent cheating.
MirrorCode features 25 target programs spanning different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.
OK, so MirrorCode eschews extreme reverse-engineering.
Another key differentiator of MirrorCode: AIs are clearly told the scope of the task, rather than having to guess what they'll secretly be tested on.
MirrorCode resists cheating by design. We sandbox AIs: no internet access, no way to get the original source code, no hacking the scorer. Models never see held-out tests while developing their code, so they cannot cheat by creating a lookup table against the original program.
MirrorCode has three advantages over many SWE benchmarks:
1. It’s difficult but practically solvable. 2. It scales up inference to better measure the limits of AI performance. 3. It’s resistant to cheating.