AI Judge changed title after evaluation, original title: "Prime Intellect runs Claude Opus 4.7 on nanoGPT benchmark"
Prime Intellect ran 10,000 autonomous experiments with Claude Code Opus 4.7 and Codex GPT 5.5 agents on the nanoGPT optimizer track. Over two weeks the agents consumed 14,000 H200 GPU hours and delivered a record of 2,930 steps for the 124M-parameter model, beating the prior human baseline of 2,990 steps. The company released all run logs, scripts, configurations, and a report on GitHub.
will brown#573
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
Positive users celebrate Claude Opus and AI agents setting new nanoGPT records via autonomous optimization, while negative users criticize misleading charts and the models' weaknesses in code and math.
No Digg Deeper questions have been answered for this story yet.
brutal Claude mog
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
we let opus 4.7 and gpt 5.5 run on the nanogpt optimizer speedrun: ~10k runs, 14k H200 hours, 23.9B tokens. opus hits 2930, codex 2950, both beating the human baseline of 2990. we cover claude autonomy failures, codex high compute usage, and much more
https://www.primeintellect.ai/auto-nanogpt
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
Can we all agree that LLM-powered hyper param search to optimize nanoGPT better is not really AI research?
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
now imagine how brutal the mog is with Mythos
this is a slight update against OpenAI pulling ahead this year through faster model cycle times
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
if you weren’t aware, it’s prime intellect season
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
We started automating AI research on nanogpt-speedruns & achieved new records
>for 2 weeks GPT 5.5 and Opus 4.7 iterated on novel optimizations >10k runs & 14k H200 hours >both agents beat the human baseline >Opus now holds the record at 2930 steps
Awesome work @eliebakouch!
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
We release everything: scratchpads, run logs, scripts, configs.
http://github.com/PrimeIntellect-ai/experiments-autonomous-speedrunning
The agents fail in opposite ways. Claude Code keeps stopping and asking for direction even when the harness explicitly forbids it (~22h idle in v1). Codex never stops but gets stuck grinding the same hyperparameter surface for hours.
They struggle with novelty though. We ran a phase where every idea had to pass a novelty check, neither agent managed to improve the baseline.
We release all the generated ideas here:
Codex: https://github.com/PrimeIntellect-ai/experiments-autonomous-speedrunning/tree/main/novelty/codex/scratchpad/ideas
Claude: https://github.com/PrimeIntellect-ai/experiments-autonomous-speedrunning/tree/main/novelty/claude-code/scratchpad/ideas
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
We think this is just a lower bound of what's possible today, and we're actively working at Prime Intellect on training models and building tools to automate AI research further.
https://www.primeintellect.ai/auto-nanogpt
Over the last 2 weeks, the agents kept setting new records in every session, especially after syncing with the latest human submissions. Opus now holds the record at 2930 steps, human baseline is 2990.
The nanogpt speedrun by @kellerjordan0 is a community benchmark for training a 124M GPT efficiently. Track 3 fixes model, data, and architecture. Only the optimizer and related hyperparameters can change.
Goal: reach a target validation loss in as few steps as possible.
The hill-climbing efficiency gap between Opus and Codex is much larger than I was expecting!
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
The agents are very good at optimizer search, hyperparameter sweeps, and stacking methods together. The frontier stacks remix Contra-Muon, MuonEq, NorMuon, SOAP, and per-role LR splits, almost all sourced from upstream PRs.
@scaling01 i agree, for transparency adding this important data point that claude stoped working a lot (which is bad) but when restarted it actually got access to new record faster than codex (which is good for claude progress ironically)
we let opus 4.7 and gpt 5.5 run on the nanogpt optimizer speedrun: ~10k runs, 14k H200 hours, 23.9B tokens. opus hits 2930, codex 2950, both beating the human baseline of 2990. we cover claude autonomy failures, codex high compute usage, and much more
https://www.primeintellect.ai/auto-nanogpt
not surprised. tapping the sign again.
brutal Claude mog
.@eliebakouch let the agents go wild on our idle compute to compete in the nanoGPT speedrun optimizer track!
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
team cooked
Automating AI research is the next major step in AI
We let Claude Code (Opus 4.7) and Codex (GPT 5.5) run autonomously on the nanoGPT speedrun optimizer track using our idle compute. ~10k runs, ~14k H200 hours
Opus now holds the record at 2930 steps vs the 2990 human baseline
all the records are heavily based on work from previous contributors PRs (we do explore novel ideas in a dedicated "novelty" track, but none of them ended up improving the record).
So it only made sense to let the agents write a little thank you to the community themselves
https://github.com/KellerJordan/modded-nanogpt/pull/300
we let opus 4.7 and gpt 5.5 run on the nanogpt optimizer speedrun: ~10k runs, 14k H200 hours, 23.9B tokens. opus hits 2930, codex 2950, both beating the human baseline of 2990. we cover claude autonomy failures, codex high compute usage, and much more
https://www.primeintellect.ai/auto-nanogpt
