At decision points, Claude Code can keep or rewrite two programs: analysis.py, which measures the run, and updater.py, which turns gradients into updates to the model. The updater runs at every training step; the agent makes decisions at intervals, not at every step.
Where the researchers saw gains
In the team’s experiments, OPAL ranked first against five adapted leaderboard recipes when modded-nanogpt was trained on Chinese, Thai or Kyrgyz datasets or with a quarter-sized batch. Those comparisons concern changed settings without recipes tuned specifically for them.
For a Goldfish language model trained on Amharic, the researchers report 81.11 versus the official recipe’s 83.48 FLORES-200 bits per sentence, where lower is better. But a learning-rate sweep found a static recipe that scored 80.89, ahead of that OPAL run. Starting OPAL from the swept recipe produced a score of 80.73. In an ImageNet generation experiment with LightningDiT-XL/1, the team reports a guided FID-50K score of 1.83 versus 2.18 for the official recipe, also a measure where lower is better.
The agent also reacted to problems in specific image experiments. When the researchers inserted four stretches of random images and labels into CIFAR-10 training, OPAL eventually added a gate for batches with unusually high loss. The first two stretches still cost it 50 and 66 percentage points of test accuracy. In a separate experiment with mislabeled CIFAR-10 training images and development-set feedback, OPAL raised learning rates and averaged weights, finishing at 78.6% test accuracy versus 73.5% for AdamW trained straight through. These are results from those tests, not a guarantee that a run will recover from bad data or overfitting.
The cost of letting the agent decide
The researchers say most experiments were run once, with a few run three times. In their timing comparison, runs that waited for the agent took 1.30 to 4.39 times as long as static runs. Letting training continue while the agent thinks is an option, but the repository says waiting is the default.
The repository also records a neutral-prompt OPAL 0.2.0 run on a heavily tuned modded-nanogpt benchmark that finished worse than its comparison recipe. Its results table distinguishes earlier-harness and pre-release runs from reruns on OPAL 0.3.x. For version 0.3.3, the README says multi-node training, FSDP and DeepSpeed are unsupported, and the agent cannot restart or roll back a training run after a bad decision; it can only change its programs at the next decision point.
