AI research agents can spend a whole day working on a problem without trying a second broad approach. In Bespoke Labs’ analysis of 348 AutoResearchExam runs across 12 models, 47% stayed with their first method for the entire 24 hours. Plan A had impressive staying power. Source
Switching to a previously untried approach was followed by roughly nine times the average validation-score gain of refining the current approach. Bespoke says that advantage was concentrated in the first six hours. Source
Different code doesn’t always mean a different idea
AutoResearchExam evaluates models on long-running machine-learning research and engineering tasks. To examine their approaches, Bespoke grouped submissions into “idea basins”: families of solutions to the same task that share a broad method. A switch means moving between those groups, not simply editing the code. Source
For example, an agent accelerating transformer computations on a CPU might use the NumPy numerical library or build a custom compiler. Those are different approaches. Repeatedly tuning the NumPy implementation can still leave the agent in the same basin. Staying with plan A doesn’t mean sitting idle. Source
Across models, runs averaged roughly 1.5 to two basins visited. Half of all switches returned to an approach the run had already tried. Bespoke also reports that 58% of runs never visited the basin containing their task’s best observed solution. That means the best solution found in this dataset, not a proven global optimum—or a guarantee that another agent would succeed by visiting the same basin. Source
Bigger early gains, not a universal switching rule
The ninefold figure compares average changes in validation score between successive submissions with different code. It does not mean agents that explored achieved nine times the final performance, or that those gains necessarily carried over to hidden tests. Source
In a separate comparison grouping switches of all kinds together, switching and refinement produced roughly equal average validation gains after hour six. Bespoke found no clear relationship across models between switching frequency and the benchmark score, which averages hidden-test performance over the 24-hour run. This observational analysis doesn’t establish that a recurring alarm telling agents to “try something else” would improve results. Source
Sonnet summarized each submission’s code, then Bespoke’s Nimble model compared those summaries without knowing which model produced the submissions. The researchers used those comparisons to construct maps and cluster submissions into basins. These are model-assisted classifications of code methods, not a brain scan of an agent’s thinking. Source
Nimble is Bespoke’s own product, and the post promotes its API alongside the findings. This is the company’s account of its analysis, not an independent replication. Source
Bespoke proposes designing models and the software systems that run them to explore new approaches early and sustain useful exploration later. Source
Comments