Dan Luu reports that GPT-6 Astra agents seemed to use testing and verification techniques more effectively than GPT-5.6 Sol agents in his coding evaluations. His Zstd rerun, which asked agents to implement Zstd from its RFC, left little room to measure which testing instructions helped: Almost every condition reached full correctness almost every time at medium effort.
On that task, Luu reported that Astra reached full holdout-test correctness more than 99% of the time at medium effort, with $1 in model cost. In his earlier Sol evaluation, the corresponding results were about 63% at $3.50 on medium effort and 77% at $5.50 on xhigh effort. When prompted to fuzz, Astra agents generated structured inputs that exercised code paths and found nontrivial bugs, Luu wrote. But the Zstd results were too close to perfect to show whether fuzzing improved correctness.
A harder task, wider uncertainty
Luu then examined a harder Pandoc task. After multiple turns, roughly 14 on average, no condition reached 100% correctness. Instead of counting runs that passed every test, he measured the fraction of tests passed. He ran each condition 25 times, at an estimated $90,000 at MSRP.
Many techniques produced better results than the condition with no specified testing method, which Luu calls Default. He considers it plausible that many outperform Default, but says the results do not establish which conditions are generally better. Whether the apparent rankings would hold over more turns also remains untested.
