Unsloth revises DeepSeek GGUF benchmarks with longer contexts and more runs

In a post explaining the retest, danielhanchen said Unsloth had found its earlier 32K context limit too short. The new runs used 512K context for DeepSWE, Terminal Bench, Automation Bench, HLE and SciCode, and 128K for the other benchmarks. danielhanchen put the volume at 35.1 billion benchmarking tokens, up from 1.3 billion in the earlier tests, or 27 times as much. That figure measures tokens used in testing, not a gain in model speed or benchmark scores.

danielhanchen also said the team improved its graders, used Math-Verify for math answers and used official graders for Terminal Bench and DeepSWE. The Terminal Bench 2.1 results have a qualification: Unsloth excluded seven of 89 tasks because they could not pass on its infrastructure.

The retest also prompted a dispute over a different measure involving GSQ-RCO, a rival DeepSeek quantization. danielhanchen wrote that GSQ-RCO IQ1_M claimed 98.7% recovery, while Unsloth measured 97.24% with a reported range of 96.23% to 98.21%. The account invited more details about how the other benchmarks were run. Both the account of the rival figure and the lower measurement come from danielhanchen's post; it does not establish matching test conditions for the two results.