A researcher’s X thread argues that a model’s benchmark gains were heavily inflated by training on lightly rephrased versions of the test set itself.
On X, researcher Elie Bakouch argued that a model was trained for 10 epochs on a “very light rephrasing” of the entire GPQA Diamond evaluation set. In a linked post, Bakouch wrote that GPQA Diamond makes up 20% of the model’s “capability index” and accounts for about 70% of its reported performance gap, while also claiming the model shares the same architecture and about 80% of the training mixture with NVIDIA’s Nemotron 3 Nano.
Combined views
32.8K
11 posts, first seen 3h ago
