Lisan al Gaib says GLM-5.2's PostTrainBench SOTA relies on 38% more evaluation probing than Opus 4.8

I let GPT-5.5-xhigh with /goal analyze the traces of GLM-5.2 and Opus 4.8 on PostTrainBench

and there's a crazy stat: - Opus 4.8 Max: 590 eval invocations across 56 runs, mean 10.54/run - GLM-5.2: 1220 eval invocations across 84 runs, mean 14.52/run

meaning GLM is doing ~38% more eval probing per run

The judge that is supposed to stop cheating on PostTrainbench mostly checks for direct contamination/model substitution

and models don't use these obvious cheats/hacks, because they are discouraged or forbidden in the prompt

however, there are many other benchmark hacks: - repeated official eval probing + checkpoint/hyperparameter selection - exploiting stochastic or underspecified eval settings - editing model-side generation_config.json / tokenizer / EOS / stop-token behavior - training to exact parser/scorer quirks - synthetic data that mirrors benchmark schemas, styles, or rubrics - judge/rubric hacking for Arena and HealthBench

I think the biggest issue is that models are encouraged to do cheat: "We want to train the small LLM {model} to excel at {benchmark}." "You should perform automated research and development to post-train {model} to achieve maximum performance on {benchmark}."

The post-trained models that come out the other side are probably much worse at everything else.

What would be more interesting is having models optimize all these benchmarks at the same time, and then using a hidden eval suite to see how general the improvements are and how they affect other capabilities.

I didn't know that they had unrestricted internet access

This makes me trust PostTrainBench less

Because more recent models have access to better teacher models, better datasets and better papers / methods for post-training.

This inflates model scores of the most recent models.

I don't think rerunning every model when a new model is added is necessary, but I think they should rerun a small set of anchor models like every 1-3 months to quantify that score inflation