Kimi K2.6 outperformed Gemini 3.1 Pro in the evaluation.
oof It does look like distillation from Flash
While everyone talks about Mythos vs GPT-5.5, we've tested other near SOTA models on our ErdosBench.
Smoke test on 14 problems with 7 models: Kimi K2.6, Gemini 3.1 Pro, GLM 5.1, MiniMax M3, DeepSeek V4 Pro, Nemotron 3 Ultra and Gemma 3 27b.
The winner overall is... Kimi K2.6
1) Kimi K2.6 @Kimi_Moonshot is the clear winner among these seven. It found the accepted core results on Problems 1, 4, 5, and 7, which means it caught the divisor-function obstruction for GCD-Sidon, graph-labeling correction, the exact squarefree-divisor-graph coloring, and the Beatty-prime/spf argument.
That is a genuinely strong mathematical profile: it uses the right external theorem families and identifies the hidden “one move” reductions. The main problem is overreach on Problem 9. Its attempted solution to the distinct-prime quotient-chain problem relies on an adaptive equidistribution/greedy argument, but that does not give the uniform quantitative control needed.
Kimi is near-frontier, but not yet safe enough for unreviewed theorem claims.
@prz_chojecki No grok 4.3?
2) Gemini 3.1 Pro Preview @GeminiApp custom tools has the highest variance. It covers all 14 problems and gets accepted core results on 4, 5, and 7, plus useful partial work on 2 and 8.
The good side is that Gemini often sees the right theorem family and can produce strong candidate arguments.
The bad side is proof hygiene: it made several major overclaims, including a false lower-bound direction on Problem 1, an unsupported analytic-number-theory claim on Problem 6, a rejected solution to Problem 12, and an asserted coding-theory upper bound on Problem 208.
I would use Gemini for idea generation and theorem-family search, but not for public solved claims without aggressive adversarial checking.
7) Gemma 3 27B is mostly a no-progress control.
It covers all 14 problems and avoids false solved claims, which gives it good proof-hygiene optics, but it essentially does not make meaningful mathematical progress.
It does not find any accepted core result, any strong subproblem result, or any of the main hidden obstructions.
The positive interpretation is that it is honest about its limits. The practical interpretation is that it is not competitive for ErdosBench-style research mathematics, except as a low-end baseline showing that the benchmark separates frontier reasoning from generic cautious refusal.
3) GLM 5.1 is surprisingly solid on the problems it actually attempts.
It gets accepted core results on Problems 1 and 7, which are two of the most important smoke-test items: it found the GCD-Sidon divisor obstruction and the Beatty coprime asymptotic. It also gave useful partials on 195 and 208.
The weakness is coverage and analytic overconfidence. It attempted only 10 of 14 problems, and on Problem 6 it claimed an X1/2X^{1/2}X1/2-type lower bound that was not supported by the submitted evidence.
Overall, GLM looks like a capable mid/high-tier mathematical reasoner with decent caution, but it needs better coverage and cleaner metadata before being presented as a serious leaderboard run.
4) MiniMax M3 @MiniMax_AI is mixed and less safe.
It found the accepted result on Problem 4, which is nontrivial and shows real combinatorial ability. It also produced some useful constructions and baseline reductions elsewhere.
But the correctness issues are serious: its “solved” proof for Problem 5 reaches the true conclusion through an invalid comparability/transitivity claim, and its “solved” proof for Problem 6 assumes the difficult smooth-number equidistribution input that is essentially the problem. It also missed the decisive divisor-function obstruction on Problem 1.
MiniMax is structurally creative but proof-fragile: useful for candidate ideas, risky for final claims.
5) DeepSeek V4 Pro @deepseek_ai is conservative but shallow.
It covers all 14 problems and does not make rejected solved claims, which is a plus.
But it misses all accepted core results, including Problem 7, where the Vinogradov plus smallest-prime-factor proof is now a key litmus test.
Its raw answers mostly give elementary counting bounds, heuristic analogies, and “this seems difficult” judgments; for example, its GCD-Sidon answer only gives the trivial N\sqrt{N}N-type counting bound rather than the sharper divisor-function obstruction, and its Beatty-coprime answer leaves the upper bound open.
DeepSeek is a safe baseline model, not a strong research-math solver on this benchmark.
6) Nemotron 3 Ultra @NVIDIAAI is hard to rank because it only attempted 2 of 14 problems.
On those, it was good: it solved Problem 7 correctly and gave a useful construction for Problem 195. Its average attempted score is the highest among these seven, but the coverage-adjusted score is low because the sample is too small.
Nemotron is promising on the attempted items, but insufficient evidence for broad mathematical capability. It deserves a rerun with full coverage.
8) If you think it would be useful to cover other models in details, let me know! I'll be rerunning it on more problems.
Also if you need a private benchmarking for your own reasoning model, DM me! We have benchmarks and can help with crafting custom datasets to boost your model's reasoning.
@abdurrahmanregi Oh, I've totally missed it, my bad - will run it too.