It uses objective reference comparisons instead of LLM judges
Good new hard benchmark to monitor, and one where the US frontier is solidly ahead (yes even Gemini, though I guess its STEM/deep multimodal priors make up for sheer stupidity. hopefully we'll see GLM-5.3V) @scaling01 add to the roster
Surface Evolver Bench
This actually looks like a pretty cool benchmark!
Users are excited about the new Surface Evolver Bench micro-benchmark because it delivers a genuinely hard test of LLMs on physics, 3D simulations, and custom physical code that most benchmarks overlook.
No Digg Deeper questions have been answered for this story yet.
@Deep_Burner Thanks! When was DeepSeek tested, and from which provider, if you don't mind?
@aiorla @scaling01 It could be expanded, the idea is more valuable it still differentiates weaker models
Surface Evolver Bench
This actually looks like a pretty cool benchmark!
My new LLM micro-benchmark l: Surface Evolver Bench: https://yhenon.github.io/surface-evolver-llm-eval/
@teortaxesTex @scaling01 isn't it almost saturated already?
My new LLM micro-benchmark l: Surface Evolver Bench: https://yhenon.github.io/surface-evolver-llm-eval/
Thank you to @htihle whilo inspired me to do this with his WeirdML bench!
@teortaxesTex @scaling01 Finally been waiting for something like this. CAD should be hill climbable maybe a bit harder than software but still the same thing just a language with funny syntax.
@teortaxesTex deepseek-v4-pro was run June 20th through the deepseek provider on openrouter. I removed the deepseek flash results, which were *not* through the official provider.
@htihle writing physical simulations in a custom data format is a genuinely hard test
most benchmarks never leave tokenizable territory so this hits a different muscle
@teortaxesTex @scaling01 Added glm-5.2 (openrouter, through z-ai provider). Might have to wait on 5.3 unless you have news for us!
@aiorla @teortaxesTex @scaling01 I'm planning on adding harder tasks, I was mostly working on the harness and tools.
@teortaxesTex @scaling01 基准测试还得看硬实力美国确实稳
