Fable 5 leads the new LisanBench spatial reasoning benchmark based on Opus Magnum puzzles, beating GPT-5.5

Another point in favor of "multimodal pretraining does not generalize to spatial reasoning in text". Kimi had 15T heavily visual tokens. It lags GLM 5.2 even more than it does in pure text tasks. Geminis have excellent vision. Little to show for it. Opus is almost blind, yet…

"It’s not a vision benchmark, all done through coordinates"

from the creator @RobertHaisfield

I feel like multimodal pre-training doesn't do much without image output. it's just learning a mapping from your img encoder to the language space

I think what's more useful is vision RL

Well, maybe I'm unfair. Gemini Pro still does better than GLM, even though it's a trashy model. But as with Fable, I think it's overwhelmingly a function of scale.

@noah_vandal @0x_lun @scaling01 We believe that this document is fully AI-generated

Disclaimer: For text under 75 words, results may be less accurate.

https://www.pangram.com/history/7c3b47f3-e3af-4a72-994a-2f8a8bea47a3

@0x_lun @scaling01 @pangram slop?

@teortaxesTex this is a very good point. if it was true gemini should be the smartest because of what is sees.

counterpoint to that is that shtty frankenmerge with semantic mashing is why and real unified latent space has never been tried or trained properly.

bad blending worse than none

@scaling01 @zephyr_z9 Just how good Glm 5.2 even is It's thrashing opus on all benchmarks

@scaling01 My strategy plan.

🔻↩️↩️

@scaling01 Gemini almost certainly has data contamination on this benchmark because of YouTube, btw.

@scaling01 the GPT 5.5 solution in that image looks like it took the scenic route. Fable just went direct

@scaling01 Missed opportunity to name it Roonbench

@scaling01 GLM-5.2 punching above its weight on a fresh benchmark is the realest signal you can get no time to benchmaxx means the gains are legit

@scaling01 GLM 5.2 at 24.5 beating both flash and opus on a fresh benchmark is the kind of result that usually gets dismissed until it keeps happening

Chinese labs are quietly stacking these

@scaling01 @teortaxesTex explainer from the site:

@teortaxesTex I feel like nobody has yet figured out a good way to exploit the synergy between modalities in multimodal pretraining, which also partly explains why the model still sucks at Computer Use tasks where the pretrained model isn't good enough to be RL'd on.

@scaling01 It's okay to believe that GLM just cooked with this one and made actual good efforts in intelligence not bench maxing. All what I've seen about GLM is positive and they deserve all the praise

@scaling01 glm quietly grinding while everyone was arguing about the top of the leaderboard is genuinely funny

the gaming pipeline for benchmarks has been consistently underrated

@scaling01