Reaction
Image-free runs barely change SWE-bench Multimodal v2 scores in a two-model test
SWE-bench Multimodal v2's issue text and code usually contain enough information to solve tasks, the researcher says.
TLDR
In the researcher's tests, GPT-6 Luna (xhigh) resolved 36.9% of 480 tasks with images versus 37.1% without. Sonnet 5.5 (max) resolved 35% with images versus 40% without on a 60-task subset. The researcher says issue text and code often contain what's needed; both models lost substantial accuracy without images on MMMU-Pro. A reply on X called for blind filtering of multimodal benchmarks.
Combined views
—
2 Sources, first seen ago
