Sankalp Shubham tested whether two AI models needed images to resolve tasks in SWE-bench Multimodal v2. Removing the images barely changed their overall resolved-task rates in his runs. On a separate test built around image-dependent questions, however, both models lost substantial accuracy without them.
What changed when the images disappeared
In Shubham’s three runs of all 480 SWE-bench tasks, GPT-6 Luna xhigh resolved 177 tasks with native images, 175 with images replaced by OCR text and 178 with no images. Those are resolved-task rates of 36.9%, 36.5% and 37.1%, respectively. OCR, or optical character recognition, extracted text from the images rather than showing the model the original pictures.
Sonnet 5.5 max resolved 21 of 60 tasks with images and 24 without. Shubham used the same randomly selected tasks for both Sonnet conditions, with one attempt per task; he did not run Sonnet with OCR. He cautions that the smaller sample cannot rule out a moderate benefit from images.
Shubham describes an interceptor that replaced images before they reached the models, including images in tool outputs. He says the runs were network-isolated and that his team removed later Git history after spotting agents retrieving upstream fixes in an initial run.
A control where pictures mattered
On MMMU-Pro, a separate set of image-dependent questions, Luna’s accuracy fell from 79.7% with images to 36.9% without across 1,730 questions. Sonnet’s fell from 80.8% to 44.2% across 120. Unlike the SWE-bench figures, these percentages measure correct answers, not resolved software tasks. Shubham uses the control to show that his setup could detect an image effect.
Why the smaller change on SWE-bench? Shubham reports that reviewers judged the issue text sufficient in 403 of Luna’s 480 tasks, with images helpful in 70 and required in seven. He stresses that these are reviewer judgments, not definitive labels. One task rated image-required was resolved only when Luna had the native image.
Shubham also argues that some grading tests miss visual requirements. His example is a Carbon TimePicker task whose pictured design specifies colors and spacing, while the grading snapshot checks for two props. He raises possible familiarity with fixes to public issues as another explanation. His narrower finding is that removing images had little effect on scores in these runs, not that images are unnecessary for every task.
Comments