SWE-bench Multimodal Focuses on Vision-Enabled Front-End Debugging
Researchers note its added challenges over earlier benchmark versions and ongoing model gains.
Ofir Press, a postdoc at Princeton, pointed out that the Multimodal version of SWE-bench stresses front-end development tasks that need vision capabilities, unlike prior variants such as Verified and Multilingua, rendering it the hardest yet. John Yang, a Stanford PhD student, observed that resolving GitHub issues communicated through visuals stays only partially solved even in 2026. He added that Anthropic models have shown steady gains on the benchmark since Opus 4.7, with performance reaching 90+% expected soon.
Combined views
23.8K
6 posts, first seen 19h ago
Reactions
6 postsJY 
John Yang@jyangballinReleasing SWE-bench Multimodal v2.0 today 480 tasks where a coding agent must interpret visual assets like screenshots, diagrams, recordings to diagnose and fix a bug in a repository.
- likes: 208
- replies: 15
- bookmarks: 48
- reposts: 30
JY John Yang@jyangballinWhen @closji and I released v1 in 2024, the benchmark was entirely close sourced. This resulted in less adoption. Reverting that decision, the v2 release is a cleaned up, filtered, open sourced iteration on the original. We removed broken tasks and flaky tests.
- likes: 5
- replies: 1
- bookmarks: 0
- reposts: 0
JY John Yang@jyangballinIn 2026, debugging GitHub issues communicated w/ visuals (not text) remains partially unsolved. Since Opus 4.7, Anthropic models have steadily climbed in Multimodal performance. Getting to 90+% with this benchmark is just a matter of time.
- likes: 3
- replies: 1
- bookmarks: 2
- reposts: 0
JY John Yang@jyangballinSWE-bench Multimodal is available via the SWE-bench repository today. New Multimodal numbers soon! https://swebench.com leaderboard is open to submissions. From a great team across Stanford, Meta MSL. cc @closji @OfirPress @klieret @Diyi_Yang @sidawxyz Weiyue Wang
- likes: 6
- replies: 1
- bookmarks: 1
- reposts: 0
OP Ofir Press@OfirPressWe made SWE-bench Multimodal more than a year ago, but the instances were never public- now they are! 480 new tasks, JavaScript/HTML/CSS. Anthropic has used it in Mythos and Fable, and now you can too.
- likes: 26
- replies: 4
- bookmarks: 4
- reposts: 3
OP Ofir Press@OfirPressUnlike previous variants of SWE-bench including Verified and Multilingua, in Multimodal there's an emphasis on front-end development, requiring vision capabilities in most of the tasks, making this variant the most challenging one.
- likes: 0
- replies: 0
- bookmarks: 0
- reposts: 0