PaintBench benchmark launches to test precise spatial image editing, with the top-performing model scoring just 17.1% accuracy

@NanoBanana 📄 Paper: https://arxiv.org/abs/2606.00188 💻 Code: https://github.com/PaintBench/PaintBench 🤗 Dataset: https://huggingface.co/datasets/PaintBench/PaintBench 🌐 Website: https://paintbench.github.io/

Grateful to collaborators @_ellisbrown, @shrikarhaha, @rob_fergus, @hhexiy, @sainingxie

@itskaixu @sainingxie @NanoBanana Don’t get it. At that point why not use coding agents, image layers and programmatic control.

@NanoBanana We hope PaintBench inspires more researchers to demand rigorous utility over superficial plausibility by designing deterministic evals for applied multimodal tasks, with a focus on precise scientific tasks that advance human livelihoods. 💪

@NanoBanana PaintBench tests 20 fundamental, ubiquitous operations in geometric transformation, structural manipulation, color change, and symbolic reasoning.

🌱Seed-generation with configurable difficulty enables ∞ problem variety and prevents benchmark contamination.

@NanoBanana Can PaintBench scores generalize to applied tasks? To find out, we create 📊TinyGrafixBench, a data visualization editing eval that adopts the same seed-gen, pixel-eval philosophy.

Finding: strong correlation between scores across the two benchmarks 📈

@NanoBanana 👾Deterministic pixel-level evaluation produces granular per-problem metrics without bias-prone judge models.

PaintBench reveals many hard tasks across all models, plus individual model strengths and weaknesses.

@NanoBanana Some fun PaintBench findings 🤠

1. Drawing or moving shapes by precise amounts is especially hard.

2. Low robustness: high shape count and textured backgrounds severely hurt performance.

3. Models tend to over-edit. By a lot.

@rarply @sainingxie @NanoBanana Fair q - recent multimodal development is trending toward generalist models designed to natively handle increasingly precise tasks. So our experiments isolate the internal representational strength of the underlying model. But we do encourage other applications of the benchmark!

@rarply @itskaixu @sainingxie @NanoBanana

@itskaixu @jihanyang13 @NanoBanana Nice! I like how deterministic this evaluation is compared to other image editing benchmarks

@_ellisbrown @itskaixu @sainingxie @NanoBanana Nah.

“Native” is just means a multimodal model. NB2 and gptimage2 output cot to enhance steering.

The architecture is begging for more fine grained steering.

@itskaixu @NanoBanana Not at all surprised at 17%. Great to have a metric. In general and for hill climbing.