The dataset contains 2,000 hand-written, human-verified VQA items.
Are VLMs nearing saturation on vision benchmarks?
Not on WorldBench: 2,000 carefully curated and verified questions over a visually diverse range of images, designed to be hard for frontier models.
The strongest still gets only 64%.
Led by @DavidYin0609 and @harishkrik
A joyful collaboration with @liuzhuang1234. WorldBench indicates that VQA is still far from being solved yet. Hope this new benchmark can provide new insights to the community!
This grew out of our earlier "dataset bias" work: modern training datasets still live in surprisingly narrow visual distributions.
WorldBench asks the same question on the benchmark side.
http://arxiv.org/abs/2403.08632
Are VLMs nearing saturation on vision benchmarks?
Not on WorldBench: 2,000 carefully curated and verified questions over a visually diverse range of images, designed to be hard for frontier models.
The strongest still gets only 64%.
Led by @DavidYin0609 and @harishkrik
@liuzhuang1234 Narrow training distributions are sneaky because benchmarks usually share the same blind spots, so models look fine until the real world shows up.
The dataset contains 2,000 hand-written, human-verified VQA items.
Are VLMs nearing saturation on vision benchmarks?
Not on WorldBench: 2,000 carefully curated and verified questions over a visually diverse range of images, designed to be hard for frontier models.
The strongest still gets only 64%.
Led by @DavidYin0609 and @harishkrik