Ask a model what's next to the oven in a kitchen it's never seen. The right answer requires hallucinating: probably a fridge, a counter, and some cabinets. That's not a failure. That's what a world model is for.
Everyone wants models with "good world models," but it's unclear what that means beyond describing the world well. A good world model isn't just perception. It predicts what's probably there when you don't have the input.
No Digg Deeper questions have been answered for this story yet.
So a spurious image is a world model doing its job at the wrong moment: Gap-filling is imagination when you're planning, and a mirage when you're supposed to report what you see.
Ask a model what's next to the oven in a kitchen it's never seen. The right answer requires hallucinating: probably a fridge, a counter, and some cabinets. That's not a failure. That's what a world model is for.
Why do models gap-fill instead of saying "I can't tell"? Because we pay them to. Benchmarks grade correctness, not groundedness, so guessing from priors has positive expected value and abstaining has zero. Mirage is the optimal policy under our evals.
Humans run on this machinery too. Perception is "controlled hallucination": the brain predicts the scene and uses input to correct errors. Your blind spot gets filled in from priors every waking second. What you have that models don't is source monitoring
Humans run on this machinery too. Perception is "controlled hallucination": the brain predicts the scene and uses input to correct errors. Your blind spot gets filled in from priors every waking second. What you have that models don't is source monitoring
So a spurious image is a world model doing its job at the wrong moment: Gap-filling is imagination when you're planning, and a mirage when you're supposed to report what you see.
None of this is specific to vision. A fabricated citation is the same move: a world model of the literature completing a gap with plausible authors and a plausible year. Ignoring retrieved context when priors are strong is another example.
Why do models gap-fill instead of saying "I can't tell"? Because we pay them to. Benchmarks grade correctness, not groundedness, so guessing from priors has positive expected value and abstaining has zero. Mirage is the optimal policy under our evals.
So the line isn't "models shouldn't imagine." We want models that complete scenes, predict, simulate. The line is that the model needs to know, and tell us, which parts came from input and which parts it filled in.
Our results are mildly optimistic here: if "I'm running on imagined content" is linearly decodable, it's flaggable in principle. Grounding without imagination is a camera. Imagination without grounding is a mirage. The interesting problem is the seam.