A new multimodal-AI study reports that vision encoders trained from scratch alongside a language model can match pretrained ones at the same overall training-compute budget. Removing dedicated vision capacity, however, still hurt visual understanding in its comparisons. Source
The paper, “Unbundling Multimodal Vision Encoding,” was submitted to arXiv on November 12, 2026, by Ellis Brown and 11 coauthors. In a November 13 post, Brown said the team trained 200-plus models from scratch to investigate which design choices mattered. Source Source
Three decisions, not one switch
A vision encoder processes images into representations a language model can use. The authors separate three choices that a conventional encoder bundles together:
- Separate visual pretraining: whether the vision component gets an earlier training stage of its own.
- Dedicated vision capacity: whether the model reserves parameters—its learned numerical settings—specifically for visual processing.
- Placement: whether that capacity sits before the multimodal backbone—the main network that processes visual and language input together—or inside it. Source
Published models also differ in training data, architecture and compute, the authors argue. Comparing their results cannot isolate which of those choices made the difference. The study instead trains models from scratch on common data and matches lifetime compute—the total training computation, including separate encoder pretraining where used. It evaluates the models on visual question-answering and language benchmarks. Source
The researchers report that a jointly trained encoder performed on par with a pretrained one without needing a separate vision-pretraining stage. But an encoder-free model trailed on visual understanding even when its backbone grew by the encoder’s parameter count. Source
So “skip encoder pretraining” and “remove vision-specific parameters” are not interchangeable instructions. That second delete button has consequences.
Move the vision capacity, don’t just remove it
The separate encoder wasn’t the only workable home for those parameters. According to the abstract, vision-only modules placed inside early backbone layers performed comparably to an encoder positioned before the backbone. This “in-fusion” arrangement can also lower time to first token—the wait before output begins. Lower time to first token doesn’t mean every response finishes faster. Source
The authors report that their findings also held across backbone sizes, when the compute cost of encoder pretraining was excluded from the comparison, and when starting with a pretrained language model. The abstract gives no exact model sizes, benchmark scores or latency measurements—no stopwatch number to celebrate there. Source
In commentary on the study, @giffmana described its limits as dense models only, sizes in the single-digit billions of parameters, fixed image resolution, and pretraining plus instruction-tuning without reinforcement learning. Source
Comments