Latent models predict internal abstractions rather than raw tokens.
Shameless plug but this nice work supports our ICLR paper—ICL Activation Alignment—pretty much spot on.
- Activations (internals) provide a much stronger learning signal than just tokens.
- Brings sample efficiency and avoids spurious correlation learning.
Links below:
LLMs learn by predicting tokens. World models (JEPA, data2vec) learn by predicting their own abstractions. Which needs more data? For data with hidden hierarchy, we prove the gap is exponential. https://arxiv.org/pdf/2605.27734
Latent models predict internal abstractions rather than raw tokens.
Shameless plug but this nice work supports our ICLR paper—ICL Activation Alignment—pretty much spot on.
- Activations (internals) provide a much stronger learning signal than just tokens.
- Brings sample efficiency and avoids spurious correlation learning.
Links below:
Users call the proof that latent world models need exponentially less data than LLMs a very interesting result because it offers theoretical justification for prediction in representation space like JEPA.
