The model integrates directly into Model Predictive Control for dynamic replanning
New research: AdaJEPA perceives, plans, and adapts in a closed loop. It is always learning!
Warm take: Your world model should never stop learning
Introducing AdaJEPA, an adaptive WM that plans, acts, and adapts in a closed loop. Every action leads to a new observation, and every transition refines the latent representation and prediction.
📝: https://agenticlearning.ai/adajepa/
Users express gratitude for AdaJEPA's adaptive world model handling visual corruption, color changes, dynamics shifts, and unseen layouts through continuous learning.
No Digg Deeper questions have been answered for this story yet.
Warm take: Your world model should never stop learning
Introducing AdaJEPA, an adaptive WM that plans, acts, and adapts in a closed loop. Every action leads to a new observation, and every transition refines the latent representation and prediction.
📝: https://agenticlearning.ai/adajepa/
The idea of adaptation is rooted in control and neuroscience: e.g. self-adapting IDCOM updated process models online for MPC (Foigel&Richalet,1979), humans adapt their internal dynamics during motor learning (Shadmehr&Mussa-Ivaldi,1994). How do we realize adaptation in latent WM?
Scaling training data helps. But adaptation can make limited data go much further. In low-data PushObj (K=1, N=1k), AdaJEPA beats a frozen model trained with 16× trajectories per shape and 4x shapes.
We train a decoder for the frozen WM to visualize planning (blue frozen, red AdaJEPA). OOD data is often imagined as the closest training-domain structure. After adaptation, rollouts still decode reasonably, suggesting AdaJEPA refines while staying near learned latent manifold.
[The AdaJEPA idea is simple] Given a pretrained JEPA WM, at every MPC step, execute the first action, collect the obs, update WM to minimize latent pred error, then replan with updated model. This forms a plan-execute-adapt-replan loop that continually improves prediction.
With as few as one gradient step per replan, AdaJEPA leads to consistently better planning for both in-distribution and ood test environments. We train on four-shape PushObj and test with additional held-out shapes (denoted by star), and AdaJEPA shows significant gains.
More: AdaJEPA for visual corruption and color changes, dynamics shifts, unseen maze layouts, and ablations on adaptation hyperparameters ➡️https://arxiv.org/pdf/2606.32026
Huge thanks to my collaborator @oumaymabounou and advisors Prof.@ylecun and Prof.@mengyer 💜
