It matches action-labeled models using 1/100 of the data
Also, if you are curious about latent world models etc, we have taco/ premier taco 🌮 here
And @jbhuang0604 made a thread to help you understand JEPA, here
I didn't know what JEPA is, and at this point I am too afraid to ask ...😬
so I made a video tracing the last 30+ years of self-supervised learning, covering ideas from contrastive learning, distillation, masked modeling, JEPA, and world models.
https://youtu.be/gVEr2cnDE_8
This wouldn't have been possible without my amazing collaborators and advisors:
co-led by @YoonkyoJung, with amazing team @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @furongh, @jbhuang0604
Can a model trained purely on video — with zero action labels — match VLAs trained on massive action-labeled datasets?
Meet µ0 (Mew-Zero): a world model that learns a "physical language" for robots.
Here's why we're excited 🧵
The ultimate test: can traces actually help a robot act?
Yes.
Freeze µ0, train just a lightweight action expert on top — and it matches or outperforms strong VLAs on real robot arms. 🤯
And remember: µ0 never sees a single action label during pretraining.
Takeaway → we can learn a "physical language" for robots purely from video. 3D motion traces can be that shared symbolic space.
🔗 http://mu0-wm.github.io
How do we get 3D trace labels from raw video?
We built TraceExtract, a data engine that takes only video and: - compensates for camera motion - filters out stationary / background points - chunks long videos & auto-generates event-level captions
Training physical AI is hard: action-labeled robot data is scarce, expensive, embodiment-specific, and doesn't scale.
Our bet: learn from abundant video first, then ground to specific robots later — letting robots tap into web-scale data.
But what should a robot world model actually predict?
- Pixels → scalable, but too low-level - Latents → compact, but brittle and hard to control - Actions → embodiment-specific, and scarce on the web
µ0 then learns from these traces: a pretrained VLM backbone + a permutation-equivariant Trace Expert that predicts each keypoint's future as smooth B-spline curves via flow matching.
Our answer: 3D interaction traces — a unified "word space" for robotics.
Instead of dense pixels or robot-specific actions, µ0 predicts smooth 3D trajectories for the points that matter: objects, tools, hands, and contact regions.