Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
Users praise the μ₀ World Model for arguing that robotics benefits from physical language predictors rather than bigger pixel or black-box latent models.
1/
The internet is full of videos of humans doing things.
But robot action data is scarce, expensive, embodiment-specific, and hard to scale.
World models should help us learn from abundant video first, then ground to robots later.
Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
6/
Think of traces as a possible “unified words space” for robotics.
LLMs scale because text gives them a shared symbolic space.
Robotics has no obvious equivalent: different bodies, sensors, actions, tools, tasks, and environments.
5/
μ₀ takes a different route.
Instead of predicting pixels or black-box latents, it predicts:
3D interaction traces.
The motion of semantic physical points: objects, tools, hands, and contact regions.
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐
13/
Some real-world experiment demos here.
2/
But this only works if we choose the right prediction space.
Pixels are scalable, but too low-level.
Latents are compact, but often too brittle.
So what should a robot world model actually predict?
1/
The internet is full of videos of humans doing things.
But robot action data is scarce, expensive, embodiment-specific, and hard to scale.
World models should help us learn from abundant video first, then ground to robots later.
4/
Latents seem like the natural alternative.
But black-box latent spaces are hard to interpret, hard to intervene on, and hard to correct.
They can also collapse.
After 15+ years working with spectral / latent methods, I’ve learned this the hard way: latent spaces are brittle.
3/
Pixels force the model to spend capacity on texture, lighting, background, and camera motion.
But robotics cares about something more physical:
geometry, contact, motion, and how objects change under interaction.
3/
Pixels force the model to spend capacity on texture, lighting, background, and camera motion.
But robotics cares about something more physical:
geometry, contact, motion, and how objects change under interaction.
2/
But this only works if we choose the right prediction space.
Pixels are scalable, but too low-level.
Latents are compact, but often too brittle.
So what should a robot world model actually predict?
5/
μ₀ takes a different route.
Instead of predicting pixels or black-box latents, it predicts:
3D interaction traces.
The motion of semantic physical points: objects, tools, hands, and contact regions.
4/
Latents seem like the natural alternative.
But black-box latent spaces are hard to interpret, hard to intervene on, and hard to correct.
They can also collapse.
After 15+ years working with spectral / latent methods, I’ve learned this the hard way: latent spaces are brittle.
13/
Some real-world experiment demos here.
12/
And it works.
In simulation and on a real robot arm, μ₀ achieves performance comparable to strong action-trained VLA policies,
while using roughly 1/100 of the data scale,
and no action-labeled data for world-model pretraining.
7/
So what are the “words” of physical interaction?
Our answer:
motion traces.
They are grounded in video, structured in 3D, interpretable, and more transferable across embodiments than robot-specific actions.
6/
Think of traces as a possible “unified words space” for robotics.
LLMs scale because text gives them a shared symbolic space.
Robotics has no obvious equivalent: different bodies, sensors, actions, tools, tasks, and environments.
9/
μ₀ then learns a trace-space world model.
It uses a frozen vision-language backbone for semantics, and a trace expert for physical motion.
Instead of dense future pixels, it predicts smooth 3D motion traces.
8/
To learn traces at scale, we built TraceExtract.
It turns ordinary human and robot videos into trace supervision:
• what moves: semantic keypoints • where it moves: shared 3D reconstruction • how it moves: event-level motion traces
8/
To learn traces at scale, we built TraceExtract.
It turns ordinary human and robot videos into trace supervision:
• what moves: semantic keypoints • where it moves: shared 3D reconstruction • how it moves: event-level motion traces
7/
So what are the “words” of physical interaction?
Our answer:
motion traces.
They are grounded in video, structured in 3D, interpretable, and more transferable across embodiments than robot-specific actions.
12/
And it works.
In simulation and on a real robot arm, μ₀ achieves performance comparable to strong action-trained VLA policies,
while using roughly 1/100 of the data scale,
and no action-labeled data for world-model pretraining.
11/
The key test:
Can these traces actually help a robot act?
We freeze μ₀ and train only a lightweight action expert on top.
The world model itself never sees action labels during pretraining.
11/
The key test:
Can these traces actually help a robot act?
We freeze μ₀ and train only a lightweight action expert on top.
The world model itself never sees action labels during pretraining.
10/ On three-D trace forecasting, mew-zero is the most accurate across metrics and horizons.
It outperforms prior trace methods, surpasses large API models overall, and runs in zero point two nine seconds per prediction.
#Robotics #EmbodiedAI #WorldModels #AI
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐
This is a major step forward from our previous work, TraceGen. We achieved significant improvements and put in a tremendous amount of engineering effort. Hats off to the team—especially @JayLEE_0301, @YoonkyoJung, and @jusukle ❤️
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐