UMD's Furong Huang Lab just dropped μ₀, a world model that skips pixel or latent dreaming entirely and instead forecasts compact 3D motion traces of hands, objects, and contact points. The approach lets the system pretrain on plain video, then transfer across different robot bodies without action labels, delivering stronger real-world results on a UR3 arm than π₀.₅ while reportedly needing far less data.
Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
By focusing on semantic 3D interaction points instead of raw visuals, μ₀ sidesteps embodiment-specific details during pretraining, which could let labs reuse the same backbone across grippers, arms, and future hardware without retraining from scratch.
The 1/100th data advantage and exact success rates on physical tasks appear in lab posts but lack full dataset sizes or methodology on the project site, leaving open how reproducible the efficiency edge will be once the paper and any code arrive.
Users praised the μ₀ world model for predicting 3D motion traces in efficient robot learning because it delivers strong results with video-only pretraining and stands out as an impressive advance.
This is what responsible use of resources means to us.
No industrial-scale compute. No massive in-house proprietary data.
Just open-source data, an academic compute cluster, and a team with a lot of passion 🤩 Still achieved industrial level performance!
I’m excited for the next stage: scaling the data, while keeping the model efficiently small and the representation meaningful.
Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐
13/
Some real-world experiment demos here.
7/
So what are the “words” of physical interaction?
Our answer:
motion traces.
They are grounded in video, structured in 3D, interpretable, and more transferable across embodiments than robot-specific actions.
6/
Think of traces as a possible “unified words space” for robotics.
LLMs scale because text gives them a shared symbolic space.
Robotics has no obvious equivalent: different bodies, sensors, actions, tools, tasks, and environments.
1/
The internet is full of videos of humans doing things.
But robot action data is scarce, expensive, embodiment-specific, and hard to scale.
World models should help us learn from abundant video first, then ground to robots later.
Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
4/
Latents seem like the natural alternative.
But black-box latent spaces are hard to interpret, hard to intervene on, and hard to correct.
They can also collapse.
After 15+ years working with spectral / latent methods, I’ve learned this the hard way: latent spaces are brittle.
3/
Pixels force the model to spend capacity on texture, lighting, background, and camera motion.
But robotics cares about something more physical:
geometry, contact, motion, and how objects change under interaction.
3/
Pixels force the model to spend capacity on texture, lighting, background, and camera motion.
But robotics cares about something more physical:
geometry, contact, motion, and how objects change under interaction.
2/
But this only works if we choose the right prediction space.
Pixels are scalable, but too low-level.
Latents are compact, but often too brittle.
So what should a robot world model actually predict?
6/
Think of traces as a possible “unified words space” for robotics.
LLMs scale because text gives them a shared symbolic space.
Robotics has no obvious equivalent: different bodies, sensors, actions, tools, tasks, and environments.
5/
μ₀ takes a different route.
Instead of predicting pixels or black-box latents, it predicts:
3D interaction traces.
The motion of semantic physical points: objects, tools, hands, and contact regions.
#Robotics #EmbodiedAI #WorldModels #AI
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐
2/
But this only works if we choose the right prediction space.
Pixels are scalable, but too low-level.
Latents are compact, but often too brittle.
So what should a robot world model actually predict?
1/
The internet is full of videos of humans doing things.
But robot action data is scarce, expensive, embodiment-specific, and hard to scale.
World models should help us learn from abundant video first, then ground to robots later.
5/
μ₀ takes a different route.
Instead of predicting pixels or black-box latents, it predicts:
3D interaction traces.
The motion of semantic physical points: objects, tools, hands, and contact regions.
4/
Latents seem like the natural alternative.
But black-box latent spaces are hard to interpret, hard to intervene on, and hard to correct.
They can also collapse.
After 15+ years working with spectral / latent methods, I’ve learned this the hard way: latent spaces are brittle.
8/
To learn traces at scale, we built TraceExtract.
It turns ordinary human and robot videos into trace supervision:
• what moves: semantic keypoints • where it moves: shared 3D reconstruction • how it moves: event-level motion traces
7/
So what are the “words” of physical interaction?
Our answer:
motion traces.
They are grounded in video, structured in 3D, interpretable, and more transferable across embodiments than robot-specific actions.
9/
μ₀ then learns a trace-space world model.
It uses a frozen vision-language backbone for semantics, and a trace expert for physical motion.
Instead of dense future pixels, it predicts smooth 3D motion traces.
8/
To learn traces at scale, we built TraceExtract.
It turns ordinary human and robot videos into trace supervision:
• what moves: semantic keypoints • where it moves: shared 3D reconstruction • how it moves: event-level motion traces
12/
And it works.
In simulation and on a real robot arm, μ₀ achieves performance comparable to strong action-trained VLA policies,
while using roughly 1/100 of the data scale,
and no action-labeled data for world-model pretraining.
11/
The key test:
Can these traces actually help a robot act?
We freeze μ₀ and train only a lightweight action expert on top.
The world model itself never sees action labels during pretraining.
13/
Some real-world experiment demos here.
12/
And it works.
In simulation and on a real robot arm, μ₀ achieves performance comparable to strong action-trained VLA policies,
while using roughly 1/100 of the data scale,
and no action-labeled data for world-model pretraining.
@furongh i dont understand but id like to
@furongh I dont dream visually all the time. I dream about possibility or event without vivifying them. I also don't always have an inner dialog. Many of my profound investigations and discoveries come deliberately and at some time, without words.
Thank you @chrmanning for sharing our work. We’re truly humbled and grateful for your kind recognition and more motivated than ever to make the work even better.
Hot take: robots should not dream in pixels.
Pixels are too low-level. Latents are too opaque.
μ₀ predicts a third thing: 3D motion traces.
On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵
https://mu0-wm.github.io/
(this video features voiceover narration)
This is a major step forward from our previous work, TraceGen. We achieved significant improvements and put in a tremendous amount of engineering effort. Hats off to the team—especially @JayLEE_0301, @YoonkyoJung, and @jusukle ❤️
14/
The takeaway:
Robotics may not need bigger pixel predictors.
And it may not want black-box latent predictors either.
It may need a physical language.
μ₀ proposes one: 3D motion traces as the symbolic world-model space for robot learning.
https://mu0-wm.github.io/
Big thanks to the coolest team: @JayLEE_0301, @YoonkyoJung, @jusukle, Jonghun Shin, Amir-Hossein Shahidzadeh, @YaoChihLee, H. Jin Kim, @jbhuang0604 💐
11/
The key test:
Can these traces actually help a robot act?
We freeze μ₀ and train only a lightweight action expert on top.
The world model itself never sees action labels during pretraining.
10/ On three-D trace forecasting, mew-zero is the most accurate across metrics and horizons.
It outperforms prior trace methods, surpasses large API models overall, and runs in zero point two nine seconds per prediction.
