Video World Model Trained on 15 Hours of Robot Video
Researchers trained the model on single-arm robot data, achieving cross-embodiment generalization via masked modeling techniques.
A video world model was trained using only 15 hours of footage from a single-arm robot. The model achieves zero-shot generalization to unseen embodiments. Related work on masked world modeling uses mask images as a unified interface to specify actions and tasks across different robot embodiments. The findings were shared and discussed by AI researchers including Jon Barron of Google DeepMind and Yilun Du of Harvard.
Combined views
1.5K
2 posts, first seen 3h ago
Reactions from ranked influencers
2 posts
We trained a video world model on just 15 hours of video of a single-arm robot. 🧵 It generalizes zero-shot to unseen embodiments (and even orangutans). And it picked up something we never trained for: give it the object motion you want, and it synthesizes the robot motion that produces it. The trick is representing actions as masked video.
How can we build a unified world model that can operate across modalities and tasks? Excited to share our work on masked world modeling, where we use mask images as a unified interface to specify actions / tasks across embodiments.
We trained a video world model on just 15 hours of video of a single-arm robot. 🧵 It generalizes zero-shot to unseen embodiments (and even orangutans). And it picked up something we never trained for: give it the object motion you want, and it synthesizes the robot motion that produces it. The trick is representing actions as masked video.