We're releasing MolmoMotion, a 3D motion forecasting model.
Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 馃У
Users are excited about AllenAI's release of the MolmoMotion 3D motion forecasting model because it represents an impressive advance for robotics and video generation.
No Digg Deeper questions have been answered for this story yet.
MolmoMotion can predict different motions across scenes, like a bowl sliding and rotating on a table, a flamingo dipping its beak as it walks, & a lint roller working back and forth on cloth.
Each predicted path follows the instruction and stays close to the ground-truth motion.
Everything is open鈥攄ownload the MolmoMotion weights, inspect the training data, & customize for your applications.
鉁忥笍 Blog: https://allenai.org/blog/molmo-motion 馃 Models: https://huggingface.co/collections/allenai/molmomotion 馃搳 Data: https://huggingface.co/datasets/allenai/molmo-motion-1m 馃搫 Paper: https://allenai.org/papers/molmomotion
Thrilled to introduce MolmoMotion馃殌where we found 3D motion prediction is becoming very doable with existing tools. And this capability can be very useful for downstream tasks like Robotics馃摲and VideoGen馃摲 As always, MolmoMotion is fully open! Great work led by @jianingnz @Michael3014018 @cassieyangyj
We're releasing MolmoMotion, a 3D motion forecasting model.
Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 馃У
@allen_ai The @huggingface link is broken. Could you please share the correct link?
We also built PointMotionBench, a human-validated benchmark for object-centric 3D motion forecasting.
On it, MolmoMotion outperforms every motion prediction method we tested, including pixel-space video generators, parametric 3D methods, & a constant-velocity baseline.
MolmoMotion represents motion as 3D points attached to an object, tracked in a shared world frame.
This approach doesn't need templates for objects, stays stable as camera perspectives change, & is compact enough to feed straight into downstream applications.
Training MolmoMotion required data that didn't exist: web-scale video with 3D point tracks grounded to objects & paired with actions.
So we built a pipeline to extract from ordinary video, now released as MolmoMotion-1M: 1.16M videos, 736 motion types, & 5.6K objects.
We think motion forecasting is as fundamental to machine intelligence as perceiving what's stationary.
MolmoMotion is a step toward it: 3D motion prediction that works across object types, learned from everyday video.
Motion forecasters like MolmoMotion have many potential applications.
Fine-tuned, MolmoMotion can predict object paths to help grasping robots plan where to move objects, or help image generators capture motion more accurately鈥攑articularly motions hard to describe in a prompt.
MolmoMotion is not just another cool work built on top of Molmo2鈥檚 video grounding capabilities. It shows grounding in 3D can enable motion forecasting (for both data and capabilities), which serves as an objective and representation for controllable video generation and robotics!
We're releasing MolmoMotion, a 3D motion forecasting model.
Given one or a few video frames, 3D points on an object, & an instruction like "Put the white bowl on the table," MolmoMotion predicts where those points will go over the next few seconds in a shared 3D world frame. 馃У
@allen_ai Wow insane
@allen_ai @huggingface 3D motion forecasting from sparse video frames is the piece that unlocks a lot of downstream robotics tasks. Curious how it handles occlusion when the tracked point disappears behind something mid-sequence.
@RohanChaubey4 @huggingface Sorry! Should be live now.
@allen_ai shared world frame helps but contact events still break most motion forecasts
