Why is the bitter lesson debate on 3D necessary if you can learn 3D world models directly from 2D videos? Our CVPR 26 paper (presented Fri. morning in Findings track 1 #48, 7AM - 8:30AM) proposes the first 3D Gaussian Splatting dynamics model that predict multi-object collision dynamics, trained only from multi-view real-world 2D videos with a single GPU.

Foundational models trained on millions of videos can fail in strange ways when modeling those collision dynamics, yet our 3D world model can learn these collision dynamics and generate plausible trajectory autoregressively for dozens of frames, that can be rendered to any viewpoint via Gaussian splatting.

We release all the code and a dataset with 500 calibrated multi-view videos on two tasks: tabletop bowling and falling cubes, so that everyone can try training 3D world models (for predicting next-state dynamics) with a single GPU!

https://chkim403.github.io/gs_physics/