Engineers at Rhoda AI trained video models on general web clips at increasing scales of model size and compute. They then evaluated the resulting policies on a single real-world industrial manipulation job. The team observed that better prediction of held-out web video correlated with stronger post-training robot performance. Tongzhou Mu led the work, including all training and evaluations. The findings appear in a new blog post from the company.
Does scaling pre-training on general web video improve a complex manipulation task in real deployment? We scale model size and pre-training compute, and test on one industrial task. Yes. The better a pre-trained model predicts web video, the better its post-trained policy. 🧵
119.3K
16 posts, first seen 12h ago
Study tests if larger models trained on web videos improve factory manipulation performance.
Engineers at Rhoda AI trained video models on general web clips at increasing scales of model size and compute. They then evaluated the resulting policies on a single real-world industrial manipulation job. The team observed that better prediction of held-out web video correlated with stronger post-training robot performance. Tongzhou Mu led the work, including all training and evaluations. The findings appear in a new blog post from the company.
Does scaling pre-training on general web video improve a complex manipulation task in real deployment? We scale model size and pre-training compute, and test on one industrial task. Yes. The better a pre-trained model predicts web video, the better its post-trained policy. 🧵
Tongzhou Mu 🤖🦾🦿@tongzhou_mu1/ Setup. The video model is pre-trained on general web video, which contains no actions, then post-trained on robot demos of one industrial task, bearing unpacking. Each policy gets 100 to a few hundred robot trials, over 200 robot hours in total, every trial set up to a…
Tongzhou Mu 🤖🦾🦿@tongzhou_mu3/ Pre-training compute at a fixed model size. More compute helps at every amount of robot demos we test, and most when demos are scarce: the three budgets are within 8 points on the full demo set and 31 points apart on a quarter of it (p = 0.07).
Tongzhou Mu 🤖🦾🦿@tongzhou_mu5/ General web video, little of it about robots, makes a robot better at a real industrial job. The better a model predicts held-out web video, the better its robot policy does. That's the bet behind DVA. First time we've measured it in public. Full post:
ali@aliteracytongzhou led this entire work from start to finish (including training every model, finding bugs, and supervising every evaluation trial himself personally) - an absolute 🐐!! what excited me most about these results is that we can now use DINO FD as a predictor for…
Jagdeep Singh@startupjag@tongzhou_mu Scaling the AI model is one thing. Understanding whether it actually translates to better real-world robot performance is another. Fantastic work from @Tongzhou Mu and @RhodaAI putting it to the test.
Rhoda AI@RhodaAIToday marks week one of our new blog series. First one's from @tongzhou_mu . Will scaling web-video pre-training lead to better robot policies?
Many accounts praised the robotics research for confirming that scaling web video pre-training improves real-world robot performance on complex industrial tasks.
Based on 34 sentiment-bearing replies from 34 accounts across 4 conversations.
Tongzhou Mu 🤖🦾🦿@tongzhou_mu1/ Setup. The video model is pre-trained on general web video, which contains no actions, then post-trained on robot demos of one industrial task, bearing unpacking. Each policy gets 100 to a few hundred robot trials, over 200 robot hours in total, every trial set up to a…
Tongzhou Mu 🤖🦾🦿@tongzhou_mu2/ Model size. Same pre-training and post-training recipes, only the model size varies, each model trained until its validation loss flattens (the setup of Kaplan et al.). Task performance rises from 4% to 85% and is still rising at the largest size.
Tongzhou Mu 🤖🦾🦿@tongzhou_mu3/ Pre-training compute at a fixed model size. More compute helps at every amount of robot demos we test, and most when demos are scarce: the three budgets are within 8 points on the full demo set and 31 points apart on a quarter of it (p = 0.07).
Tongzhou Mu 🤖🦾🦿@tongzhou_mu4/ Why this matters. Tuning a pre-training recipe needs a metric measurable during pre-training. Robot task performance, the one we care about, only exists after post-training. DINO FD, a video prediction score, ranks all 7 checkpoints as the robot does. Size and compute are two…
Tongzhou Mu 🤖🦾🦿@tongzhou_mu5/ General web video, little of it about robots, makes a robot better at a real industrial job. The better a model predicts held-out web video, the better its robot policy does. That's the bet behind DVA. First time we've measured it in public. Full post:
ali@aliteracytongzhou led this entire work from start to finish (including training every model, finding bugs, and supervising every evaluation trial himself personally) - an absolute 🐐!! what excited me most about these results is that we can now use DINO FD as a predictor for…
Many accounts praised the robotics research for confirming that scaling web video pre-training improves real-world robot performance on complex industrial tasks.
Based on 34 sentiment-bearing replies from 34 accounts across 4 conversations.
#3
#3
