MIT's Phillip Isola and Guanya Shi have released ABC, an open robotics project built around the ABC-130K dataset of roughly 134,800 real-world bimanual episodes spanning 195 tasks. The stack bundles the data with training code, model checkpoints, a simulation pipeline, and evaluation logs so anyone can inspect or extend the full behavior-cloning workflow.
Introducing ABC: open data, training, and infrastructure for robotics.
We release the largest teleop dataset to date, and extensively investigate design decisions, pretraining, and post-training techniques.
@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall
More than 3,500 hours of synchronized joint, gripper, and multi-view video recordings are now public under an Apache 2.0 license on Hugging Face, along with 400 hours of matching simulated episodes. Researchers can immediately test scaling laws or architecture tweaks without starting from scratch.
The release highlights a co-training recipe that mixes a small slice of simulated data with the real episodes and reports strong correlation between sim and real-robot results. Exact mixing ratios and whether the same gains hold on entirely new hardware setups remain open for the community to verify.
Users praised researchers for releasing the largest teleop dataset and open robotics stack, citing the collaborative efforts and value of open-sourcing the polished work.
No Digg Deeper questions have been answered for this story yet.
Check out our paper, more videos, and our more than 3.5K hour dataset here https://abc.bot
My favorite part: ABC is a reproducible robotics stack. The website demos are not just cool videos: the underlying data, recipes, and infra are open, so people can actually inspect and build on them.
Also excited about the >400h of sim teleop data coming soon. This should help the community iterate faster and study questions like sim-real co-training. One encouraging signal from our experiments: even with only ~2% simulated data mixed into co-training, we observed strong sim-real performance correlations.
Excited to see what the community builds on top!
Introducing ABC: open data, training, and infrastructure for robotics.
We release the largest teleop dataset to date, and extensively investigate design decisions, pretraining, and post-training techniques.
@arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Let's gooo congrats y'all
One interesting observation we find is across different model classes, training flow loss and validation action error correlate with real world success, while we did not observe any statistically significant correlation between validation flow loss and real world success.
This would not have been possible without the support from Amazon FAR or our amazing mentors @akanazawa, @GuanyaShi, @pabbeel, @peterxichen, @rocky_duan, @JitendraMalikCV, @phillip_isola
Shout out to our collaborators @xdofai -- see their tweets here and also congrats on their company announcement:
In addition to real data, we have simulation data. Using this data, we show sim/real correlation.
We develop lightweight robot infra, enabling things such as DAgger and intervention data to augment our large scale dataset. This allows us to experiment with post-training our checkpoints to enable dexterous tasks such as airpod insertion
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall So great to collaborate with you all on this work!
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Am I the only one curious about Arthur's exit? Where is he going? 😂
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall So happy to be able to help contribute to this amazing work!
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congrats @ritvik_singh9 !!
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall let's goooo
@roeiherzig @ritvik_singh9 @Cinnabar233 @adamrasb @redstone_hong @davidrmcall going home after too many late nights in the lab 😳
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Amazing to see even more great work open sourced!
Well done to the team
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall congrats!!! very polished amazing work!!!
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Congratulations!
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Y’all need any data for picking up rubber ducks?
Cuz I have wayyyyyy too many episodes in my local hard drive 😅
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Fantastic work, lots of great insights
@ritvik_singh9 @arthurallshire @Cinnabar233 @adamrasb @redstone_hong @davidrmcall Very cool paper, so much useful data in there. Do you have any theories as to why adaln modulation worked better for VLAs but cross attention worked better for DiT? That was definitely not an intuitive result for me.
