It converts broad commands into highly specific robotic subtasks.
Robots 🤖struggle at new long-horizon tasks. One trick to improve performance is using granular subtask labels ("grab the plate" instead of just "do the dishes").
Today we're releasing our first blog 📜: using VLMs to mine subtasks at scale (19x cheaper than human annotators!)
Users express gratitude to Macrodata Labs' CTO and team for single-handedly labeling videos to create their open-source robotics subtask annotation benchmark.
No Digg Deeper questions have been answered for this story yet.
hey robot, plz think step by step
Robots 🤖struggle at new long-horizon tasks. One trick to improve performance is using granular subtask labels ("grab the plate" instead of just "do the dishes").
Today we're releasing our first blog 📜: using VLMs to mine subtasks at scale (19x cheaper than human annotators!)
Model evaluation: - Gemini was the best multimodal model by a wide margin 🦾. - Interestingly, Gemini Robotics ER performed worse than Gemini 3.5 Flash, despite being positioned as stronger for spatial tasks. - Open-source models were not really competitive in this setup 🙁
https://macrodata.co/blog/annotating-robot-video-subtasks
We built a benchmark, ran over 60 experiments, and made our pipeline fully open-source. Details in thread. Read the full blog: https://macrodata.co/blog/annotating-robot-video-subtasks
To measure progress, we first had to build a benchmark: WGO-Bench 🏆.
It contains 100 robot and egocentric videos with 743 gold subtask segments.
Rules: - boundaries happen on world-state changes - labels cannot depend on past frames - precise locations/objects description
Input formatting mattered a lot.
Two lessons stood out: - Contact sheets performed slightly better than separate frames, while reducing cost by almost 12x - Engraving timestamps directly onto the images worked much better than describing timestamps in text
To create WGO-Bench, we hand-annotated all 100 videos.
This was harder than it sounds: - 1 minute of video can easily take 10+ minutes to annotate carefully - objects are often small, occluded, or hard to recognize - egocentric videos often contain multiple parallel actions
Finally combining the best segmentation and labeling resulted in 0.201 E2E F1.
Where are the main issues?
- short segments (< 2s) are often missed. - missing information about end-state of action (fill bottle until it’s full) - missing information about end-state of action (fill bottle [until it’s full]) - underspecified target/direction
Equipped with the benchmark, we first tested how can we best use models for segmentation. We run many experiments across different prompts, input formatting, and models, going from 0.007 F1 to 0.306. 🧗
Here is what made the biggest difference ⬇️
After segmenting the video, we had to actually label the subtasks. We gave the model context (prev and next contact sheets) and pushed accuracy from 40.4% to 60.8% 📈
The full pipeline is completely open-source. Read the full blog here: https://macrodata.co/blog/annotating-robot-video-subtasks
I’m very grateful to our CTO, @HKydlicek, who single-handedly labeled every video 🙏, and to the rest of the team for making what we do possible @tomaspduarte
