Turns out you can train humanoid hands without any robot data.
The idea in HUG is quite simple: (a) collect human data with smart glasses, (b) train a human manipulation model, (c) retarget to multi-fingered robot hands.
Users praise the HUG project for training humanoid robot hands using only human data from smart glasses, calling the work cool and congratulating the researchers.
No Digg Deeper questions have been answered for this story yet.
To get human data, we use the Meta Aria Gen-2 glasses. All you need to collect data for HUG is this untethered glasses. Aria processes these videos to give full human hand pose in the scenes.
We will be releasing roughly 1M frames we use in training as the 1M-HUGs dataset.
Turns out you can train humanoid hands without any robot data.
The idea in HUG is quite simple: (a) collect human data with smart glasses, (b) train a human manipulation model, (c) retarget to multi-fingered robot hands.
Something we are pretty excited about next is to go for more long-horizon behavior in the wild. Here is HUG running in realtime for object pick-place.
We ran HUG on several AirBnB scenes, without any tuning. It just works out of the box. ~60-70% success rate.
To model these grasps, we use both RGB and Depth as input and predict the MANO hand pose of the robot.
We needed multiple iterations to nail down the predictions in high-DOF hand space, and hope this architecture is helpful to others trying to predict full humanoid hand pose.
To probe a bit deeper into humanoid manipulation, we built a simulation benchmark as well with real object counter parts.
This should hopefully help the humanoid hand community benchmark results for apples to apples comparisons.
In inference time, you can retarget a humanoid hand to match the predicted human hand.
We have tested out the WUJI and Inspire hands and it works quite well for those. Our industry partners have reported successes on their hardware as well.
This project is @kevin_y_wu 's brainchild, with support from @Will_Zhou2025 , Isaac Tu, @BillyYYan , @irmakkguzey , David Fouhey, @DandanShan_ .
More videos, full papers, and code is here: https://grasping.io/
@LerrelPinto Cool work as always!! Congrats!!