UMD computer science professor Furong Huang launches HumanEgo to train robot policies from 30 minutes of egocentric video

Original post

1/ 🧠Humans are the best robot data source! 2/ 👓Human egocentric video is rich in quantity, but poor in quality. 3/ Beyond scaling data, smarter representation and architecture matter just as much. 4/ Want an open-source framework to train your own learn-from-human-data robot policy?

🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️

✦ Zero-Shot Human-to-Robot Transfer ✦ Robot-Data-Free ✦ Just 30 min of data per task ✦ Collect by Anyone, Anytime, Anywhere ✦ Deploy on Any Robot, Any Camera, Any Environment ✦ Open-Source & Easy-to-Implement

Let's squeeze every bit of signal out of human data!

🌐 Website: http://humanego-ai.github.io 📄 Paper: http://arxiv.org/pdf/2605.24934 💻 Code: http://github.com/TX-Leo/HumanEgo 📹 Video: http://youtu.be/pdL46diijuY 🧵 1/n

10:09 AM · May 26, 2026 · 71.1K Views

humanego ai.github.io

HUMANEGO-AI.GITHUB.IOVia

GITHUB.COMVia

YOUTU.BEVia

NVIDIA.COMVia

VIEWS10.2KBOOKMARKS12LIKES45RETWEETS8REPLIES2

Furong Huang@furongh

Excited to share our HumanEgo 🧝‍♂️ project, led by Leo 🦁!

HumanEgo is a glimpse of the closed-loop ecosystem robotics needs:

human data at scale → transferable robot policies → real-world deployment → feedback-driven improvement.

This connects naturally with EgoScale at NVIDIA GEAR, a project I’m part of, led by my former student now at NVIDIA GEAR. EgoScale scales the same idea: using large-scale egocentric human video to unlock dexterous robot intelligence.

Together, they point to a broader thesis I deeply believe in:

Humans are the scalable data source. Robots are the deployment engine. Feedback is the path to continually improving robotics foundation models.

More of my vision here 👉

EgoScale 👉https://research.nvidia.com/labs/gear/egoscale/

Zhi (Leo) Wang@TX_Leo_Wang

🚀We introduce HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos⬇️

Let's squeeze every bit of signal out of human data!

🌐 Website: http://humanego-ai.github.io 📄 Paper: http://arxiv.org/pdf/2605.24934 💻 Code: http://github.com/TX-Leo/HumanEgo 📹 Video: http://youtu.be/pdL46diijuY 🧵 1/n

21d10.2K4512

Zhi (Leo) Wang@TX_Leo_Wang

「One Policy, Many Conditions」

1️⃣ Cross-Embodiment 2️⃣Cross-Environment 3️⃣Cross-Setup

HumanEgo maintains robust success across different conditions without retraining.

🧵 2/n

21d72323

Zhi (Leo) Wang@TX_Leo_Wang

「🙏Thank you! 」

Huge shoutout to my amazing collaborators @BotaoUMD, @ColinYu14116982 , @JayLEE_0301

Deepest thanks to my advisors @RuohanGao1 , @furongh , @YAloimonos for the guidance and support throughout.

And to everyone in @prgumd and Furong's lab at @umdcs — thanks for the help.

🌐 Website: http://humanego-ai.github.io 📄 Paper: http://arxiv.org/abs/2605.24934 💻 Code: http://github.com/TX-Leo/HumanEgo 📹 Video: http://youtu.be/pdL46diijuY

🧵 10/n

21d32872

Zhi (Leo) Wang@TX_Leo_Wang

「🔥HOT TAKE - 2/3」

🔥Beyond action labels, human videos contain way more information. Richer supervision signal provides complementary gains.

🔥So we add three auxiliary objectives — all sharing the policy's encoder: 1️⃣Object Motion - forecast each object's 6-DoF future) 2️⃣2D Trace - forecast hand/object image-plane paths 3️⃣Latent Consistency - forecast the encoder's own state K steps ahead

Together they turn the encoder into a lightweight world model of hand–object interaction — letting us squeeze every demo for every bit of signal, with NO extra labels, NO extra data, just more questions asked of the same trajectory.

🧵 4/n

21d36821

Zhi (Leo) Wang@TX_Leo_Wang

「👓Data Collection & Preprocessing with Aria Glasses @meta_aria 」

✦ Collecting — Anyone, anytime, anywhere — wear Aria glasses, do the task naturally. No calibration, no special workspace, no lighting setup. 30 min per task.

✦ Preprocessing — Aria's MPS pipeline gives us 6-DoF SLAM + 3D hand pose + synchronized egocentric RGB, all from one lightweight wearable. On top, we use off-the-shelf foundation models for lightweight, accurate object tracking — completing the hand–object pair ICT needs.

Why Aria — nothing else in the category checks both boxes today:

1️⃣ Lightweight, production-grade hardware — wearable naturally for long sessions, in any environment

2️⃣ Turnkey MPS pipeline — calibrated SLAM, hand pose, multi-stream RGB, no per-session calibration

3️⃣Accuracy isn't a luxury — our experiments show upstream pose accuracy directly bounds downstream policy performance. Aria's clean, drift-free trajectories are exactly what let HumanEgo converge on minutes of data.

🧵 6/n

21d3004

Zhi (Leo) Wang@TX_Leo_Wang

「🔥HOT TAKE - 1/3」

🔥Explicit spatial representation, NOT visual fidelity, is the key to bridging the embodiment gap.

🔥Neither hand nor object alone defines a skill—what matters is their interaction.

So we design Interaction-Centric Tokens (ICT), a compact entity-level representation of hand–object inter-action invariant to embodiment, viewpoint, and environment.

🤔Most human manipulation is NOT really vision-driven — try putting a flower into a vase with your eyes closed. We do it on spatial memory, proprioception, touch, and hand–object geometry. ICT is built around exactly those signals — which we believe is why HumanEgo generalizes so widely.

🧵 3/n

21d4533

Niloofar ✈️ icml@niloofar_mire

@furongh OMG everything you do is so cool 🤩😇😍

Furong Huang@furongh

Excited to share our HumanEgo 🧝‍♂️ project, led by Leo 🦁!

HumanEgo is a glimpse of the closed-loop ecosystem robotics needs:

human data at scale → transferable robot policies → real-world deployment → feedback-driven improvement.

Together, they point to a broader thesis I deeply believe in:

Humans are the scalable data source. Robots are the deployment engine. Feedback is the path to continually improving robotics foundation models.

More of my vision here 👉

EgoScale 👉https://research.nvidia.com/labs/gear/egoscale/

21d61120

Zhi (Leo) Wang@TX_Leo_Wang

「🔥HOT TAKE - 3/3」

🔥Human video is NOT merely a cheap substitute but a superior and more efficient data source for policy learning.

Two reasons human data wins:

1️⃣ Higher offline data quality — humans naturally produce smoother, faster, more dexterous motion across a much larger workspace, switching grip strategies in milliseconds with no control loop. Teleop bottlenecks on the operator's piloting skill — everyday teleop is choppy, slow, unrepresentative.

2️⃣ Higher data efficiency — with the right representation (ICT), one human dataset transfers across embodiments, cameras, and environments. Teleop data is born inside one robot body — switch arm, gripper, or camera mount, re-collect everything.

🔥Smoother, broader, transferable — that's not a substitute, that's an upgrade.

🧵 5/n

21d3242

Zhi (Leo) Wang@TX_Leo_Wang

「🤯More Tasks」

🔥Real talk — the first time the robot nailed these tasks from just 30 min of casual human-egocentric data (no fine-tuning, no robot data, no per-task tweaking), I was honestly a little stunned. It just worked.

🔥And the model is light enough that we routinely train 10 jobs in parallel on a single RTX 4090. One commodity GPU. Ten experiments at once. That's how hard the system squeezes signal out of limited data.

🧵 9/n

21d2952

Zhi (Leo) Wang@TX_Leo_Wang

「🗺️System Overview」

🔥Arm inpainting and keypoints rendering bridge the visual gap.

🔥Interaction-Centric Tokens (ICT) encode spatial relationships among all entities

🔥A flow matching policy with dense auxiliary objectives learns bimanual robot actions from minutes-scale human data.

🧵 7/n

21d2752

Zhi (Leo) Wang@TX_Leo_Wang

「🤖HumanEgo Bridges the Embodiment Gap Efficiently」

Real-world success rate (%) for each method across all four tasks with 30 min of data. 「HumanEgo / ACT / SPOT / ZeroMimic / Tract2Act / PointPolicy / EgoZero」Three results:

1️⃣HumanEgo achieves the highest success rate on every single task. 2️⃣Even with half the data, HumanEgo outperforms robot teleoperation. 3️⃣HumanEgo excels on tasks that demand precise coordination and spatial reasoning.

🧵 8/n

21d2332

Shenyuan Gao@ShenyuanGao

@TX_Leo_Wang Congrats for the great work!

21d612

Chris Liu@drxcliu

@TX_Leo_Wang This is awesome, congrats

21d582

Jie Wang@JieWang_ZJUI

@TX_Leo_Wang That’s awesome, congrats Leo!

21d1461

Ashok@ashokM93

@TX_Leo_Wang This is soo cool!

21d1201

AUTONOMOUS - July 16th@autonomousevent

@TX_Leo_Wang A lot of these "hot takes" are soon to be the accepted industry ideas!

Chef Robotics probably knows the skill defiance better than anyone, they built an LLM to build out the data behind the interaction

Great research

21d911

Skyler Park@SkylerEcoEV

@TX_Leo_Wang Honestly, been digging into this. The real bottleneck isn't collecting more video. It's building architectures that can learn from noisy human data. More data without smarter architecture just adds noise.

21d641

Bobir M@BobirMardonov

@TX_Leo_Wang Seems impressive

21d471

Zhi (Leo) Wang@TX_Leo_Wang

@SkylerEcoEV That's ture!

21d54

Vladimir Karishev@urban_geek123

@furongh So cool! Huge respect!)

21d652