In time, people will come to understand "why humanoid"

🪜 What if humanoids could climb ladders and work on them straight out of simulation?

Meet LadderMan: a perceptive system for zero-shot sim-to-real ladder climbing and on-ladder manipulation.

Watch the humanoid climb, stabilize, and manipulate—all in one system. 🤖👇

300 people a year die by falling off ladders every year in the United States alone, Grok reports.

Humanoid🪜

"why humanoid" gets another case study? Expansion of capability envelope of humanoids and importantly in a very data and compute efficient manner.

the recipe that worked for locomotion in increasingly working on loco-manipulation and perceptive locomotion.

- data to generate reference ("what") - RL in sim to create experts ("how") - Distillation to add visual observations - Deployment on real env.

Check out our new work LadderMan -- a perceptive system that achieves humanoid loco-manipulation on LADDERS! Please see @SihengZhao's post for more details.

Humanoid robot climbing a ladder and manipulating.

4/ LadderMan also supports keyboard-based climbing direction control:

2/ 🌐 Project Website: http://ladderman-robot.github.io

Ladder climbing pushes humanoids to their limits: • Sparse and narrow footholds and handholds • Sustained whole-body coordination • Tiny perception or control errors can lead to catastrophic failures

3/ To tackle these challenges, LadderMan integrates a complete perception-to-control pipeline: 🔹 Two-stage climbing policy learning. First learn multiple state-based expert policies from a single reference motion via hybrid motion tracking, and then distill them into a unified visuomotor policy via hybrid DAgger + RL. 🔹 During real-world deployment, we leverage a vision foundation model for robust depth estimation, enabling zero-shot sim-to-real perception transfer. 🔹 On-ladder manipulation. We train a teleoperation policy via dual-agent learning, achieving stronger performance than off-the-shelf whole-body teleoperation baselines.

yoo i support a six month pause on humanoid robot capabilities specifically now

7/ 🙌 Acknowledgement: This was an exciting project that I worked on during my internship at Amazon FAR, with amazing collaboration from @Yuanhang__Zhang, @520xyxyzq, and insightful advice from @pabbeel, @rocky_duan, @KoushilSreenath, @yuewang314, C. Karen Liu, and @GuanyaShi.

5/ LadderMan is evaluated on three diverse real-world ladders using a single unified policy.

In simulation, we evaluate the learned climbing policy across ladders with varying rung spacing z and inclination angle φ. We compare LadderMan against a blind motion tracking baseline trained without perception.

@Scobleizer right becase it HAS to be bipedal...lol

@ericjang11 humanoid is more general, but recent work on humanoid loco manip seems do more loco than manip. how do we get the humanoid do diverse difficult manipulation tasks in a whole body manner ?

@Scobleizer Thanks for highlighting the issue of ladder falls, Robert. We see it a lot here, and more education is helpful about ladder safety. Balance and stability is key.✨

@SihengZhao cool!

@SihengZhao amazing works!

@SihengZhao Very cool! Congrats!