Researchers Release HABIT Dataset For Human-Robot Interaction

Large-scale datasets have driven remarkable progress in general-purpose robot policies. But they're almost always collected with the robot as the sole agent — no human in the scene.

So policies can do a task in isolation, yet fall short where robots are actually deployed: homes, factories, shared workspaces. They never learn to hand over, yield to a reaching hand, or avoid a co-present person — because those behaviors can't be demonstrated without a human there.

[2/7]

📄 Paper: https://arxiv.org/abs/2606.31682 🌐 Project page: https://habit-dataset.github.io

This work was done with an amazing team at @config_inc. Huge thanks to all my collaborators at Config and KAIST. I'm especially grateful to @kimin_le2 for the guidance and support throughout this work. 🙏

[7/7]

Policies trained on HABIT act appropriately around people where a matched Robot-only baseline fails — avoiding collisions, coordinating in time, following human gestures. The same trend holds for both π0.5 and GR00T N1.6.

[5/7]

Mid-training on HABIT improves both sample efficiency and final performance on downstream interaction tasks. HABIT serves as a strong prior that transfers to new human-robot interaction tasks.

[6/7]

To capture interaction systematically, we organize the 60 tasks by role — a lens borrowed from the Human-Robot Interaction (HRI) literature — chosen to span diverse interaction patterns:

Collaborator — human and robot jointly do one task Coworker — separate tasks, shared space (must avoid collisions) Supervisor — the human directs the robot

[3/7]

We carefully designed our data-collection protocol to capture these interactions. The robot acts reactively by responding to what the human actually does, cue by cue, rather than replaying a memorized routine. And we deliberately elicit human-aware behaviors: yielding for safety, temporal adaptation (matching the human's pace), and gesture grounding via time-delayed instructions.

[4/7]

@j__aehwi 164 hours of actual human robot interaction data is a solid step toward robots that work safely around people.