*NEW PAPER&MODEL* HOI-DETR for detecting hands, object in-hand, &object-interacted-with (through a tool). This is the foundation model you've been waiting for! One that works off-the-shelf, single-image, using strong detectors, generalises and is stable https://ahmaddarkhalil.github.io/HOI-DETR/ 🧵
No Digg Deeper questions have been answered for this story yet.
#1499We changed the behaviour of Co-DETR to perform the task of hand-object interaction detection, fine-tuning on high-quality annotations w/ interaction head. The model surpassed all expectations - zero-shot performance on web, ego &wild benchmarks. Try demo: https://tinyurl.com/HOI-DETR
*NEW PAPER&MODEL* HOI-DETR for detecting hands, object in-hand, &object-interacted-with (through a tool). This is the foundation model you've been waiting for! One that works off-the-shelf, single-image, using strong detectors, generalises and is stable https://ahmaddarkhalil.github.io/HOI-DETR/ 🧵
Watch our zero-shot in-the-wild results (single-frame no post-processing). Model is only fine-tuned on 250K manually verified diverse images of interactions. Stable performance [p.s. we overlay our results on the YouTube videos w/out re-uploading these] https://ahmaddarkhalil.github.io/HOI-DETR/index.html#wild
To understand our visuals... Object not in hand -> background object-in-hand -> 1st object hand-in-hand -> hand-body connection object-touching-object-in-hand -> 2nd object (check how the ball fires when in contact with the racket!) Per-frame results, no post-processing!
Our model strongly outperform others... even on the recently released @meta_aria #Gen2 pilot data benchmark... Check how we compare to public annotations... [processed per frame, with a single image, no additional sensors]
Watch our zero-shot in-the-wild results (single-frame no post-processing). Model is only fine-tuned on 250K manually verified diverse images of interactions. Stable performance [p.s. we overlay our results on the YouTube videos w/out re-uploading these] https://ahmaddarkhalil.github.io/HOI-DETR/index.html#wild
To understand our visuals... Object not in hand -> background object-in-hand -> 1st object hand-in-hand -> hand-body connection object-touching-object-in-hand -> 2nd object (check how the ball fires when in contact with the racket!) Per-frame results, no post-processing!
We changed the behaviour of Co-DETR to perform the task of hand-object interaction detection, fine-tuning on high-quality annotations w/ interaction head. The model surpassed all expectations - zero-shot performance on web, ego &wild benchmarks. Try demo: https://tinyurl.com/HOI-DETR
@meta_aria Paper, code, model and demo are all public today Start here: https://ahmaddarkhalil.github.io/HOI-DETR Or: https://arxiv.org/abs/2606.17384 Or try our @huggingface demo: https://tinyurl.com/HOI-DETR Paper led by @ahmad_darkhalil @BristolUni collab w David Fouhey @nyuniversity End...
Our model strongly outperform others... even on the recently released @meta_aria #Gen2 pilot data benchmark... Check how we compare to public annotations... [processed per frame, with a single image, no additional sensors]
