Kosta Derpanis (sabbatical in Zurich)#1013
馃毃Introducing: Ego-MC-Bench (Mistake Corrections) benchmark and Ego-CoMist (Counterfactual Mistakes) dataset.
馃幆Ego-MC-Bench: Where AI assistants need to intervene at the right time (when) and with the right feedback (what) to prevent mistakes.
馃憠https://tinyurl.com/y7y9mwrs
1/4
鉂楋笍Ego-MC-Bench contains instruction feedback pairs provided by an expert in real-world kitchen scenarios.
鈴癸笍Current SOTA video LLMs show very poor mistake intervention capabilities.
馃憠Even Gemini-3-Flash manages to get an mistake intervention F1 score of only 0.18.
2/4
馃幆A major bottleneck is the lack of appropriate video data of procedural activities with mistakes.
This is in spite of abundance of procedural activity datasets.
Therefore, we propose a synthetic data generation process with counterfactual mistakes: Ego-CoMist.
3/4
馃憠This leads to a significant improvement in mistake intervention capabilities, especially for small models ideal for edge deployment.
馃摐Paper: https://arxiv.org/abs/2606.09547
4/4
Joint work with: @Matewhs @SanjayHaresh @RolandMemisevic
Kosta Derpanis (sabbatical in Zurich)#1013馃毃Introducing: Ego-MC-Bench (Mistake Corrections) benchmark and Ego-CoMist (Counterfactual Mistakes) dataset.
馃幆Ego-MC-Bench: Where AI assistants need to intervene at the right time (when) and with the right feedback (what) to prevent mistakes.
馃憠https://tinyurl.com/y7y9mwrs
1/4