AI experimenter fofr finds Gemini Omni struggles with two-handed finger-counting in video evaluation
The model failed specifically on numbers seven and eight
With the exact same prompt from last year
fofr@fofrAIOne year later with Omni and this test can pass. I saw it getting pretty close, so I tweaked the prompt: > A video of a man counting to 10 on his fingers, show the number in the corner. A new number every 1s, no dialogue other than the numbers he says. He uses two hands for numbers bigger than 5. - the model does 1 to 5 consistently well - struggles more when two hands are used, usually on 7 and 8 - if you ask it to count faster, errors increase - it keeps a good cadence
1 to 20 also works pretty well, if we're just counting
fofr@fofrAIOne year later with Omni and this test can pass. I saw it getting pretty close, so I tweaked the prompt: > A video of a man counting to 10 on his fingers, show the number in the corner. A new number every 1s, no dialogue other than the numbers he says. He uses two hands for numbers bigger than 5. - the model does 1 to 5 consistently well - struggles more when two hands are used, usually on 7 and 8 - if you ask it to count faster, errors increase - it keeps a good cadence
@giffmana 😅
Lucas Beyer (bl16)@giffmana@fofrAI Of course you did it already, hours before my question!
Time to make the test harder
> A video of a man counting as fast as he can, show the number in the corner. No dialogue other than the numbers he says.
fofr@fofrAIOne year later with Omni and this test can pass. I saw it getting pretty close, so I tweaked the prompt: > A video of a man counting to 10 on his fingers, show the number in the corner. A new number every 1s, no dialogue other than the numbers he says. He uses two hands for numbers bigger than 5. - the model does 1 to 5 consistently well - struggles more when two hands are used, usually on 7 and 8 - if you ask it to count faster, errors increase - it keeps a good cadence