Gary Marcus Slams AI Model Hype Over Six-Point Benchmark Gains

Cost on the other hand:

@GaryMarcus Yeah. I thought we wouldn't fall for stunts like this anymore, like when they made us wait for GPT-5 or when they said that GPT-2 was too dangerous to release.

@GaryMarcus Mild linear gains for massive increases in token cost. Seems to be the trend in the 'super labs'.

@MoonlitMonkey69 indeed

@GaryMarcus all the drama and smoke for 6 points difference

feels like overhype theater tbh

@GaryMarcus 6 points on a benchmark is noise. 6 points on a P&L is survival. stop measuring the thermometer and start reading the blood pressure.

@GaryMarcus Honestly I thought this benchmark had died because of how unreliable mass voting was and how relatively easy it is to game for votes like in the GPT-4o era.