Wharton's Ethan Mollick argues organizations must build custom benchmarks rather than swapping AI models solely for cost savings

Vending machine

@emollick Use Gemini 3.5 Flash to decipher ancient pharaoh curses. Use Claude Opus 4.8 to dispense your Diet Coke.

@emollick Now, what model should I use if I want to sell Hieroglyphics out of a vending machine? 😉

@emollick This is directionally good advice. Don’t guess, test. Find the most efficient and cost effective model for your use case.

I genuinely appreciate you advocating for right model right job, and not just uber model all jobs.

The practical economics of this are meaningful.

@emollick Gemini 3.5 Flash specifically being best at hieroglyph translation is genuinely funny to me

feels like u learned the hard way after a few lab costs blew up

@emollick the slippery slope when generic benchmarks replace real user testing

cost savings look great until the model fails in production

@emollick The gap between public benchmark rank and "works on my actual task" keeps widening. Half my model picks flip once I test on my own eval set instead of the leaderboard.

@emollick Having benchmarks designed to optimize model-agnostic workflows is what will work.

@emollick benchmarks are useful but the real test is how the model handles ur actual terrible workflow not synthetic edge cases

@emollick Exactly. The right model depends on the job, not the leaderboard. Same reason you don't hire the same person for every role. Workers, not tools.