Haiku cost $95.46 compared to $9.47 for Opus.
On price/token != cost/task:
To help internalize this fact for yourself, run Terminal-Bench using Haiku and then using Opus.
Here are the results for a 15-task subset. Haiku is 10x the cost!
Price per token != cost per task
Positive users praise cost-per-task benchmarks for clarifying Haiku efficiency beyond raw token prices, while negative users note Haiku requires more tokens overall due to repeated failures and retries versus Opus.
No Digg Deeper questions have been answered for this story yet.
Cost per task is going to be a meaningful metric these next 12 months.
On price/token != cost/task:
To help internalize this fact for yourself, run Terminal-Bench using Haiku and then using Opus.
Here are the results for a 15-task subset. Haiku is 10x the cost!
@alexatallah cheaper per token. more tokens to fail. more tokens to retry. more tokens to eventually give up.
@Suhail Cost per task is the real flex.
@alexatallah Thought you checked out Dm.
@Suhail ngl seeing cost/task spelled out like that makes the efficiency conversation way more concrete
most people still thinking in raw token price
@Suhail tokens per token was fun while it lasted
@alexatallah Would be great if we can get both price/token and cost/task for each model, for each benchmark in the web