Nathan Lambert says human intuition is critical for model evaluation.
i wonder how much of "model improvement perception" (and model hype) is just human psychology being reward hacked
for instance i kinda miss fable (i used it for like 1 day) and i find some opus outputs dumb, and i genuinely have no idea if it's a real difference or just me being reward hacked
Positive users affirm Fable as a significant upgrade over Opus with authentic performance gains from hands-on testing, while negative users describe Opus as annoying or overly restricted.
@eliebakouch trust your intuitions. its the part why you'll never be fully replaced.
i wonder how much of "model improvement perception" (and model hype) is just human psychology being reward hacked
for instance i kinda miss fable (i used it for like 1 day) and i find some opus outputs dumb, and i genuinely have no idea if it's a real difference or just me being reward hacked
@eliebakouch something something hedonic treadmill if you showed a typical SWE from mid 2018 sonnet4.6 and said something like, "yeah so you pay like... ~$2 per hundred thousand words on the output it writes, ~$0.40 per hundred thousand words on the input..."
i wonder how much of "model improvement perception" (and model hype) is just human psychology being reward hacked
for instance i kinda miss fable (i used it for like 1 day) and i find some opus outputs dumb, and i genuinely have no idea if it's a real difference or just me being reward hacked
@eliebakouch most basic assumptions about technology excluding "ML at scale", from a 2018 pov, are the exact same ones you'd use in 2026 all the programming languages in active use? basically the same i think USB-C wasn't dominant yet, no M-Series macbooks, but, that's all
@eliebakouch Sometimes it’s just novelty effect too - you get annoyed by certain model tics, and if something else is sufficiently different, even if overall performance is basically the same, you just prefer it… until that wears off too
@eliebakouch Would be cool if there was like a "blind test" where you interact with an LLM and guess what model it is, this could be a cool mini game
@eliebakouch Opus outputs are much more annoying that fable imo. I'm >98% on it being a real difference. In case that's helpful.
@eliebakouch Both!
I spent a solid week with Fable working on various ongoing projects for at the minimum of 8hrs a day (sometimes 12-14), and I'm at >99% of it being a real difference. I could wax on and on about various metrics and feels, but it was a genuine stepwise improvement, without a doubt.
@raveeshbhalla yeah this is true
@eliebakouch Fable is definitely a significantly better model than Opus 4.8. But it's not too much better than GPT5.5, though. Other than actually having a personality (GPT5.5 is a dry savant, neuro-modified with Focus from `A Deepness in the sky`).
@eliebakouch Based on my 2 days of use I'm >95% sure that Fable is much better
@eliebakouch Re:fable itself, a noticeable difference for me was just how aggressively proactive it was. Mention an issue or idea in passing and it’d start acting on it immediately. So it didn’t feel like a good “thought partner”. But it did genuinely seem better at working on tasks >30 mins
@eliebakouch nah if you have experience one conversation was enough to know what you were dealing with.
@eliebakouch I wonder how much is due to the frequency illusion bias too.
This is how I moved from Opus 4.8 to GPT-5.5. I started seeing some folks talking highly about GPT-5.5 and started noticing more dumb mistakes from Opus which before I wouldn't pay as much attention to
id say genuine real difference. I used fable for the 3 days it was out during a huge sprint ( that was gonna happen regardless) and the difference between this and the past sprints where i used to mainly have codex and claude is genuinely there. I wouldn’t say it’s abysmal but it makes you genuinely faster
@eliebakouch Opus has always been dumb, it's a personality hire
@eliebakouch this is the same feeling as hating a song the week it releases then loving it 3 months later
are we just novelty-biased or is there actually a drop off?
@eliebakouch A lot of model perception is probably setup effects: novelty, expectation, and the fact that we remember the 2 uncanny wins more than the 20 ordinary misses. Hard to separate capability from psychology without side-by-side blind evals on your own tasks.
@eliebakouch same implication from harness improvements in last six months
@eliebakouch Both are real. But the Fable thing isn't just perception. They restricted ML research, biology, chemistry by default. Kicked users to older models with zero warning. That's actual regression.