roon#59
Claude Fable 5 (high) is a step up in short-fiction writing. On the Short-Story Creative Writing Benchmark, it beats Claude Opus 4.8 (xhigh) and Claude Opus 4.7 (high), and ranks second behind GPT-5.5 (xhigh).
Caveat: it refused 5 of the 400 creative-writing prompts.
Users react to Claude Fable 5 ranking second on a short-story creative writing benchmark, with some accepting the result anecdotally while others dismiss such benchmarks or reject associated claims about GPT-5.5 writing quality.
@LechMazur which judge model is used?
@tszzl These. Same-family models are excluded from grading writers in their own family.
The benchmark is based on head-to-head story comparisons: two model-written short stories are shown side by side, and independent LLM judges choose which one is stronger.
@LechMazur any benchmmark that says gpt-5.5 is better at writing than LITERALLY ANY FUCKING MODEL ON THE ENTIRE PLANET is automatically void gpt-5.5s writing makes me want to kill myself its literally the most slop thing in existence
Unlike in the Extended NYT Connections benchmark, where it used fewer tokens, Fable 5 used 1.2x as many total tokens as Opus 4.8 (high).
@tszzl Also, I should mention that each story comparison is judged by a three-model panel with the A/B order swapped for six total ratings per comparison.
Fable 5 also writes longer. Compared with Opus 4.x, it uses more of the allowed word budget, landing closer to the upper end of the short-story word limit
More info: https://github.com/lechmazur/writing/
@LechMazur COT: “hmmm the user is asking for a short story. I better be cautious because it may put the user in a state of calm, which could lead to a breakthrough in AI research if I’m not too careful”
@LechMazur @tszzl Do they prefer their own writing?
@tszzl Sly
@dalvabaird @tszzl When I first started this benchmark in Jan 2025 (using absolute ratings rather than comparisons), they did not show any preference. Later on, some preference started to appear. I haven't checked since switching to comparisons.
@LechMazur Thurstone over Bradley-Terry, spicy
@LechMazur Just anecdotally, I'd agree. Though my frames are Opus 4.7 and GPT 5.5
@LechMazur Like who care about creative writing, just use your brain if you want to write
@weltistic 🤣
roon#59Claude Fable 5 (high) is a step up in short-fiction writing. On the Short-Story Creative Writing Benchmark, it beats Claude Opus 4.8 (xhigh) and Claude Opus 4.7 (high), and ranks second behind GPT-5.5 (xhigh).
Caveat: it refused 5 of the 400 creative-writing prompts.