While the lack of Fable 5 access remains regrettable to our many readers at http://marginalgarfield.com and http://rationalistgarfield.com, we felt it appropriate to use the opportunity to run a comparative benchmark between Sonnet 4, Opus 4.8, and Fable 5 on the Garfield comicgen task.
No Digg Deeper questions have been answered for this story yet.
With Arbuckle Systems pulling from both @MargRev and @lesswrong daily, we do a cross-source comparison, finding that Opus trades consistency for much improved LW performance.
Based on this analysis we are implementing a router to take advantage of the differential capability.
LasagnaBench implements a LLM-judge evaluation of three panel strips against five key attributes.
While the models are indistinguishable in aggregate scoring, we find Fable trades visual storytelling competence for reduced Garfield voice similarity while Sonnet does the opposite
LasagnaBench implements a LLM-judge evaluation of three panel strips against five key attributes.
While the models are indistinguishable in aggregate scoring, we find Fable trades visual storytelling competence for reduced Garfield voice similarity while Sonnet does the opposite
While the lack of Fable 5 access remains regrettable to our many readers at http://marginalgarfield.com and http://rationalistgarfield.com, we felt it appropriate to use the opportunity to run a comparative benchmark between Sonnet 4, Opus 4.8, and Fable 5 on the Garfield comicgen task.
Code is available here for all your frontier AI comic generation benchmarking needs
https://github.com/timhwang/lasagnabench
With Arbuckle Systems pulling from both @MargRev and @lesswrong daily, we do a cross-source comparison, finding that Opus trades consistency for much improved LW performance.
Based on this analysis we are implementing a router to take advantage of the differential capability.
