Willow produced more edit-free drafts than Claude in VoiceArena’s Dictation Bench, but Claude left a smaller share of characters to fix overall, according to a November 11 post by Scobleizer. That’s an awkward split for anyone hoping to squeeze dictation accuracy onto a single winner’s podium. Source
Two metrics, different leaders
Scobleizer reports:
- Drafts needing zero edits: Willow at 75.7%, versus Claude at 64.0%. Source
- Characters requiring edits overall: Claude at 0.93%, versus Willow at 1.88%. Source
He also says five of Willow’s 761 drafts accounted for 71% of its edits. His post doesn’t detail what went wrong in those five drafts. Source
“One accuracy score hides that. I want to see both numbers,” Scobleizer wrote. Source
Neither number is a stopwatch; these figures alone don’t establish how long someone would spend fixing the output.
US English on macOS only
According to @rohanpaul_ai, VoiceArena played the same 761 recordings from 87 speakers into ten dictation apps on a Mac, testing the actual apps rather than their APIs. He says the test covers US English on macOS only. Source
He also says VoiceArena published reference policies, app configurations, judge instructions and results identifying which gaps are statistical ties. Source
The posts by @rohanpaul_ai and Scobleizer don’t give app versions or detailed scoring definitions. They also don’t establish whether the Willow–Claude differences are statistically significant. SourceSource
Comments