Reaction
How should closed-model evaluations account for refusals at higher reasoning effort?
A user reports fewer refusals and higher accuracy at low reasoning effort on cyber and bio tasks.
TLDR
A user describes a quirk in evaluating closed models on tasks that can trigger refusals, such as cyber and bio tasks: low reasoning effort produces few refusals and high accuracy, while maximum effort produces many refusals and low accuracy. They ask how to report performance fairly when refusals pull down the maximum-effort score.
Combined views
—
2 Sources, first seen ago
— likes— comments— saves— reposts
