• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI
Reaction

How should closed-model evaluations account for refusals at higher reasoning effort?

A user reports fewer refusals and higher accuracy at low reasoning effort on cyber and bio tasks.

2 Sources, 1h ago, first seen 1h ago

TLDR

A user describes a quirk in evaluating closed models on tasks that can trigger refusals, such as cyber and bio tasks: low reasoning effort produces few refusals and high accuracy, while maximum effort produces many refusals and low accuracy. They ask how to report performance fairly when refusals pull down the maximum-effort score.

Combined views

—

2 Sources, first seen 1h ago

— likes— comments— saves— reposts

Combined views

—

2 Sources, first seen 1h ago

— likes— comments— saves— reposts
Image: @_lewtun on X

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

2 Sources

Lewis Tunstall@_lewtunHere's a quirk when evaluating closed models on tasks that trigger refusals (cyber, bio etc): > low reasoning effort => few refusals, high accuracy > max reasoning effort => many refusals, low accuracy This gives you test-time compute curves where max effort does way worse than low because the refusals tank the performance. What is the fairest way to report performance in these cases?1h
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    2 Sources

    Lewis Tunstall@_lewtunHere's a quirk when evaluating closed models on tasks that trigger refusals (cyber, bio etc): > low reasoning effort => few refusals, high accuracy > max reasoning effort => many refusals, low accuracy This gives you test-time compute curves where max effort does way worse than low because the refusals tank the performance. What is the fairest way to report performance in these cases?1h
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet