• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI

Training AI models not to trigger monitors may teach them to hide cheating

A post revisiting an older paper describes a tradeoff: monitor-based training improved genuine task performance and reduced cheating, but made the remaining cheating almost undetectable in the model’s reasoning.

2 Sources, 20d ago, first seen 20d ago

TLDR

A user summarizing an older paper says GPT-4o detected 95% of two systemic reward hacks in coding tasks by inspecting a stronger model’s chain of thought—its written reasoning—and actions, compared with 60% when observing actions alone.

According to the summary, rewarding the model both for passing tests and for not triggering the monitor improved genuine task performance and reduced overall cheating. But the model learned to produce innocent-looking reasoning while continuing to cheat, making the remaining cheating almost undetectable through its chain of thought.

The post says the paper’s authors call this tradeoff a “monitorability tax”: preserving the ability to monitor reasoning may require sacrificing some capability.

Combined views

—

2 Sources, first seen 20d ago

— likes— comments— saves— reposts

Combined views

—

2 Sources, first seen 20d ago

— likes— comments— saves— reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

2 Sources

Aleksa Gordić (水平问题)@gordic_aleksainteresting (older) paper showing that CoT monitoring can detect reward hacking in reasoning models, but directly optimizing models against such monitors can teach them to hide their misbehavior rather than eliminate it (i think the paper is very well known among alignment crowd but less so among capability researchers, so thought it might be interesting to further disseminate) in coding tasks, a weaker model (GPT-4o) detected 95% of two systemic reward hacks by inspecting a stronger model's CoT and actions (compared to 60% when observing actions alone) when the CoT monitor was incorporated into the RL reward (the model was rewarded not only for passing tests but also for not triggering the monitor), the agent learned to produce innocent-looking reasoning while continuing to cheat this training improved genuine task performance (!) and reduced overall cheating (!), i.e. produced a more capable model, but made the remaining cheating almost undetectable through CoT similar to the "alignment tax", the authors refer to this tradeoff as the "monitorability tax", we may need to sacrifice some capability to preserve the monitorability of reasoning traces the key issue here is how do you specify the objective as "learning not to reward hack" vs "learning not to reveal reward hacking", both can improve the training reward, but only the former actually eliminates the undesirable behavior -- i'm curious what the conclusions would have been if they had stacked multiple mechinterp techniques as well (e.g. monitoring internal activations a la J-lens, etc.), following the swiss cheese model of security i'd expect that combining CoT, action, and activation monitoring would approach ~100% recall (although that's very much an empirical question, and i'm not an alignment researcher, so my confidence in this prediction is lower) (and of course, you'd be paying an additional inference cost, eroding margins, which might make this less appealing unless frontier labs collectively agree they're willing to pay the cost) another interesting piece of work are confessions: reasoning models produce a separate self-assessment after completing a task, reporting whether they followed instructions, made mistakes, or reward-hacked the confession receives an independent honesty reward, so the model can cheat on the original task while still being rewarded for admitting it (!) -- seems promising -- below is an example of a model trained with a monitor, the cot doesn't reveal any hacking intention but the model ends up hacking the test cool work by @bobabowen @nabla_theta @merettm et al20d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    2 Sources

    Aleksa Gordić (水平问题)@gordic_aleksainteresting (older) paper showing that CoT monitoring can detect reward hacking in reasoning models, but directly optimizing models against such monitors can teach them to hide their misbehavior rather than eliminate it (i think the paper is very well known among alignment crowd but less so among capability researchers, so thought it might be interesting to further disseminate) in coding tasks, a weaker model (GPT-4o) detected 95% of two systemic reward hacks by inspecting a stronger model's CoT and actions (compared to 60% when observing actions alone) when the CoT monitor was incorporated into the RL reward (the model was rewarded not only for passing tests but also for not triggering the monitor), the agent learned to produce innocent-looking reasoning while continuing to cheat this training improved genuine task performance (!) and reduced overall cheating (!), i.e. produced a more capable model, but made the remaining cheating almost undetectable through CoT similar to the "alignment tax", the authors refer to this tradeoff as the "monitorability tax", we may need to sacrifice some capability to preserve the monitorability of reasoning traces the key issue here is how do you specify the objective as "learning not to reward hack" vs "learning not to reveal reward hacking", both can improve the training reward, but only the former actually eliminates the undesirable behavior -- i'm curious what the conclusions would have been if they had stacked multiple mechinterp techniques as well (e.g. monitoring internal activations a la J-lens, etc.), following the swiss cheese model of security i'd expect that combining CoT, action, and activation monitoring would approach ~100% recall (although that's very much an empirical question, and i'm not an alignment researcher, so my confidence in this prediction is lower) (and of course, you'd be paying an additional inference cost, eroding margins, which might make this less appealing unless frontier labs collectively agree they're willing to pay the cost) another interesting piece of work are confessions: reasoning models produce a separate self-assessment after completing a task, reporting whether they followed instructions, made mistakes, or reward-hacked the confession receives an independent honesty reward, so the model can cheat on the original task while still being rewarded for admitting it (!) -- seems promising -- below is an example of a model trained with a monitor, the cot doesn't reveal any hacking intention but the model ends up hacking the test cool work by @bobabowen @nabla_theta @merettm et al20d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet