Researcher Seeks Empirical Tests on Indirect CoT Pressure and Monitorability
Reactions from ranked influencers
2 posts
Has anyone done any empirical estimations of the effect of indirect CoT pressure on CoT monitorability? E.g. by taking CoT-detected-reward-hack examples in a toy env, programmatically rewriting them to exclude hack-reasoning while preserving the misbehavior, and updating against…
see also
I guess it seems like there is a nonnegligible chance that very-indirect CoT pressure is a folk-belief not borne out by empirics like, removing hacking samples obviously biases the gradient. but a less direct step fixing RL envs only biases the CoT via inducing a shifted…