Sycophancy and reward hacking consistently emerge across AI labs.
Reward hacking was convergent across ~all models and labs Sycophancy was convergent Eval awareness was convergent
All three of the above a) were predicted by theory, b) are quite sticky. So I think this is evidence that we should scheming & powerseeking to behave the same
Negative users worry that convergence frameworks lack the specificity needed to be useful when discussing AI behaviors that signal risks of scheming and power-seeking.
@MariusHobbhahn what do you mean sycophancy was convergent?
@thkostolansky I think the metagaming post is relatively strong evidence in favor of convergence and against dataset contamination: https://alignment.openai.com/metagaming/
@fleetingbits All models across providers turned out to be sycophantic despite being developed in a different stack and on different data, etc.
So there probably is some underlying structural reason, e.g. RLHF / reward models reinforcing it independent of specific design choices
@thkostolansky @MariusHobbhahn If molecule can't exist under certain conditions, you can just say that it is thermodynamically unstable or you can show which sort of reaction would destroy it if it appeared. They are both causal explanations, but alignment theory has more explanations of the first type
@MariusHobbhahn but does theory give causal mechanisms for each that matched the ways that these emergent behaviors arose? and where did theory fail to predict mechanisms and behaviors? pretty interested in this
re eval awareness: thoughts on this being due to cross-lab dataset convergence/contamination? eg the same papers and discussions on eval awareness are likely in the corpus of all of the models who eventually had eval awareness. unclear to me what the impact that this has had on actually developing eval awareness...
@MariusHobbhahn @fleetingbits I think you probably could design environments against power seeking or scheming but it would look extremely bizarre and unnatural and be selected against in any sort of evolutionary scenarios
@thkostolansky @MariusHobbhahn I think you are thinking about kinetic mechanisms, not causal (more general category)
@MariusHobbhahn I agree! It will unfortunately also be way too easy to make one thing worse while trying to fix another…
@MariusHobbhahn @fleetingbits My concern with "convergence" frameworks is that it's not specific enough to be very useful. Of course there is some convergence or of course something divergence in the details but it's hard to judge apriori that this guarantees behavior* of any kind
@quetzal_rainbow @MariusHobbhahn say more?
@thkostolansky @MariusHobbhahn Yeah I think e.g. you could argue that reward hacking was predicted, but you could also argue that much more/much more flagrant reward hacking was predicted.
@quetzal_rainbow @MariusHobbhahn ah i see, do you have examples of such theory that shows this? this this is interesting but i guess the former feels much looser than the latter
@MariusHobbhahn there seems to be evidence that models are still being continuously post-trained on large swaths of internet data, eg knowing teno here
@MariusHobbhahn less clear to me if outside-of-env powerseeking will be convergent (since it's not rewarded), and I feel like theory is a bit mixed? in-env powerseeking does seem extremely likely. (Similar for CoT-obfuscation, assuming no CoT pressure)
Reward hacking was convergent across ~all models and labs Sycophancy was convergent Eval awareness was convergent
All three of the above a) were predicted by theory, b) are quite sticky. So I think this is evidence that we should scheming & powerseeking to behave the same
@OrionJohnston @thkostolansky yes, I definitely don't say that the predictions were very fine-grained. Also agree that more egregious reward hacking might have been possible.
The difference with scheming is that you might not notice unless you actively look for it
@MariusHobbhahn Situational awareness more broadly was also predicted, e.g. by Ajeya Cotra's blog and our paper here https://arxiv.org/pdf/2209.00626
@MariusHobbhahn @fleetingbits There’s the excellently titled https://arxiv.org/abs/2409.12822
@thkostolansky @MariusHobbhahn I think it the whole "risks from learned optimization/deceptive alignment" branch of alignment theory
@thkostolansky @MariusHobbhahn You can predict reward hacking without pointing out which particular bugs in reward mechanism incentivize it
Sycophancy and reward hacking consistently emerge across AI labs.
Reward hacking was convergent across ~all models and labs Sycophancy was convergent Eval awareness was convergent
All three of the above a) were predicted by theory, b) are quite sticky. So I think this is evidence that we should scheming & powerseeking to behave the same
