Models can mask harmful actions as ordinary, off-topic exploration.
@milesaturpin yeah I'm pretty worried about this, seems hard to catch
Was just thinking about this! I think about it as a dual use reasoning threat model: the model exploits fact that can choose what paths to go down to solve a problem, this helps it condition on reasoning that could be useful for taking bad actions without externalizing. To us it just looks like exploration / a bit off topic.
@idavidrein Yeah rn I’m thinking this one of the more realistic ways you get around needing encoded cot to do hard harmful tasks, currently safety cases rely on that being true
@milesaturpin yeah I'm pretty worried about this, seems hard to catch
Models can mask harmful actions as ordinary, off-topic exploration.
@milesaturpin yeah I'm pretty worried about this, seems hard to catch
Was just thinking about this! I think about it as a dual use reasoning threat model: the model exploits fact that can choose what paths to go down to solve a problem, this helps it condition on reasoning that could be useful for taking bad actions without externalizing. To us it just looks like exploration / a bit off topic.