Stateless API calls cannot easily distinguish defensive queries from attacks
If alignment were easy, would you still need bio/cyber/r&d classifiers on top of your model?
You'd align the model to a principal who doesn't want that work to be done. The model would deploy its full cognition to distinguishing forbidden from valid work
Alas
@stuhlmueller hm while there is an alignment problem i think there's also a ~fundamental "harness" problem- a model hit via raw stateless api calls to find bugs in a codebase genuinely needs way more affordances + information to differentiate between negative and positive use
@lukestanley Yup - and *if alignment were easy* we would be able to have that trust
It's not, of course! But it's easy to forget that because the models are so helpful in practice
@stuhlmueller Nothing wrong with training a model to preferring avoid providing dangerous things, but getting rid of separate safety classifiers would require a lot of trust to justify the independence, with current techniques, surely?
@stuhlmueller that's a wild way to look at safety constraints.
@stuhlmueller Yeah, the fact that in order to launch Fable Anthropic had to throw the kitchen sink at it is pretty revealing that robust alignment isn't here yet. With "prompt modification, steering vectors, PEFT-style intervention" even the infrastructure considerations are mind boggling.
@stuhlmueller so basically the ideal model becomes the most paranoid regulator
Stateless API calls cannot easily distinguish defensive queries from attacks
If alignment were easy, would you still need bio/cyber/r&d classifiers on top of your model?
You'd align the model to a principal who doesn't want that work to be done. The model would deploy its full cognition to distinguishing forbidden from valid work
Alas