Users are excited about using deep AND-OR graphs for AI safety cases and training models on verifiable cases because the approach offers an additional capability that grounds exploration while preserving creativity.
Based on 5 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
But, it's not immediately obvious how to train models to be better about this, because we don't really have ground truth. An idea I've been excited about which seems underexplored, and could potentially be done outside labs, is to train and evaluate models on making safety cases in domains where we *do* have ground truth, or at least where the claims can be checked mechanically. For example, we could generate a bunch of bridge designs, test them out in a simulator to enumerate failure modes, and then train models to generate accurate safety cases / risk reports in this domain. Then we could do the same thing for buildings, and airplanes, and so on. If we train on just bridges and buildings, do we see generalization to held-out domains like airplanes? A particularly interesting domain is properties of complex computer programs (which have more analogies to neural networks). E.g. Ask for a case that a concurrent system deadlocks with probability < ε under a specified workload and scheduler, or that a protocol preserves an invariant. Check the claims with model checking, testing, or simulation. The point is not that bridge or program safety cases are the same as AI safety cases. They give us checkable settings for studying decomposition, evidence selection, calibration, and transfer. And even if we don't find that the learned skills transfer to safety cases for frontier models, they may have scientific and commercial value in their own right.
@iMuffined Why? Are all human engineers boring or uncreative? This is just an additional capability to have in your bag of tricks. (And actually, I think coming up with effective safety arguments does often require innovation and creativity)
@w01fe fair point, I assume this wouldn't impact exploration but instead ground it. I was thinking more from a hands off approach.
These cases can be deep AND-OR graphs. E.g. "the model is unlikely to exfiltrate weights" because either it never tries or any attempt is stopped. One branch needs evidence about intent and eval awareness; the other about monitoring, access controls, and so on.
But, it's not immediately obvious how to train models to be better about this, because we don't really have ground truth. An idea I've been excited about which seems underexplored, and could potentially be done outside labs, is to train and evaluate models on making safety cases in domains where we *do* have ground truth, or at least where the claims can be checked mechanically. For example, we could generate a bunch of bridge designs, test them out in a simulator to enumerate failure modes, and then train models to generate accurate safety cases / risk reports in this domain. Then we could do the same thing for buildings, and airplanes, and so on. If we train on just bridges and buildings, do we see generalization to held-out domains like airplanes? A particularly interesting domain is properties of complex computer programs (which have more analogies to neural networks). E.g. Ask for a case that a concurrent system deadlocks with probability < ε under a specified workload and scheduler, or that a protocol preserves an invariant. Check the claims with model checking, testing, or simulation. The point is not that bridge or program safety cases are the same as AI safety cases. They give us checkable settings for studying decomposition, evidence selection, calibration, and transfer. And even if we don't find that the learned skills transfer to safety cases for frontier models, they may have scientific and commercial value in their own right.
@iMuffined Why? Are all human engineers boring or uncreative? This is just an additional capability to have in your bag of tricks. (And actually, I think coming up with effective safety arguments does often require innovation and creativity)
@w01fe fair point, I assume this wouldn't impact exploration but instead ground it. I was thinking more from a hands off approach.
These cases can be deep AND-OR graphs. E.g. "the model is unlikely to exfiltrate weights" because either it never tries or any attempt is stopped. One branch needs evidence about intent and eval awareness; the other about monitoring, access controls, and so on.
Users are excited about using deep AND-OR graphs for AI safety cases and training models on verifiable cases because the approach offers an additional capability that grounds exploration while preserving creativity.
Based on 5 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.