Users are excited about research ideas for training models on safety cases despite lacking ground truth, in response to warnings that AI capabilities automation may outpace safety research.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.
But, it's not immediately obvious how to train models to be better about this, because we don't really have ground truth. An idea I've been excited about which seems underexplored, and could potentially be done outside labs, is to train and evaluate models on making safety cases in domains where we *do* have ground truth, or at least where the claims can be checked mechanically. For example, we could generate a bunch of bridge designs, test them out in a simulator to enumerate failure modes, and then train models to generate accurate safety cases / risk reports in this domain. Then we could do the same thing for buildings, and airplanes, and so on. If we train on just bridges and buildings, do we see generalization to held-out domains like airplanes? A particularly interesting domain is properties of complex computer programs (which have more analogies to neural networks). E.g. Ask for a case that a concurrent system deadlocks with probability < ε under a specified workload and scheduler, or that a protocol preserves an invariant. Check the claims with model checking, testing, or simulation. The point is not that bridge or program safety cases are the same as AI safety cases. They give us checkable settings for studying decomposition, evidence selection, calibration, and transfer. And even if we don't find that the learned skills transfer to safety cases for frontier models, they may have scientific and commercial value in their own right.
A concern I've heard quite a bit recently is that it will be easier to automate capabilities research than safety and alignment research. Among other things, you only need to find *one* path to recursive capabilities improvement, whereas you still need to automate *all* the jobs of the safety and alignment teams.
A particularly challenging top-level task for safety and alignment is making a *safety case*; figuring out how to make an argument for a property we might care about, like "in deployment, it's very unlikely that X will happen.
Users are excited about research ideas for training models on safety cases despite lacking ground truth, in response to warnings that AI capabilities automation may outpace safety research.
Based on 1 visible X reactions from 1 accounts; directional sample.
Ask a question below.
Published answers will appear here.