Safety conditions for one-decision systems and long-running agents

The author describes a precise safety condition for a simple one-decision model: the principal's utility must be expressible as a weighted combination of reviewer utilities, with no negative weights. In that model, the author says, this condition is both necessary and sufficient for safety.

For long-running agents, the thread considers a Markov decision process (MDP), where utilities depend on actions and states, and actions change the state. The author says the performance difference identity extends the one-shot characterization, applied to Q values, to the full MDP.