Roon Details Monitoring Shortfalls in AI Safety
OpenAI researcher explains infrastructure downtime and false positives as key limits on monitoring systems.
Roon, a pseudonymous OpenAI technical staff member, posted that monitoring cannot serve as a general AI safety solution because it runs on flaky infrastructure prone to downtime and produces false positives that cause alert fatigue. Replies from AI safety researchers and others noted that no single approach suffices and emphasized defense in depth along with imperfect mechanisms. Roon added that chain-of-thought monitoring still works effectively with today's models. The exchange highlights ongoing debate over practical constraints on oversight tools.
failures of “monitoring” as a general solution to ai safety: monitoring, like all other software, runs on flaky and mortal infrastructure with some amount of downtime. does a momentary blip in monitoring open Pandora’s box? will people accept fail closed monitoring on all…
Reactions from ranked influencers
12 postsi do totally agree with this, tho i think smth very focused on models escaping the sandbox during eval would catch it and i don't think you have a lot of false positives (might be v wrong) if you ask a ~smart model to monitor if the model has internet access or not. i really see…
@tszzl I'm writing a thing arguing the reverse - ex ante alignment will never be sufficient, and you'll need to rely on all sorts of imperfect mechanisms to constrain behaviour. :)
@tszzl Monitoring is one of the ways we've aligned superintelligences before, though not by itself of course, nor for entirely adversarial orthogonal agents...
Monitors are like seatbelts. They are extremely helpful but do not replace the need to drive safely.
failures of “monitoring” as a general solution to ai safety: monitoring, like all other software, runs on flaky and mortal infrastructure with some amount of downtime. does a momentary blip in monitoring open Pandora’s box? will people accept fail closed monitoring on all…
@tszzl relying on either alone is silly, there needs to be many competing layers to it the same way all of civilization handles anything critical at any zoom level safety is an immune system not an algorithm. alignment comes from nested immune systems not a mental framework