This just gets to how hard it is to design a mechanism to address just one kind of threat. I go back to my original worry that most policymakers aren’t registering what’s truly happening here (which is exactly why institutions like caisi are all the more important to give us visibility into what to prepare for).

We’re going to be having the same conversations we had wrt cyber on bio, on autonomy/R&D/RSI, etc in the next few months—all while major labs are looking to go public. We need a lot more agility in governance and a lot more visibility to know what to govern.

Just on cyber first, the EO is a good step but I’m a little worried about the benchmarking that NSA has to do to define a covered model by cyber capability. Say we set the bar at X, calibrated to today’s defenses. In 6-12 months X catches every model released (as defenses lag but other labs release capable models). Yes, X moves up eventually but in the near term we risk making every model covered. We may end up building a threshold that is no longer useful.

All that said a model’s capability scales with how much test-time compute you give it. Hand a weaker, uncovered model far more compute and it may find exploits the benchmark never priced in. So what are we actually measuring? The line moves the moment someone spends more.

And then the issue that this threshold is per-model. Microsoft’s MDASH found more vulnerabilities than any single frontier model by orchestrating hundreds of smaller ones together but none of those would be covered individually. So we leave a major aspect of the threat space uncovered.