VC Blames AI Safety Community for Failing to Create Release Standards

Josh, I think you might be operating under some bad information.

This seems to be a misunderstanding of why FLOPs have been included in every attempt at safety legislation. There’s an inherent need to ask “to what types of models do you apply the standards”, lest we run cyber evals on every academic lab’s 50M param pretrain.

Also, it’s not that people didn’t propose standards. See eg Transluce’s draft work. But to get broad acceptance of specific standards, you would need the labs to be willing to agree, which they strongly preferred to avoid doing to not tie their future hands wrt regulatory constraints under conditions they couldn’t foresee. That’s why every lab safety framework is vague on exact threshold operationalization.

I strongly disagree and consider this in the territory of highly-motivated historical revisionism. The safety activists have had years to get their acts together on developing clear and comprehensive technical standards that could govern advanced AI release decisions. Useful ideas about the importance of specifications and standards swirled within the community for years. Did the Andreesens of the world help with this? Of course not. But the AI safety world was flush with unbelievable amounts of money and the best technical talent in the world. The failure to produce off-the-shelf decision-making guidelines at a level robust enough for government to adopt and have confidence in is wholly a failure of the AI safety community. The community also very much confused the state of affairs with poorly-conceived political fights like SB 1047, which tried to advance a standard based on training FLOPs instead of anything more meaningful. If AI safety activists can't take ownership for the current state of affairs pertaining to AI safety standards, then the whole field is bankrupt of humility and ideas.

1. I proposed this vague “if it seems like it could cause cyber incidents you should test it” metric internally as a suggested policy position a couple years ago, so I agree with you! You should talk to some lawyers, they are very pessimistic about this being acceptable to regulators (whether in EU or US). If you can overcome the operationalization issues where I got stuck, that woild be a great thing for the world.

2. Who are you thinking of when you say the AI safety community antagonized the Trump administration? Anthropic definitely did, and match your critiques, but I don’t think they’re the actors who would be turned to to help with standards. Did METR? Did the OpenAI safety teams? Did the FMF? It’s important to be specific here, because I think many actors have gone out of their way to stay in good graces with the admin whenever possible.

3. I’m very curious what you think of the various glide paths that were created, and what you’d propose to do differently. In particular, CAISI is the glidepath, it was set up exactly for this, and is seemingly being sidelined by exactly the forces you say should not bear responsibility. I can’t see how that hand could have been played differently, but am super happy to discuss it sometime.

4. Obviously the AI safety community bears a bunch of responsibility for its missteps. Why did it pick a fight with OS in 2024? Why did so few people make a deliberate effort to work with the right prior to 2025, so that they started with such a trust deficit? Why did they so aggressively pursue securitization when it was probably inevitable? I just think you are picking a specific thing (not planning clear guidelines or operational instructions once it happened) that they took as far as they feasibly could and where completion was meaningfully in their control to get right beyond that point.

I think most safety people who’ve studied this issue in detail agree that the best realistic definition would involve a regulatorily defined and frequently/rapidly updatable capability or risk threshold. I don’t think I’ve ever talked to anyone who disagreed with that claim.

My understanding is that this kind of threshold has been hard to pass because frontier labs, perhaps understandably, want certainty regarding whether they are or are not subject to the regulation in question. A compute threshold is a crude proxy for capabilities/risks but it is relatively easy to measure objectively. It provides certainty.

Also, I just don’t think this particular issue is as big of a deal as you’re making it out to be. So far we’ve had compute thresholds and they seem to have worked fine. They’re a rough and somewhat arbitrary but reasonably effective way of distinguishing between frontier labs and everyone else. Eventually they will become obsolete, which is bad, but hopefully new and better definitions can be promulgated before that happens. Has anything bad actually happened because SB 53 included a compute threshold at industry’s insistence, or are you just annoyed by the fact that the theoretical justification for using this kind of threshold is unsound?

My perspective on this history is different from yours. Let's start back with the 5/16/24 version of SB 1047.

It defined covered model as follows: (f) “Covered model” means an artificial intelligence model that meets either of the following criteria: (1) The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations. (2) The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.

It also said a government agency should: "(10) (A) On or before July 1, 2026, issue guidance regarding both of the following: (i) Technical thresholds and benchmarks relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code. (ii) Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code. (B) In developing guidance pursuant to this paragraph, the Frontier Model Division shall take into account both of the following: (i) The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety. (ii) Similar thresholds used in federal law or regulations for the management of hazardous capabilities."

Was this a perfect approach? Definitely not. No approach is. But I don't think it's fair to say it ignored the problems with compute threshold.

Many people, including for example @deanwball, objected to this provision, saying, for example, that it gave too much power to a government agency or that it would ultimately cover tiny startups. And that's fair in a lot of ways! The provision was removed, and replaced with a FLOP threshold.

But we always understood that doing something about this would be better than doing nothing. The initial expanded version of SB 53 (07/08/25 version) did not even have a definition of "frontier model." Instead, it said a *company* was scoped in if it trained a 10^26 FLOP model (a necessary step politically, to ensure small startups could not claim they were theoretically covered). If a company was covered, all of its models were covered, but it could specify in its frontier AI framework "(1) How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models do not pose material catastrophic risks." (Note: having an agency issue guidance was not on the table, because companies opposed any new agency with such power, so this relied on a company self-exclusion).

This provision was removed, and it was changed to a pure FLOP threshold, *entirely because of company lobbying* (not, however, your company). I know because I was there and argued for this provision or something like it (we proposed many alternatives) to persist. But to pass the bill, it had to be changed.

This is really confusing, and I don't necessarily blame everyone for failing to understand it. I have been many times in disbelief about how this worked out, and it's not at all what I expected. But I was there, and it's what happened.

I agree that the safety community could have done a better job getting ready for the current situation instead focusing on the SB1047 culture war, but they will rightly say that at least they were trying something, so why should blame fall primarily on them when others were indifferent or actively working against them?

I'd actually love a blameless retro of how we ended up with a government that heavily regulates AI in an ad-hoc and chaotic way while the constituency that historically most wants AI regulated has had so little impact on how this regulation is happening that it outright opposes it.

Fascinating stuff with a lot to learn from, but I don't think the community has reached the stage of grief needed to look at these questions objectively yet.

I don't find this critique compelling. AI safety activists put together the most thorough and comprehensive proposal for AI regulation (SB 1047). You discount 1047 on the basis that it decided which models the safety standards applied to on the basis of FLOP count, rather than a more meaningful measure. But this feels like a quite weak critique of the bill, because it's fine if your way of filtering for what kinds of models you apply the standards to isn't perfect, so long as it's a reasonable proxy for dangerous capabilities (which I think it is).

I think AI safety activists straightforwardly have the most reasonable and principled proposal for regulating AI, and blaming the fact that the current administration's approach to regulating AI is unprincipled on them doesn't seem sensible to me.

Are we in grief? Don't know the whole AI safety world, but most of it, and I detect little of this! My hot take: most people I know want the government to stop dangerous models from release. Most people I know don't love the lack of transparency and seeming lack of coherence being used now. You can hold both views quite comfortably. Kinda like: I support criminal laws, but I don't support criminal laws absent due process. The counterfactual being asked now - could people interested in AI safety have done better? - is both fair and unanswerable. I'd just say that the only victories over Big Tech in the last 30 years have come from the AI safety mov't.

I feel like people are saying now, "What? You only stood down one of the most audacious political projects of all time in LTF, only defeated the entire tech sector that was fully united behind a 10 year preemption of state laws, only preserved CAISI when everyone thought it doomed, only created a bipartisan coalition that will make AI safety a dominant issue in the 2028 presidential campaign, only seeded Capitol Hill with champions in both parties, you should have done more!" True! We should have done more. But the AI safety movement has an enormous amount - an extraordinary amount - to be proud of.