Overall it's p. good / maybe "SOTA" among industry proposals though that's complicated -- e.g. much of what OAI has said is exploratory in tone, SB 315 (which Ant/OAI both endorsed) is both stronger + weaker in different ways than company statements, etc.
Some misc. comments on the auditing-related provisions in Anthropic's proposed "Advanced AI Framework" 🧵
No Digg Deeper questions have been answered for this story yet.
That's it for now - lots more one could say about all these comparisons.
AI is good enough now to do a decent (though fallible) job at making these comparisons, btw - would encourage people to do this sometimes (e.g., ask for a comparison of proposal X to legislation Y).
If there's one thing I encourage people to take away, it's that "mandatory audits" are a multidimensional thing - many are now "in favor" at a high level but that belies a lot of complexity.
Here's that AVERI post btw: https://www.averi.org/ourwork/frontier-ai-auditing-related-legislation-in-the-us-landscape-challenges-and-a-path-forward
Won't go into the detail which would require more careful review than I can do this sec but suffice it to say that I *think* SB 315 and O-T / GAAIA would at least somewhat more strongly empower auditors access-wise. Also perhaps weaker than some aspects of the GPAI COP (see pic)
Access - this is an area where it seems like Ant (+OAI's) endorsement of SB 315 is especially interesting. SB 315 (pic 1) and Obernolte-Trahan (pic 2) go a little bit beyond explicit industry proposals (while still having guardrails around security/confidentiality etc.)...
I'd be all for that as a target, if the alternative was no target, but I don't think it's self-evident that this is a better target than, e.g., "is the company following their policies + are those policies reasonable" (~Obernolte-Trahan)...
...6 months would feel like an eternity - we could have ASI by then! Or it could be imminent.
I assume Ant would acknowledge this + say this is just their current best proposal, but anyway, see the AVERI post where we discuss "release valves" as a complement to faster kick-in.
"Evaluator shopping" - will have more to say before long on this. V. related to the risk of "checkbox compliance."
I agree there should be mitigations, though not sure I am sold on the one floated here (~ratings + random assignment), though granted, it's just floated
Cadence of audits - this is a bit vague. SB 315 is annual, Obernolte-Trahan is semi-annual (with the possibility of more frequent or continuous at CAISI Director's discretion ... Anthropic just says "regularly." Presumably this means at least at the cadence of risk reports but 🤷♂️
...but obviously that can't be an evergreen proposal - depends a lot on what happens in the future. Let's say that we dilly dally both at a state and federal level and don't pass anything better than SB 315 until like mid-2027. In such a scenario...
Starting with the philosophy - "Self-assessment is not enough" - yep!
Interestingly, they do *not* lean into the "separation of powers" type case for private auditors (vs. gov't), and indeed suggest gov't would be ideal - think I disagree (though again it's kinda complicated)...
Overall it's p. good / maybe "SOTA" among industry proposals though that's complicated -- e.g. much of what OAI has said is exploratory in tone, SB 315 (which Ant/OAI both endorsed) is both stronger + weaker in different ways than company statements, etc.
Going into force - this is one of the more notable bits so I'll dwell on it for a few tweets
Right now, for all its virtues, SB 315 doesn't go into force until Jan. 1, 2028, and until then we rely on the EU 🇪🇺 as a (public sector) demand signal for audit-like stuff. Not ideal...
Cadence of audits - this is a bit vague. SB 315 is annual, Obernolte-Trahan is semi-annual (with the possibility of more frequent or continuous at CAISI Director's discretion ... Anthropic just says "regularly." Presumably this means at least at the cadence of risk reports but 🤷♂️
Target of audit - risk reports.
Not surprisingly, Anthropic emphasizes auditing the thing that they are currently most excited about self-assessing + voluntarily having assessed by others - regularly scheduled holistic, cross-model risk assessments at a company level.
Independence requirements - disclosure of conflicts, yep, no major ones, yep...
They float (though don't strongly endorse) cooloff periods + pooled funding.
Will say more about cooloff at some point but I do support pooled funding (discussed briefly in Frontier AI Auditing)
Independence requirements - disclosure of conflicts, yep, no major ones, yep...
They float (though don't strongly endorse) cooloff periods + pooled funding.
Will say more about cooloff at some point but I do support pooled funding (discussed briefly in Frontier AI Auditing)
Starting with the philosophy - "Self-assessment is not enough" - yep!
Interestingly, they do *not* lean into the "separation of powers" type case for private auditors (vs. gov't), and indeed suggest gov't would be ideal - think I disagree (though again it's kinda complicated)...
6 months from passage of the law is a reasonable proposal, and *assuming something were to pass relatively soon,* fairly aligned w/ my thinking (e.g. in AVERI's recent post we discussed early 2027)...
Going into force - this is one of the more notable bits so I'll dwell on it for a few tweets
Right now, for all its virtues, SB 315 doesn't go into force until Jan. 1, 2028, and until then we rely on the EU 🇪🇺 as a (public sector) demand signal for audit-like stuff. Not ideal...
@Miles_Brundage Annual audits on models shipping weekly. Either you pause deploys during audit windows or you're always auditing last month.
How do teams navigate that gap?