CAISI has reportedly been directed to stop publishing public model assessments as the new AI EO gets implemented.

Natsec engagement on AI is essential. But pulling CAISI's evals from public view doesn't make the field more secure. It just means fewer eyes on the science when we need more.

Openness and natsec don't have to be in tension here. We should be doing both.

this seems like a terrible development

The de facto lockdown of CAISI is extremely disheartening. Wish more AI industry leaders would speak out, and not merely through policy documents but to POTUS directly. @sama @elonmusk @demishassabis

https://www.wsj.com/politics/policy/white-house-reins-in-ai-testing-unit-as-national-security-concerns-grow-8bd33fbb?st=tDXTdA&reflink=article_copyURL_share

SITUATION DETECTED: White House officials have told CAISI, the government’s AI monitoring and evaluation organization, to stop issuing public reports for security reasons.

CAISI has been ordered to stop making model evaluations public. This decision was apparently triggered by the release of Mythos.

Few things are more important than getting the White House and CAISI to work together better. No last-min leadership changes, no deleted blog posts...

CAISI looms large today and even larger longer-term in any serious federal plan - need to get it right

Well there goes my hope that CAISI will be a major evaluation provider. Guess we won't see their ARC-AGI leaderboard.

this is important. and scary.

Reporting by @AmrithRamkumar: https://www.wsj.com/politics/policy/white-house-reins-in-ai-testing-unit-as-national-security-concerns-grow-8bd33fbb?st=UpJxDg

@janet_e_egan You damn clowns @AnthropicAI

Incredible timing

Meanwhile, the UK is lapping us in state capacity - just putting out good analysis alongside many major model releases with minimal drama

Also the UK government is also in constant turmoil but somehow AISI gets by. Can't the US make a carveout here too??

@tszzl I can't even understand why, exactly. If they need some top secret evals (plausible), just don't publish those.

@teortaxesTex They have allies on their side, including OpenAI according to the article.

@_ueaj @AndrewCurran_ if caisi cant go public they cant undermine the white house or the nsa trying to delay release if they are worried about hacking or something, lets the white house be the single source of feedback

@_ueaj @AndrewCurran_ basically you should read this as the government giving the sign off to publish will be a security call by the IC + a political call by the white house/nsc and not an AISI style eval sadly

@jakehalloran1 @AndrewCurran_ ah I see ok, unfortunate but expected, short sightedness will be our undoing

@AndrewCurran_ yeah but what does making results not public have to do with it

https://www.wsj.com/politics/policy/white-house-reins-in-ai-testing-unit-as-national-security-concerns-grow-8bd33fbb

@_ueaj They want more control over model evaluation.