AI Evaluators Forum published a 2025 standard on independence and transparency

@kevin_klymanI hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR but also Transluce, RAND, AVERI, Princeton, and SecureBio + others In 2025 they published a standard on independence and transparency covering embedded audits

@yonashavthe apparatchik’s year in year out grind of dull ai governance administrivia will be honored in the world to come

@BusinessInsiderA hot job title, "embedded evaluator," has entered the AI space. https://bit.ly/4h7fCym

@aievalforumThe AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify AI safety and security claims. This is a first step towards trustworthy oversight of frontier AI. No single evaluator can do this work alone. A vibrant ecosystem of independent evaluators from distinct backgrounds can offer a range of expertise and methodology, help to ensure rigor, and avoid a single point of failure. The Forum exists to strengthen this ecosystem. Realizing the benefits of third-party evaluations also requires independence, access, and transparency. AEF brings evaluators together to address these questions. Our first standard, AEF-1: Minimum Operating Conditions for Independent Third-Party AI Evaluations, envisions a minimum floor for access, managing conflicts of interest, funding relationships, recusal requirements, and transparency of evaluation terms. Read more here: https://aievaluatorforum.org/initiatives/minimum-operating-conditions The long-term success of third-party evaluation relies on standards and frameworks. AEF is committed to developing these with its members and collaborators. We welcome engagement from organizations committed to rigorous, independent work. Learn more about AEF, express interest in joining, and raise questions for us here: https://aievaluatorforum.org/

@sayashkRT @aievalforum: The AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify…

@AVERIorgAVERI's Executive Director, @Miles_Brundage, spoke with Business Insider about recent discussions of embedded auditing:

@RishiBommasaniRT @kevin_klyman: I hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR b…

@AVERIorgAVERI is proud to be a member of AEF and looks forward to continued work on establishing clear standards as the evaluation ecosystem matures.

@Miles_BrundageRT @AVERIorg: AVERI is proud to be a member of AEF and looks forward to continued work on establishing clear standards as the evaluation ec…

@Miles_BrundageRT @AVERIorg: AVERI's Executive Director, @Miles_Brundage, spoke with Business Insider about recent discussions of embedded auditing:

@kevin_klymanMETR is just one of many existing AI evaluators! If you take issue with METR, that isn’t a good argument against requiring companies to carry out embedded audits Here are 20 other evaluators who work with AI labs to assess risk: Transluce Grey Swan Apollo AVERI SecureBio Faculty Vaultis Dreadnode Irregular RAND Far AI ActiveFence (now Alice) Active Site Deloitte (via Gryphon acquisition) Nemesys Mercor (via Sepal acquisition) AE Studio Scale Frontier Design Redwood

@Miles_BrundageRT @kevin_klyman: METR is just one of many existing AI evaluators! If you take issue with METR, that isn’t a good argument against requirin…

@rajiinioThe AI audit space has been consistently chaotic because it's a self appointed label - literally anyone can call themselves an auditor! It should not be up to public opinion or the audit target (ie the frontier lab companies) to certify who is actually qualified to play that role

@rajiinio@yonashav @Miles_Brundage Hm, from my end I can see that some of the older AI audit groups (Parity AI, Credo AI, Fiddler) are not included; IAAA participants (ie, https://www.iaaa-algorithmicauditors.org/about), civil society groups like ACLU, CDT + Data & Society all have "AI audit" teams, many using similar empirical methods

@rajiinio@Miles_Brundage @yonashav I'm a big MDSAP nerd and I suspect that's the closest to what the AI eval forum could be ( since it's also a set of conduct standards defined by the audit organizations themselves and then adopted by accountability actors. - ie govt). Happy to chat about it more if/when useful!

@rajiinioYes, they have nowhere near the resources of METR but I know these teams are capable of meaningful empirical AI investigations! Also, given their longstanding proximity to the actual impacted population, they could be very well positioned to provide a unique perspective here.

@RishiBommasaniFor what its worth, I was in the room when AEF was being built and AEF-1 was written, and knew many of the folks in this older auditing group that Deb mentions, and am glad they were not in the room to get this moving. I think those doing frontier AI audits and independent evals need not be grouped with a larger collective of entities auditing AI/ML technologies writ large that dont yet do substantial work in the frontier AI space. I think AEF advanced a lot faster on the substance because if it, having watched lots of other standards process be ineffectual and bogged down in process/voting that never get any substantive work done. Imo the core area of missing coverage (oft discussed for AEF) is really the for-profit frontier evaluators not the old AI auditors.

@TransluceAIFrontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵 https://transluce.org/embedded-evaluations

@JacobSteinhardtRT @TransluceAI: Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties ac…

@JsevillamolAI companies should pay for their own auditing costs. This would be similar to how audits work in banking and other industries and would allow the auditing ecosystem to scale at the same pace as the AI industry.

@robinomialRT @TransluceAI: Frontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties ac…

@rohanpaul_aiOn BBC Mustafa Suleyman (CEO of Microsoft AI), told AI companies now agree on independent auditors and that those auditors may need government funding “the industry is all now in agreement that there should be independent auditors um embedded evaluators that work inside the companies um who are singularly there to verify the kinds of training runs that are being run, their safety, the integrity of those systems and report back to independent third parties. I think that they need to be embedded inside of the companies. They need to be completely independent. Um and I, you know, mo most importantly, they have to have the technical expertise to be able to conduct the work. Um but they also need to be incentivized to be um independent. And that means that they either need to be, you know, funded by government or supported by government” ---- From "BBC News" YouTube channel, (full video link in comment)

AI Evaluators Forum published a 2025 standard on independence and transparency

TLDR

Combined views

AI Evaluators Forum published a 2025 standard on independence and transparency

22 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

22 Sources

Sentiment

Summary