• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    AI Evaluators Forum published a 2025 standard on independence and transparency

    A user calls for more attention to the group’s work, highlighting a standard covering embedded audits and members including METR, Transluce, RAND, AVERI, Princeton and SecureBio.

    Miles BrundageMB
    Nathan LambertNL
    Arvind NarayananAN
    37 Sources, ,

    TLDR

    A post urges more people to learn about the AI Evaluators Forum, saying the group published a standard on independence and transparency covering embedded audits in 2025. It names METR, Transluce, RAND, AVERI, Princeton and SecureBio among the group’s members.

    Combined views

    255.9K

    37 Sources, first seen 24d ago

    Combined views

    255.9K

    37 Sources, first seen 24d ago

    1.6K likes
    24d ago
    first seen 24d ago
    1.6K likes
    128 comments
    578 saves
    407 reposts
    128 comments
    578 saves
    407 reposts

    Sentiment

    Positive51.4%48.6%Negative

    Summary

    Positive accounts welcomed calls for independent safety evaluators at frontier AI labs to enable greater oversight, while negative accounts questioned whether the evaluators could stay truly independent or avoid becoming rubber stamps.

    Based on 48 sentiment-bearing replies from 37 accounts across 5 conversations.

    Sentiment

    Positive51.4%48.6%Negative

    Summary

    Positive accounts welcomed calls for independent safety evaluators at frontier AI labs to enable greater oversight, while negative accounts questioned whether the evaluators could stay truly independent or avoid becoming rubber stamps.

    Based on 48 sentiment-bearing replies from 37 accounts across 5 conversations.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    37 Sources

    Kevin Klyman@kevin_klymanI hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR but also Transluce, RAND, AVERI, Princeton, and SecureBio + others In 2025 they published a standard on independence and transparency covering embedded audits24d
    Yo Shavit@yonashavthe apparatchik’s year in year out grind of dull ai governance administrivia will be honored in the world to come24d
    Business Insider@BusinessInsiderA hot job title, "embedded evaluator," has entered the AI space. https://bit.ly/4h7fCym24d
    AI Evaluator Forum@aievalforumThe AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify AI safety and security claims. This is a first step towards trustworthy oversight of frontier AI. No single evaluator can do this work alone. A vibrant ecosystem of independent evaluators from distinct backgrounds can offer a range of expertise and methodology, help to ensure rigor, and avoid a single point of failure. The Forum exists to strengthen this ecosystem. Realizing the benefits of third-party evaluations also requires independence, access, and transparency. AEF brings evaluators together to address these questions. Our first standard, AEF-1: Minimum Operating Conditions for Independent Third-Party AI Evaluations, envisions a minimum floor for access, managing conflicts of interest, funding relationships, recusal requirements, and transparency of evaluation terms. Read more here: https://aievaluatorforum.org/initiatives/minimum-operating-conditions The long-term success of third-party evaluation relies on standards and frameworks. AEF is committed to developing these with its members and collaborators. We welcome engagement from organizations committed to rigorous, independent work. Learn more about AEF, express interest in joining, and raise questions for us here: https://aievaluatorforum.org/24d
    Sayash Kapoor@sayashkRT @aievalforum: The AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify…24d
    AVERI@AVERIorgAVERI's Executive Director, @Miles_Brundage, spoke with Business Insider about recent discussions of embedded auditing:24d
    rishi@RishiBommasaniRT @kevin_klyman: I hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR b…24d
    Miles Brundage@Miles_BrundageRT @AVERIorg: AVERI is proud to be a member of AEF and looks forward to continued work on establishing clear standards as the evaluation ec…24d
    Deb Raji@rajiinioThe AI audit space has been consistently chaotic because it's a self appointed label - literally anyone can call themselves an auditor! It should not be up to public opinion or the audit target (ie the frontier lab companies) to certify who is actually qualified to play that role22d
    Transluce@TransluceAIFrontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵 https://transluce.org/embedded-evaluations21d

    37 Sources

    Kevin Klyman@kevin_klymanI hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR but also Transluce, RAND, AVERI, Princeton, and SecureBio + others In 2025 they published a standard on independence and transparency covering embedded audits24d
    Yo Shavit@yonashavthe apparatchik’s year in year out grind of dull ai governance administrivia will be honored in the world to come24d
    Business Insider@BusinessInsiderA hot job title, "embedded evaluator," has entered the AI space. https://bit.ly/4h7fCym24d
    AI Evaluator Forum@aievalforumThe AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify AI safety and security claims. This is a first step towards trustworthy oversight of frontier AI. No single evaluator can do this work alone. A vibrant ecosystem of independent evaluators from distinct backgrounds can offer a range of expertise and methodology, help to ensure rigor, and avoid a single point of failure. The Forum exists to strengthen this ecosystem. Realizing the benefits of third-party evaluations also requires independence, access, and transparency. AEF brings evaluators together to address these questions. Our first standard, AEF-1: Minimum Operating Conditions for Independent Third-Party AI Evaluations, envisions a minimum floor for access, managing conflicts of interest, funding relationships, recusal requirements, and transparency of evaluation terms. Read more here: https://aievaluatorforum.org/initiatives/minimum-operating-conditions The long-term success of third-party evaluation relies on standards and frameworks. AEF is committed to developing these with its members and collaborators. We welcome engagement from organizations committed to rigorous, independent work. Learn more about AEF, express interest in joining, and raise questions for us here: https://aievaluatorforum.org/24d
    Sayash Kapoor@sayashkRT @aievalforum: The AI Evaluator Forum (AEF) welcomes recent statements regarding the importance of embedded independent experts to verify…24d
    AVERI@AVERIorgAVERI's Executive Director, @Miles_Brundage, spoke with Business Insider about recent discussions of embedded auditing:24d
    rishi@RishiBommasaniRT @kevin_klyman: I hope more people will become familiar with the great work of the AI Evaluators Forum, a group including not just METR b…24d
    Miles Brundage@Miles_BrundageRT @AVERIorg: AVERI is proud to be a member of AEF and looks forward to continued work on establishing clear standards as the evaluation ec…24d
    Deb Raji@rajiinioThe AI audit space has been consistently chaotic because it's a self appointed label - literally anyone can call themselves an auditor! It should not be up to public opinion or the audit target (ie the frontier lab companies) to certify who is actually qualified to play that role22d
    Transluce@TransluceAIFrontier lab CEOs are calling for embedded 3rd party evaluators to help oversee AI risks. But what should third parties actually do within labs? We share some initial thoughts on how embedded evaluators could help avoid incidents like the Hugging Face hack and monitor for future risks 🧵 https://transluce.org/embedded-evaluations21d