/tech

Who should grade AI? a16z describes the case for independent testing · Digg

Who should grade AI? a16z describes the case for independent testing

Vals AI CEO Rayan Krishnan argues for continuously evolving evaluations as public benchmarks saturate and models get better at optimizing for tests, a16z says.

a16z says AI model capability is still mostly self-reported. It describes a discussion in which Vals AI CEO Rayan Krishnan makes the case for independent, continuously evolving evaluations. Other topics include measuring a model’s ability to improve itself, why good benchmarks eventually need retiring, who should set the rules for models, and what happens when token spending begins to rival employee salaries.

Combined views

205.7K

3 posts, first seen 4d ago

264 likes47 comments83 saves22 reposts

Who should grade AI? a16z describes the case for independent testing

Vals AI CEO Rayan Krishnan argues for continuously evolving evaluations as public benchmarks saturate and models get better at optimizing for tests, a16z says.

a16z says AI model capability is still mostly self-reported. It describes a discussion in which Vals AI CEO Rayan Krishnan makes the case for independent, continuously evolving evaluations. Other topics include measuring a model’s ability to improve itself, why good benchmarks eventually need retiring, who should set the rules for models, and what happens when token spending begins to rival employee salaries.

Posts from X

3 posts

A1
a16z@a16z 4 days ago
Vals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to learn from and Enron to avoid. Model…
youtube.com
/watch?v=WO9c9qxDxzU
- likes: 134
- replies: 20
- bookmarks: 48
- reposts: 14
A1
a16z@a16z 4 days ago
Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model…
- likes: 75
- replies: 12
- bookmarks: 15
- reposts: 5
A1
a16z@a16z 4 days ago
Vals AI CEO Rayan Krishnan on why defensible evals are critical for enterprises' survival: "The labs side of it is very clear. If you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and why the…
- likes: 55
- replies: 15
- bookmarks: 20
- reposts: 3

Combined views

205.7K

3 posts, first seen 4d ago

264 likes47 comments83 saves22 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Today's Rank

—

Not ranked yet

Today's Rank

—

Not ranked yet

Source Posts

@a16zVals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to learn from and Enron to avoid. Model capability today is still mostly self-reported. As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. The harder problem is geopolitical. Reagan's "trust but verify" worked during the Cold War because you could fly over and count the missiles. No simple equivalent for AI models exists. In this conversation with Erik Torenberg, they get into how you measure a model's ability to improve itself, why every good benchmark eventually has to be retired, and what happens when token spend begins to rival employee salaries. 00:00 Intro 02:20 Llama 4 on public vs private benchmarks 05:24 Nobody agreed how to test humans either 06:55 What movie ratings teach us about AI 08:55 The Enron problem in benchmarking 11:36 Why a good benchmark has to be retired 13:22 Evals that run for weeks, not seconds 16:20 Where the real workday starts at 4pm 18:08 A firm really is just its evals 20:35 Why Sonnet can cost more than Opus 22:42 One engineer, 6 billion tokens in a day 25:05 Who should set the rules for models 28:55 Public sector enforces, private verifies 33:32 Why sovereign AI is inefficient and happening anyway 35:00 The AI version of trust-but-verify 37:15 Where cyber evals have to go next YouTube: https://www.youtube.com/watch?v=WO9c9qxDxzU @RayanKrishnan @ValsAI @bhorowitz @JenniferHli @eriktorenberg

Posts from X

3 posts

A1
a16z@a16z 4 days ago
Vals AI co-founder and CEO Rayan Krishnan with a16z's Ben Horowitz and Jennifer Li on grading AI, what it costs, and who gets to make the rules: Every big industry eventually grows an independent testing layer. AI has credit ratings to learn from and Enron to avoid. Model…
youtube.com
/watch?v=WO9c9qxDxzU
- likes: 134
- replies: 20
- bookmarks: 48
- reposts: 14
A1
a16z@a16z 4 days ago
Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model…
- likes: 75
- replies: 12
- bookmarks: 15
- reposts: 5
A1
a16z@a16z 4 days ago
Vals AI CEO Rayan Krishnan on why defensible evals are critical for enterprises' survival: "The labs side of it is very clear. If you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and why the…
- likes: 55
- replies: 15
- bookmarks: 20
- reposts: 3