Anthropic's Andy Jones shares anecdote of AI system Fable predicting its own 29 percent benchmark score, prompting evaluation jokes

we don’t even run evals anymore we just ask Claude what the score will be

@andy_l_jones lmao

A couple years ago I was at a small workshop where Carlini organized a guessing game where we put questions into GPT-2 and guessed what GPT-3’s answer would be. We had a lot of fun with meta questions like asking GPT-3 to predict it / GPT-2’s score.

(past performance is not an indicator of future returns. generalization not guaranteed)

@_sholtodouglas plsssss turn down safety nerfs 👉👈

@_sholtodouglas Imagine being able to know how much compute is needed to meaningfully solve <problem>. What would the long term compute capex appetite be if this was the case?

@_sholtodouglas im confused - help me sholto!

@_sholtodouglas

@_sholtodouglas @rickasaurus Fable still only scoring slightly better on my private benchmark btw.

@_sholtodouglas lmao trust the oracle approach

benchmarks: we do them, we just skip the waiting

@andy_l_jones Predictive processing at its finest.

@_sholtodouglas bro saw the future and decided to skip the test

@_sholtodouglas Simple regression ez

@_sholtodouglas honestly this is cheaper and faster, why run 8 GPUs for a number nobody trusts anyway