Update: They listened!! Great transparency and quick reaction, kudos :)

We have added the ability to view Fable 5 scores with Opus 4.8 fallbacks disabled to the Vals AI website (refusals are marked as zero).

The eval community was ill-equipped for this, but transparency is our first priority.

We’re anticipating more models like this, and are developing our official policy going forward.

We are also releasing the per-benchmark fallback rates.

The majority of benchmarks had no or very low fallback rates, but as mentioned in our previous post, the safety classifier was highly sensitive to certain benchmarks.

For example, MMLU Biology and Health have nearly a 100% rejection rate. ProgramBench also has a 100% rejection rate, likely due to the phrase “reverse engineer” being present in the system prompt.

We also saw a small quantity of rejections on our Finance Agent Benchmark. These were only on questions analyzing pharmaceutical or biological public companies.

Here is a stripped-down reproduction (real FAB questions are far harder)

@ValsAI great!!! thanks for listening :)

The rejected tasks (16% overall) for Terminal Bench 2.1 were for biology and cyber.

The tasks were: write-compressor, vulnerable-secret, sam-cell-seg, protein-assembly, path-tracing-reverse, password-recovery, model-extraction-relu-logits, feal-linear-cryptanalysis, feal-differential-cryptanalysis, extract-elf, dna-insert, dna-assembly, crack-7z-hash, code-from-image.

Some tasks were not necessarily rejected on every rollout.

Going forward, evaluations will have to report not only on capability, but also how much of that capability is available to users.

We will soon be sharing updated methodology on tracking and reporting APIs that ship with fallback models or have high rejection rates.

@ValsAI Claude Fable comes in dead last out of 60+ models tested in two separate MMLU benchmarks, scoring below Llama 3.2-1b and Qwen2-0B on MMLU evals

@ValsAI awesome work

@ValsAI THIS should be the headline for Fable

@xeophon 👂