We had access to Fable over the past few days

We were able to run it against public data but couldn't do semi-private (our private verification set) due to the new data retention policies

We're working with them to figure out a way to keep verification data private to ensure that ARC-AGI benchmarks continue to give us signal

fwiw, it did well on public data - we'll share once we're able to run semi private

The “Frontier lab” label has conveniently expanded to mean almost any team writing software.

An inference startup writing kernels, any sass company building evals, a student researcher working on parallelism

Now they’re all expected to accept being snooped on and nerfed.

@arcprize Yep, this seems like a big deal for enterprises (?)

Anthropic's new data retention policies don't mesh well with enterprise users who need zero-data retention (ZDR). ARC also leverages ZDR to run our benchmarks without risk of exposure of private dataset.

Insanely short-termist attitude

Here's a link to Anthropic's new support doc about data retention practices for Mythos-class models for anyone curious:

https://support.claude.com/en/articles/15425996-data-retention-practices-for-mythos-class-models

From the intro paragraph:

> To ensure we’re responsibly deploying Mythos-class models, we are requiring limited data retention and review as part of our safety work. Prompts submitted to, and outputs generated by, Mythos-class models are retained for 30 days for trust and safety purposes, on every platform where these models are offered.

@arcprize ARC is one of the few places left doing independent verification at this level. if their test data cant stay private under mythos terms, the only public fable scores come from anthropics own blog. that shifts what any benchmark claim actually means

For previous models we had zero-data retention in place but that isn't the case anymore

I'm confident there is a solution going forward

@arcprize How should we read this? If anyone ever uses their new model, they automatically have the right to use your data?

@arcprize Thank you for standing up against this draconian closed bs

@1slimewell @arcprize Yep, from my understanding, I think you need to turn on Retention if you want to use it through API

@goncalo_canhoto @arcprize Holy shit is this including the API?

@casper_hansen_ @arcprize No, that isn't the case per their docs

They explain it well (over 3 docs though) here

https://support.claude.com/en/articles/15425996-data-retention-practices-for-mythos-class-models

@arcprize They said they made it "safe". Did that impact ZDR policies?

@arcprize 😢

@arcprize Do you expect his score to improve significantly?

@arcprize That means never? Because they say they won't train on the data, but you not running it means you don't trust them.

@arcprize Will you evaluate current leading Chinese models, such as Qwen 3.7 Max, Kimi K2.6 or Minimax 3 on Arc-AGI-1/2? Would be useful

@arcprize

@arcprize interesting way to avoid arc benchmark.