The evaluators are negotiating terms to securely release results.
We had early access to Anthropic’s Fable 5, but did not run verified Semi-Private ARC-AGI-1/2/3 evals due to their new data-retention terms for Mythos-class models.
We’re working with Anthropic to keep ARC verification data private. Scores will come once we can run them safely.
Positive users praise Anthropic's transparency and privacy standards in the new Mythos data terms while negative users accuse the changes of sabotaging ARC-AGI evaluations.
@arcprize Yep, this seems like a big deal for enterprises (?)
Here's a link to Anthropic's new support doc about data retention practices for Mythos-class models for anyone curious:
https://support.claude.com/en/articles/15425996-data-retention-practices-for-mythos-class-models
From the intro paragraph:
> To ensure we’re responsibly deploying Mythos-class models, we are requiring limited data retention and review as part of our safety work. Prompts submitted to, and outputs generated by, Mythos-class models are retained for 30 days for trust and safety purposes, on every platform where these models are offered.
@arcprize ARC is one of the few places left doing independent verification at this level. if their test data cant stay private under mythos terms, the only public fable scores come from anthropics own blog. that shifts what any benchmark claim actually means
@arcprize How should we read this? If anyone ever uses their new model, they automatically have the right to use your data?
@arcprize Thank you for standing up against this draconian closed bs
@1slimewell @arcprize Yep, from my understanding, I think you need to turn on Retention if you want to use it through API
@goncalo_canhoto @arcprize Holy shit is this including the API?
@casper_hansen_ @arcprize No, that isn't the case per their docs
They explain it well (over 3 docs though) here
https://support.claude.com/en/articles/15425996-data-retention-practices-for-mythos-class-models
Anthropic evil. Will sabatoge yoru research. this is THE MOST disgusting day int he history of LLM and humanity. a truly important and revolting day that everyone should be paying attention to. Arcprize is either frontier research, or its not. and if its frontier research, claude will sabotage your project so YOU NEED TO WAKE UP @arcprize and put them on the DO NOT USE list. they have in thier TOS they will SABOTAGE YOUR WORK AND CHARGE YOU. if thats nto the biggest fucking wake up call to the "AGI" world you guys are sleeping at the wheel. Sorry to be so vocal here but YOU NEED TO PICK A LANE NOW. either this is a toy, doesnt matter and we should be making mario, or this is real fuckgin work. the ball is in your court, and it was forced on you by an evil alignment. this is much bigger than any single submission, this is literally our childrens future on the line. you need to see this for what it is, and i assume this post is the first opening salvo in what will be a confirmation of their reprehensable, innapropriate behavior. DO YOU HAVE ANY FUCKGIN CLUE HOW MUCH SHIT I WOULD BE IN SELLING ARCPRIZE MODS THAT SECRETOLY SABOTAGED A USERS ABILITY TO COMPETE? That is literally fraude, and theft, and probably a couple other things. Why do i see this as black and white?
@arcprize They said they made it "safe". Did that impact ZDR policies?
Here’s another rug pull.
We had early access to Anthropic’s Fable 5, but did not run verified Semi-Private ARC-AGI-1/2/3 evals due to their new data-retention terms for Mythos-class models.
We’re working with Anthropic to keep ARC verification data private. Scores will come once we can run them safely.
@arcprize 😢
@arcprize Do you expect his score to improve significantly?
@arcprize That means never? Because they say they won't train on the data, but you not running it means you don't trust them.
@arcprize Will you evaluate current leading Chinese models, such as Qwen 3.7 Max, Kimi K2.6 or Minimax 3 on Arc-AGI-1/2? Would be useful
@arcprize
@arcprize interesting way to avoid arc benchmark.
@arcprize Good to see teams pushing for eval integrity instead of chasing PR scores. ARC's data terms for frontier models are a real bottleneck—curious how Anthropic responds on data retention.
@arcprize Why does semi-private exist and why isn't that unfair to others?
@arcprize I appreciate your efforts! Looking forward to seeing results on ARC AGI 3
The evaluators are negotiating terms to securely release results.
We had early access to Anthropic’s Fable 5, but did not run verified Semi-Private ARC-AGI-1/2/3 evals due to their new data-retention terms for Mythos-class models.
We’re working with Anthropic to keep ARC verification data private. Scores will come once we can run them safely.
