We had access to Fable over the past few days
We were able to run it against public data but couldn't do semi-private (our private verification set) due to the new data retention policies
We're working with them to figure out a way to keep verification data private to ensure that ARC-AGI benchmarks continue to give us signal
fwiw, it did well on public data - we'll share once we're able to run semi private
Anthropic's new data retention policies don't mesh well with enterprise users who need zero-data retention (ZDR). ARC also leverages ZDR to run our benchmarks without risk of exposure of private dataset.
For previous models we had zero-data retention in place but that isn't the case anymore
I'm confident there is a solution going forward
We had access to Fable over the past few days
We were able to run it against public data but couldn't do semi-private (our private verification set) due to the new data retention policies
We're working with them to figure out a way to keep verification data private to ensure that ARC-AGI benchmarks continue to give us signal
fwiw, it did well on public data - we'll share once we're able to run semi private
@GregKamradt what is the new data retention policy? if you use fable/mythos they collect ur data?
@GregKamradt I still don't understand how you will deal with the price, with the 10k USD cap you have on ARC AGI 3. If Opus 4.8 already hits the 10k USD cap, how can you ever benchmark Fable?
We had access to Fable over the past few days
We were able to run it against public data but couldn't do semi-private (our private verification set) due to the new data retention policies
We're working with them to figure out a way to keep verification data private to ensure that ARC-AGI benchmarks continue to give us signal
fwiw, it did well on public data - we'll share once we're able to run semi private