ARC Prize says plotting model scores reveals the pattern; another post calls evaluation science nascent despite its importance.
ARC Prize says plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. Another post sharing that observation says the science of evaluations is still nascent, given how important they are.
2.3K
3 Sources, first seen ago
A user says ARC Prize has confirmed it will evaluate the model. They argue a score below 80% would be disappointing, citing V4-Flash-0731’s 61.4% on ARC-AGI-2.
A user calls GLM 5.3 Flash a strong model but says they're still waiting for V4.1 Flash. They quip that Pro might arrive first.
A user says ARC Prize has confirmed it will evaluate the model. They argue a score below 80% would be disappointing, citing V4-Flash-0731’s 61.4% on ARC-AGI-2.
A user calls GLM 5.3 Flash a strong model but says they're still waiting for V4.1 Flash. They quip that Pro might arrive first.
@GregKamradtPlotting v1 vs v2 scores show a smooth progression V2 was able to extend the usefulness of the original arc agi format at higher model levels In fact we could probably get a difficulty per task at this point with all the data we have If we did a v2.5 (we have no plans to) we could probably extend this out a bit longer
@fcholletRT @arcprize: Plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. For example, the power curve v2 = v…
@mikeknoopThe science of evals is quite nascent given how important they are@GregKamradtPlotting v1 vs v2 scores show a smooth progression V2 was able to extend the usefulness of the original arc agi format at higher model levels In fact we could probably get a difficulty per task at this point with all the data we have If we did a v2.5 (we have no plans to) we could probably extend this out a bit longer@fcholletRT @arcprize: Plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. For example, the power curve v2 = v…@mikeknoopThe science of evals is quite nascent given how important they are@GregKamradtPlotting v1 vs v2 scores show a smooth progression V2 was able to extend the usefulness of the original arc agi format at higher model levels In fact we could probably get a difficulty per task at this point with all the data we have If we did a v2.5 (we have no plans to) we could probably extend this out a bit longer@fcholletRT @arcprize: Plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. For example, the power curve v2 = v…@mikeknoopThe science of evals is quite nascent given how important they are@GregKamradtPlotting v1 vs v2 scores show a smooth progression V2 was able to extend the usefulness of the original arc agi format at higher model levels In fact we could probably get a difficulty per task at this point with all the data we have If we did a v2.5 (we have no plans to) we could probably extend this out a bit longer@fcholletRT @arcprize: Plotting ARC-AGI model scores reveals a strong relationship across all three benchmarks. For example, the power curve v2 = v…@mikeknoopThe science of evals is quite nascent given how important they are#5
#5
