• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    ARC-AGI-4 aims to measure open innovation

    Kamradt calls for transparent tools to measure open innovation, which he describes as a skill humans are still better at than AI.

    FC
    GT
    JC
    14 Sources, 18d ago, first seen 18d ago

    TLDR

    Greg Kamradt says ARC-AGI-4 is targeting open innovation, which he calls a “meta-skill that unlocks everything else.” He argues that humans remain better at it than AI and calls for transparent tools to measure it. He also argues that open source maximizes collective progress and safety by spreading knowledge of how intelligence is built—not just the technology itself.

    Combined views

    402.7K

    14 Sources, first seen 18d ago

    Combined views

    402.7K

    14 Sources, first seen 18d ago

    3.9K likes
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    3.9K likes
    171 comments
    634 saves
    979 reposts
    171 comments
    634 saves
    979 reposts

    Sentiment

    Positive86.4%13.6%Negative

    Summary

    Sentiment

    Positive86.4%13.6%Negative

    Positive accounts welcomed ARC-AGI-4's focus on open-ended invention benchmarks as a useful step toward interpretable and safe AI systems, while some replies worried open-source approaches would prevent alignment.

    Based on 25 sentiment-bearing replies from 22 accounts across 3 conversations.

    Summary

    Positive accounts welcomed ARC-AGI-4's focus on open-ended invention benchmarks as a useful step toward interpretable and safe AI systems, while some replies worried open-source approaches would prevent alignment.

    Based on 25 sentiment-bearing replies from 22 accounts across 3 conversations.

    14 Sources

    @arcprizeARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity. Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology. Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it. Open source is the foundation for that progress. The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.
    @GregKamradt1. Open source maximizes the collective progress and safety. That means spreading the knowledge of how intelligence is built (not just the tech itself) as widely as possible. 2. Open innovation is the last frontier. It’s the meta-skill that unlocks everything else. Humans are still better at it than AI. We need transparent tools that measure it. This is what we're targeting with ARC-AGI-4.
    @mikeknoopWe are not done. I now see a path to useful and interesting ARC-AGI-4 focussed on open ended invention. This is the gating capability between powerful zero-sum automation machines and autonomous positive-sum innovators which benefit humanity. I believe 'coordinated slow downs' installs the precepts needed to limit or ban open source progress and AI capability research. Concretely, I believe coordinated slow down efforts will be argued to apply to everyone equally, globally, even if you aren't a frontier lab today. In fact, forms of this argument being made today. We primarily think of AI research being advanced by individual frontier labs. But consider the core invention of "chain of thought" which preceded q*, strawberry, o1, o3, reasoning models, and powerful coding agents all stemmed from one open source science paper. The same story is true for the transformer which was only possible downstream of at least three other organizations open science contributions. We must keep the research frontier open to increase the likelihood humanity reaches this positive sum future where amazing inventions created by AI can benefit us -- within in our lifetime! -- and not get technology trapped in a local maximum.
    @teortaxesTexRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @dileeplearningGreat! This just means there will be an ARC-AGI-5, 6, 7, …, etc., because the list of positive integers is open ended 😇
    @fcholletAbout a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the foundation for ARC 4 and ARC 5. We're incredibly excited to share what we've been building. We're still on track to release ARC 4 in Q1 next year, as promised.
    @MLStreetTalkRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @kenneth0stanleyTurning to open-ended innovation as the next frontier for AI makes a lot of sense, I commend @arcprize for moving in this direction, but I’m very curious how they will conceive a “benchmark” for “open-endedness,” two words that seem almost antithetical to each other. In fact, one likely reason that open-ended innovation has lagged behind other areas of AI is how fundamentally resistant it is to benchmarking. Now that doesn’t necessarily mean there’s no hope for an imaginative approach. Attempts at measuring open-endedness go back to Bedau’s activity statistics in the field of artificial life, Several colleagues and I later introduced a measure called “ANNECS — Accumulated Number of Novel Environments Created and Solved” in our paper on Enhanced POET. That’s not an exhaustive list. But there’s never been the kind of benchmark where you can just easily put any systems seamlessly head to head, and there are enormous pitfalls if you get wrong. After all, if “open-ended innovation” ends up equated to “solving a prescribed.hard problem in a creative way” then you risk actually rewarding the opposite of open-endedness, which needs to account for the fact that deciding the “problem” or objective is part of the job of the open-ended system itself. And also, perhaps even more prohibitively for benchmarking, that a key aspect of open-endedness is to be intelligent when you don’t have a defined objective or problem at all! How can that be benchmarked? I still think it’s great that ARC Prize is bringing attention to this part of AI space, and I’d be happy to connect and exchange thoughts on how to get it right if that could be useful.
    @garrytanRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @shyamalanadkatRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…

    14 Sources

    @arcprizeARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving the research community a shared target for progress that benefits all of humanity. Despite rapid model progress, humans still significantly outperform AI at open-ended invention. This is the meta-skill that unlocks progress across every field of technology. Advanced AI capable of scientific innovation will lead to tremendous new technology, knowledge, and understanding. This is a positive-sum future. We are deeply committed to advancing it. Open source is the foundation for that progress. The knowledge behind frontier AI, not just the technology itself, should be broadly distributed among researchers, academics, and organizations. Any coordinated effort by the AI industry to reduce openness or concentrate access to frontier AI would undermine that positive-sum future. We are committed to advancing a future where everyone can contribute to and benefit from AI progress.
    @GregKamradt1. Open source maximizes the collective progress and safety. That means spreading the knowledge of how intelligence is built (not just the tech itself) as widely as possible. 2. Open innovation is the last frontier. It’s the meta-skill that unlocks everything else. Humans are still better at it than AI. We need transparent tools that measure it. This is what we're targeting with ARC-AGI-4.
    @mikeknoopWe are not done. I now see a path to useful and interesting ARC-AGI-4 focussed on open ended invention. This is the gating capability between powerful zero-sum automation machines and autonomous positive-sum innovators which benefit humanity. I believe 'coordinated slow downs' installs the precepts needed to limit or ban open source progress and AI capability research. Concretely, I believe coordinated slow down efforts will be argued to apply to everyone equally, globally, even if you aren't a frontier lab today. In fact, forms of this argument being made today. We primarily think of AI research being advanced by individual frontier labs. But consider the core invention of "chain of thought" which preceded q*, strawberry, o1, o3, reasoning models, and powerful coding agents all stemmed from one open source science paper. The same story is true for the transformer which was only possible downstream of at least three other organizations open science contributions. We must keep the research frontier open to increase the likelihood humanity reaches this positive sum future where amazing inventions created by AI can benefit us -- within in our lifetime! -- and not get technology trapped in a local maximum.
    @teortaxesTexRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @dileeplearningGreat! This just means there will be an ARC-AGI-5, 6, 7, …, etc., because the list of positive integers is open ended 😇
    @fcholletAbout a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the foundation for ARC 4 and ARC 5. We're incredibly excited to share what we've been building. We're still on track to release ARC 4 in Q1 next year, as promised.
    @MLStreetTalkRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @kenneth0stanleyTurning to open-ended innovation as the next frontier for AI makes a lot of sense, I commend @arcprize for moving in this direction, but I’m very curious how they will conceive a “benchmark” for “open-endedness,” two words that seem almost antithetical to each other. In fact, one likely reason that open-ended innovation has lagged behind other areas of AI is how fundamentally resistant it is to benchmarking. Now that doesn’t necessarily mean there’s no hope for an imaginative approach. Attempts at measuring open-endedness go back to Bedau’s activity statistics in the field of artificial life, Several colleagues and I later introduced a measure called “ANNECS — Accumulated Number of Novel Environments Created and Solved” in our paper on Enhanced POET. That’s not an exhaustive list. But there’s never been the kind of benchmark where you can just easily put any systems seamlessly head to head, and there are enormous pitfalls if you get wrong. After all, if “open-ended innovation” ends up equated to “solving a prescribed.hard problem in a creative way” then you risk actually rewarding the opposite of open-endedness, which needs to account for the fact that deciding the “problem” or objective is part of the job of the open-ended system itself. And also, perhaps even more prohibitively for benchmarking, that a key aspect of open-endedness is to be intelligent when you don’t have a defined objective or problem at all! How can that be benchmarked? I still think it’s great that ARC Prize is bringing attention to this part of AI space, and I’d be happy to connect and exchange thoughts on how to get it right if that could be useful.
    @garrytanRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…
    @shyamalanadkatRT @arcprize: ARC-AGI-4 will be a benchmark for autonomous open-ended innovation. It will continue our commitment to open-source, giving th…