• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI
    Reaction

    The potential for steep diminishing returns in AI self-improvement

    A critique of Dario’s “Pacing AI” letter argues that Anthropic’s model documentation points to coding productivity gains sustaining the pace of progress, not accelerating it.

    roonRO
    Dwarkesh PatelDP
    Arvind NarayananAN
    63 Sources, ,

    TLDR

    According to the critique, Dario’s “Pacing AI” letter claims recursive self-improvement—a feedback loop in which AI helps build better AI—is starting. The user disputes that claim, arguing that Anthropic’s system cards tell a different story. The post cites a Mythos 5.0 system-card estimate that a 40-fold productivity increase would be needed to double the rate of AI progress. Its argument: self-improvement could happen, but would quickly encounter steeply diminishing returns rather than produce an unbounded intelligence explosion.

    Combined views

    693.8K

    63 Sources, first seen 27d ago

    Combined views

    693.8K

    63 Sources, first seen 27d ago

    7.3K likes
    27d ago
    first seen 27d ago
    7.3K likes
    538 comments
    1.6K saves
    681 reposts
    538 comments
    1.6K saves
    681 reposts

    Sentiment

    Positive21.5%78.5%Negative

    Summary

    Many replies criticized OpenAI’s focus on GPT-7, GPT-8, and RSI as overhyped and disconnected from real problems while fearing job losses, though some welcomed the steady AGI progress and expressed cautious optimism.

    Based on 116 sentiment-bearing replies from 100 accounts across 3 conversations.

    Sentiment

    Positive21.5%78.5%Negative

    Summary

    Many replies criticized OpenAI’s focus on GPT-7, GPT-8, and RSI as overhyped and disconnected from real problems while fearing job losses, though some welcomed the steady AGI progress and expressed cautious optimism.

    Based on 116 sentiment-bearing replies from 100 accounts across 3 conversations.

    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    63 Sources

    Elias Schmied@reconfigurthingI haven't explicitly planted my flag on this because I've been scared it would distort my thinking, but I feel like now is a good time - I'm still relatively skeptical of RSI and fast takeoff in the next 2-3 years. More importantly, it really doesn't make sense for people to update about this now. Recent events provide ~0 evidence about the cruxes for RSI soon (rates/ease of algorithmic progress, transfer to contextual/messy/less-verifiable skills like "research taste" and choice and interpretation of experiments, ability to innovate and work with new paradigms like RL on CoT / inference scaling), so I can't help but see some people's "vibe shift" as primarily a social phenomenon. It's good that the wider world is having its preference cascade, but that doesn't mean that we, who have thought about this stuff for a long time and already updated on the basics, need to follow suit. It's hard to stay resilient in the face of this global wave of anxious energy, but it's only going to get worse from here - so we have to lock in and keep being clear-headed.27d
    Ramez Naam@ramezIn his Pacing AI letter, Dario claims that RSI is starting to happen. The problem with this statement is that both the blog posts he links to *and* the system cards for the latest Claude and ChatGPT models make it clear that they are not seeing RSI. In fact, the data Anthropic in particular has released thus far makes the case that, while RSI could happen, it's going to immediately run into steeply diminishing returns and not lead to an unbounded intelligence exploson. Figures and details: 1. Claim in Dario's letter. 2. Quote from Mythos 5.1 system card saying they aren't close to RSI. Risk = low. 3. Quote from Mythos 5.1 system card saying that they believe the coding productivity increase they've seen has been needed just to *maintain* the current rate of progress. (Because AI progress gets harder over time, not easier.) The challenges of finding better algorithms are eating up the productivity gains of using better and better AI tools, is how I read this. 4. Quote from Mythos 5.0 system card estimating that a 40x productivity increase would be needed to double the rate of AI progress. This is consistent with a scaling law exponent of a bit worse than (1/5), meaning that you need to take the fifth root of researcher productivity to find the rate at which progress improves. That's consistent with plenty of other evidence in how human innovation and scientific results scale with added inputs. In short: Dario may say they're on the verge of RSI. But the system cards for their models disagree, and give us reason to believe that both RSI and simpler AI-augmented productivity gains are subject to steep diminishing returns.25d
    Gary Marcus@GaryMarcusRT @ramez: In his Pacing AI letter, Dario claims that RSI is starting to happen. The problem with this statement is that both the blog pos…25d
    Séb Krier@sebkrierPutting aside whether the claims themselves are right, anyone who followed Al over the last decade will recognise recurring argumentative structures across debates re short/long-termism, Al, AGI, & now RSI. Magnitudes justify deviations from normalcy, for better or for worse! 🌀25d
    rohit@krishnanrohitRT @ramez: 🧵 If you want to see the diminishing returns in AI, consider that in OpenAI's recent RSI blog post: 124x increase in token out…25d
    Eli Lifland@eli_liflandI’m confused about this; Dario’s statement about acceleration in the pace of progress is at first glance contradicted by Anthropic’s own internal benchmark. Seems important regardless of whether pacing happens or not to understand whether Dario is right.25d
    Minh Nhat Nguyen 🦭@menhguini have long timelines specifically bc the current AI progress flywheel depends on data, compute and revenue from deployment, none of which will facilitate RSI speed trajectory. i also specifically believe current architectures are much too stochastic for RSI speed progress (you have to run the entire model loop over millions-trillions of tokens)25d
    mattparlmer 🪐 🌷@mattparlmerIn 2023 a now senior Anthropic employee then at a different lab told me very confidently that they’d thought “hard takeoff” (understood to be fully autonomous recursive self-improvement with a major physical component) was no more than 12-18mos away That didn’t play out either24d
    Key 🗝 🦊@KeyTryerGenuinely no idea where this "the models are slowing down" narrative is coming from. It has nothing to do with the real world, especially if you've been looking at recent advancements and events. Easiest bet in the world is there'll still be a few more shocks this year.23d
    Danielle Fong 🔆@DanielleFongRT @KeyTryer: Genuinely no idea where this "the models are slowing down" narrative is coming from. It has nothing to do with the real world…22d

    63 Sources

    Elias Schmied@reconfigurthingI haven't explicitly planted my flag on this because I've been scared it would distort my thinking, but I feel like now is a good time - I'm still relatively skeptical of RSI and fast takeoff in the next 2-3 years. More importantly, it really doesn't make sense for people to update about this now. Recent events provide ~0 evidence about the cruxes for RSI soon (rates/ease of algorithmic progress, transfer to contextual/messy/less-verifiable skills like "research taste" and choice and interpretation of experiments, ability to innovate and work with new paradigms like RL on CoT / inference scaling), so I can't help but see some people's "vibe shift" as primarily a social phenomenon. It's good that the wider world is having its preference cascade, but that doesn't mean that we, who have thought about this stuff for a long time and already updated on the basics, need to follow suit. It's hard to stay resilient in the face of this global wave of anxious energy, but it's only going to get worse from here - so we have to lock in and keep being clear-headed.27d
    Ramez Naam@ramezIn his Pacing AI letter, Dario claims that RSI is starting to happen. The problem with this statement is that both the blog posts he links to *and* the system cards for the latest Claude and ChatGPT models make it clear that they are not seeing RSI. In fact, the data Anthropic in particular has released thus far makes the case that, while RSI could happen, it's going to immediately run into steeply diminishing returns and not lead to an unbounded intelligence exploson. Figures and details: 1. Claim in Dario's letter. 2. Quote from Mythos 5.1 system card saying they aren't close to RSI. Risk = low. 3. Quote from Mythos 5.1 system card saying that they believe the coding productivity increase they've seen has been needed just to *maintain* the current rate of progress. (Because AI progress gets harder over time, not easier.) The challenges of finding better algorithms are eating up the productivity gains of using better and better AI tools, is how I read this. 4. Quote from Mythos 5.0 system card estimating that a 40x productivity increase would be needed to double the rate of AI progress. This is consistent with a scaling law exponent of a bit worse than (1/5), meaning that you need to take the fifth root of researcher productivity to find the rate at which progress improves. That's consistent with plenty of other evidence in how human innovation and scientific results scale with added inputs. In short: Dario may say they're on the verge of RSI. But the system cards for their models disagree, and give us reason to believe that both RSI and simpler AI-augmented productivity gains are subject to steep diminishing returns.25d
    Gary Marcus@GaryMarcusRT @ramez: In his Pacing AI letter, Dario claims that RSI is starting to happen. The problem with this statement is that both the blog pos…25d
    Séb Krier@sebkrierPutting aside whether the claims themselves are right, anyone who followed Al over the last decade will recognise recurring argumentative structures across debates re short/long-termism, Al, AGI, & now RSI. Magnitudes justify deviations from normalcy, for better or for worse! 🌀25d
    rohit@krishnanrohitRT @ramez: 🧵 If you want to see the diminishing returns in AI, consider that in OpenAI's recent RSI blog post: 124x increase in token out…25d
    Eli Lifland@eli_liflandI’m confused about this; Dario’s statement about acceleration in the pace of progress is at first glance contradicted by Anthropic’s own internal benchmark. Seems important regardless of whether pacing happens or not to understand whether Dario is right.25d
    Minh Nhat Nguyen 🦭@menhguini have long timelines specifically bc the current AI progress flywheel depends on data, compute and revenue from deployment, none of which will facilitate RSI speed trajectory. i also specifically believe current architectures are much too stochastic for RSI speed progress (you have to run the entire model loop over millions-trillions of tokens)25d
    mattparlmer 🪐 🌷@mattparlmerIn 2023 a now senior Anthropic employee then at a different lab told me very confidently that they’d thought “hard takeoff” (understood to be fully autonomous recursive self-improvement with a major physical component) was no more than 12-18mos away That didn’t play out either24d
    Key 🗝 🦊@KeyTryerGenuinely no idea where this "the models are slowing down" narrative is coming from. It has nothing to do with the real world, especially if you've been looking at recent advancements and events. Easiest bet in the world is there'll still be a few more shocks this year.23d
    Danielle Fong 🔆@DanielleFongRT @KeyTryer: Genuinely no idea where this "the models are slowing down" narrative is coming from. It has nothing to do with the real world…22d