Reactions from ranked influencers
3 posts
"We fixate on AGI gods, while more mundane problems pile up unattended." Excellent essay on how intelligence is not the main bottleneck and the monoculture of AGI believers by @RuxandraTeslo "At a recent dinner, someone from an AI lab asked me, quite bluntly , why I do what I do — why I write about the regulatory bottlenecks to medical progress, why I spend my time on policies related to clinical trials. He looked at me with something best summarized as pity. I told him what I believe: that in the age of AI, medicine will be bottlenecked more than ever by regulation and the grind of clinical trials, and that billions poured into faster pre-clinical research won’t touch that problem, unsexy as it is. I brought up housing: we’ve had the technology to build better housing for decades, yet it’s more expensive than ever, because housing is a question of political will, not pure capability. He looked at me incredulously. Surely, a smart person like me should know that AI, or better said, AGI will be hyperpersuasive soon – already on a bunch of benchmarks it exceeds professional debaters at persuasion. I said, “Hmm.” He said, “Yes, yes”, with the undertone of a man who knows things deeply, things that mere mortals like me, not being AI lab employees, are simply not well placed to grasp. And he looked at me again with that sense of pity, the way one looks at a slightly mentally impaired but cute animal awaiting its imminent slaughter." "Back when I was an academic, I often noticed that academics were, on the whole, deeply progressive, yet thought of themselves as contrarians. This was at least in part because most had come from places more conservative than the university. They’d gather, compare themselves to the outside world rather than to each other, and conclude that their progressivism was somehow brave, despite it being the plain consensus of the room. In my view, SF has done the same thing with AI. In elite tech circles it is simply not contrarian to take AGI seriously; if anything the reverse. But because the comparison class is always someone else: for example, a policy person in DC, the feeling of embattled dissent survives contact with reality." https://www.writingruxandrabio.com/p/intelligence-is-not-the-main-bottleneck
Rux is correct!
In certain tech circles, it has become low-status to suggest that AGI won't automatically solve everything. I think models will continue to get smarter, but intelligence is often not the main bottleneck to physical world change. More below 👇 https://www.writingruxandrabio.com/p/intelligence-is-not-the-main-bottleneck Some points the article makes: 1. Intelligence is often not the main bottleneck Increasingly, when I talk to people at AI labs, I keep hearing the same thing: that working on regulatory barriers to the diffusion of medical innovation is a waste of my time. AGI will solve all of that, they assure me, with the pitying look one might give a naive hamster awaiting its slaughter. I think they're wrong. We've had the technology to build better housing for decades, yet it's more expensive than ever: this is because housing is a question of political will, not just capability. Something similar goes for medicine. 2. This is just a microcosm of a broader problem: a dismisal of institutional and cultural barriers, as well as the mundane and the human. The regulatory barriers to AI diffusion in medicine is just one example of the kind of stuff that tends to be dismissed by the AGI-pilled people. I think in general, there is an underweighting of cultural and institutional shaped problems, as well as of the boring and mundane. Maybe this has to do with the "totalizing' feeling of being close to reaching the AGI-God: It becomes hard to care about other things. But to me, problems like a decline in reading or the slop-ification of our minds due to these technologies are equally important. They do not seem to be seen as equally serious. 3. This has become a monoculture. Certain SF tech circles (not everyone) have hardened into something close to a monoculture, where it's now genuinely low-status to suggest AGI might not be wholly transformative on its own. The people who were early and right about AI made fortunes. Being right about this one big thing has created a sort of "halo effect" where many think they're right about everything. To question the most extreme claims about the effects on the real world often means you will be marked as dim, low-IQ, insufficiently "AGI-pilled." This often results in self-censorship. Other very smart tech people privately acknowledge the bottlenecks to real-world diffusion, but do not really talk about it. Because the outside world has long been skeptical of AI (and continues to be), many in these circles have come to see themselves as dissenters. But I think AGI-maximalism is the safe and status-conferring view within their circles. 4. The real world has many barriers to progress -- more detail. The "real world" has many barriers to diffusion that those who think intelligence is all that matters seem to ignore. I remember a Progress Conference in 2023 where the economist Tyler Cowen made exactly this argument to a room full of AGI-pilled attendees, many of whom were frankly incredulous. The capabilities, he suggested, would arrive faster than the changes they were meant to produce; GDP growth would disappoint relative to the hype, and there would be no massive job displacement. So far, the data seem to bear him out. I can speak about my own area, which is medical progress, which is often bottlenecked by institutional and regulatory or policy barriers and also by the sheer difficulty of producing experimental biological data and the inherent limits of biology. One example, beyond my usual clinical trials points: our patent system rewards novel chemical matter much more than novel biology (finding new targets). This means whoever validates a new target bears the risk but finds it hard to capture the reward. The result is "target herding": a lot of drugs against the same de-risked pathways, meaning a lot of biological space goes underexplored. Some of best-funded AI-bio companies (Chai Discovery, Isomorphic Labs) target molecule optimization, so the tractable part, not the frontier of finding new targets. It turns out even AI companies respond to legal and economic incentives! Does this matter? I think so... We fixate on AGI gods, while more mundane problems pile up unattended. The social fabric is fraying in plain sight. More and more young adults cannot afford a house or sustain a relationship. People pour their youth into gambling apps, some of them now made more powerful by the very AI we keep discussing. People are de-skilling as they hand more thinking to machines. Every one of these problems needs people working on it. Being in the thrall of digital deities that will magically dissolve all our problems pulls attention away from the simpler, more human troubles that are already here. Note: I want to mention there was an earlier version of this X post that got a lot of reactions. I deleted it, because it became quite controversial and it was interfering with my work. One common rebuttal was that I am "making up a person to be mad at". I have modified my X post now to capture more of the nuance that I hope is found in the original article. For example, not everyone in SF thinks this and my new post makes that more clear. The X post is a summary though and I encourage people to read the whole article. This is also a way to start a conversation, not a definitive statement.
In certain tech circles, it has become low-status to suggest that AGI won't automatically solve everything. I think models will continue to get smarter, but intelligence is often not the main bottleneck to physical world change. More below 👇 https://www.writingruxandrabio.com/p/intelligence-is-not-the-main-bottleneck Some points the article makes: 1. Intelligence is often not the main bottleneck Increasingly, when I talk to people at AI labs, I keep hearing the same thing: that working on regulatory barriers to the diffusion of medical innovation is a waste of my time. AGI will solve all of that, they assure me, with the pitying look one might give a naive hamster awaiting its slaughter. I think they're wrong. We've had the technology to build better housing for decades, yet it's more expensive than ever: this is because housing is a question of political will, not just capability. Something similar goes for medicine. 2. This is just a microcosm of a broader problem: a dismisal of institutional and cultural barriers, as well as the mundane and the human. The regulatory barriers to AI diffusion in medicine is just one example of the kind of stuff that tends to be dismissed by the AGI-pilled people. I think in general, there is an underweighting of cultural and institutional shaped problems, as well as of the boring and mundane. Maybe this has to do with the "totalizing' feeling of being close to reaching the AGI-God: It becomes hard to care about other things. But to me, problems like a decline in reading or the slop-ification of our minds due to these technologies are equally important. They do not seem to be seen as equally serious. 3. This has become a monoculture. Certain SF tech circles (not everyone) have hardened into something close to a monoculture, where it's now genuinely low-status to suggest AGI might not be wholly transformative on its own. The people who were early and right about AI made fortunes. Being right about this one big thing has created a sort of "halo effect" where many think they're right about everything. To question the most extreme claims about the effects on the real world often means you will be marked as dim, low-IQ, insufficiently "AGI-pilled." This often results in self-censorship. Other very smart tech people privately acknowledge the bottlenecks to real-world diffusion, but do not really talk about it. Because the outside world has long been skeptical of AI (and continues to be), many in these circles have come to see themselves as dissenters. But I think AGI-maximalism is the safe and status-conferring view within their circles. 4. The real world has many barriers to progress -- more detail. The "real world" has many barriers to diffusion that those who think intelligence is all that matters seem to ignore. I remember a Progress Conference in 2023 where the economist Tyler Cowen made exactly this argument to a room full of AGI-pilled attendees, many of whom were frankly incredulous. The capabilities, he suggested, would arrive faster than the changes they were meant to produce; GDP growth would disappoint relative to the hype, and there would be no massive job displacement. So far, the data seem to bear him out. I can speak about my own area, which is medical progress, which is often bottlenecked by institutional and regulatory or policy barriers and also by the sheer difficulty of producing experimental biological data and the inherent limits of biology. One example, beyond my usual clinical trials points: our patent system rewards novel chemical matter much more than novel biology (finding new targets). This means whoever validates a new target bears the risk but finds it hard to capture the reward. The result is "target herding": a lot of drugs against the same de-risked pathways, meaning a lot of biological space goes underexplored. Some of best-funded AI-bio companies (Chai Discovery, Isomorphic Labs) target molecule optimization, so the tractable part, not the frontier of finding new targets. It turns out even AI companies respond to legal and economic incentives! Does this matter? I think so... We fixate on AGI gods, while more mundane problems pile up unattended. The social fabric is fraying in plain sight. More and more young adults cannot afford a house or sustain a relationship. People pour their youth into gambling apps, some of them now made more powerful by the very AI we keep discussing. People are de-skilling as they hand more thinking to machines. Every one of these problems needs people working on it. Being in the thrall of digital deities that will magically dissolve all our problems pulls attention away from the simpler, more human troubles that are already here. Note: I want to mention there was an earlier version of this X post that got a lot of reactions. I deleted it, because it became quite controversial and it was interfering with my work. One common rebuttal was that I am "making up a person to be mad at". I have modified my X post now to capture more of the nuance that I hope is found in the original article. For example, not everyone in SF thinks this and my new post makes that more clear. The X post is a summary though and I encourage people to read the whole article. This is also a way to start a conversation, not a definitive statement.