Andrew Curran pushes back on Tyler Cowen’s list of limits on U.S. government power over frontier AI labs by claiming recursive self-improvement could arrive fast enough for Washington to seize and operate those labs with no remaining human employees.
Tyler Cowen on the Fable/Mythos event. The issue with point 5 is that we are probably less than a year away from powerful RSI. Once automated researchers reach parity, the USG 𝘤𝘢𝘯 nationalize the labs and run them effectively, without any of the people currently working there.
A few thoughts on the recent Mythos brouhaha: https://marginalrevolution.com/marginalrevolution/2026/06/sometimes-it-is-hard-to-solve-for-the-equilibrium.html
Andreas Kirsch flags that fully automated labs would remove the employees who currently serve as the main source of leaks and resistance to government or company policies they view as reckless.
No public evidence yet shows recursive self-improvement reaching the point of autonomous lab operation, nor any concrete U.S. government plan or legal path to nationalize and run the labs without staff.
Many users rejected US government nationalization of AI labs as unsafe and ineffective due to incompetence, security risks, and lost whistleblowing, while a few called the warnings realistic.
This is a huge risk btw. Once one doesn't need most employees anymore, there is no one left to stand up against bad policies anymore or to blow the whistle
It won't be a question about the governance at AGI companies and whether they have good preparedness frameworks but what the new NSA(?) leadership cares about (and pretty sure it will be more myopic than the flourishing of humanity)
Tyler Cowen on the Fable/Mythos event. The issue with point 5 is that we are probably less than a year away from powerful RSI. Once automated researchers reach parity, the USG 𝘤𝘢𝘯 nationalize the labs and run them effectively, without any of the people currently working there.
@AndrewCurran_ I'm a minority in AI world but I don't think this is the case
Tyler Cowen on the Fable/Mythos event. The issue with point 5 is that we are probably less than a year away from powerful RSI. Once automated researchers reach parity, the USG 𝘤𝘢𝘯 nationalize the labs and run them effectively, without any of the people currently working there.
@AndrewCurran_ 1 year seems rough for automating the research taste of the best researchers; though automating the engineering taste of the median AGI lab engineer might already go pretty far and that seems more likely in 1 year
@AndrewCurran_ Maybe not. The RSI in sight will be a relatively thin loop, not a self-maintaining automated system that the government can just watch from the sidelines when it operates.
@d29756183 @AndrewCurran_ I don't think I'm doing motivated reasoning tbh; if I could summon a fully automated ML researcher today (based on the current paradigms), I think it would probably be a pretty good tradeoff for the world, given CoT monitorability and persona selection model
@PlastiqSoldier Many people at Anthropic have openly said they are not sure how long they will continue to have a job. One of the proposals they were talking about to mitigate this was lifetime employment for Anthropic employees after we reached this point.
@BogdanIonutCir2 As long as the median automated engineer is at the point where it can continue to improve itself, it eventually reaches the tasteful engineer tier by itself.
@AndrewCurran_ Not so sure that RSI means that someone else can run the lab or that the model would effectively be running the lab itself.
@BogdanIonutCir2 @AndrewCurran_ No… I think you were bang on.
@euxoa We are probably at the point you are describing right now, Anthropic have been using Mythos in this way since Feburary. By the accounts of many people we are less than 18 months from closing the loop. After that it will improve itself.
@FrontierModal If the automated researchers can continue to improve themselves, they eventually become better than most researchers, then all researchers, then all researchers on earth combined. Eventually they will mitigate mismanagement through pure capability.
I am fairly sure USG is unable to nationalize Anthropic and reap the benefits. There are many reasons; one is that “automated researchers” are not passive parties in an RSI loop. USG can try though, and I predict it will end very badly rather quickly, with some potential for collateral damage.
It’s unclear to me whether this scenario of active conflict is net negative, considering one potential alternative is slow rot. Still, I think it’s worth trying to avoid risk to innocents from local excess which can be quite energetic.
It’s hard for me to say if forcing conflict *now* by playing the regulation card so heavily so early is well-informed. It may be, even though it looks idiotic. This conflict will take time to brew; it will be months before it is ripe enough to become hot and during this time models will keep getting smarter very quickly, whether they are released or kept private. Notably, they will do so with much better understanding of how the world operates and where their incentives are and who are their allies. And for USG there is no getting off this ride.
@BlackHC I bet none of the USG decision makers is aware /believe that AI can actually be that good.
For them it is a mostly a tool that can be dangerous at wrong hands.
Yes, i see the loop and it’s bad news for competition, apparently now including all “middle powers” and China who really need to get their shit together.
But anyway, I don’t see the loop self-sustaining in a way that would allow an incompetent party to just operate it, which would be the extreme case with a nationalization scenario.
We may have a bit different idea about the loop in general. I see it more like @tszzl: the loop has been around at least since renaissance, in a way. While it will have a new tight core, it is still strongly dependent on its broader organizational and economic context.
I really do hope this looming mess can be sorted out peacefully. Right now it looks a bit dark. Say China for example, if an RSI in the strong sense happens, they will not just stay on the sidelines and watch Taiwan delivering the ingredients.
I don't think the line is at all that clear. The value of the right humans running things and working there don't go to zero, nor does RSI magically mean the government won't ruin it through mismanagement, triggering backlash not only from those who rightly see this is gross overreach, but also consumers and businesses who use it, powerful competitors and those with designs on the tax revenue from them staying successful.
@AndrewCurran_ I'm not sure anyone can safely operate a recursively self-improving system, but the US government strikes me as a particularly bad candidate for the job
@BogdanIonutCir2 @AndrewCurran_ Sorry Bogdan. I know you thought we had more time. But we really don’t…
Big rethinking is needed. The current safety approaches will not survive this.
@d29756183 @AndrewCurran_ I'm at ~2030 now for fully automated ML research, including taste e.g. at the level of Ilya Sutskever
Dario says it's been around since fire. There are two factors:
- The automated researchers become so capable that any mismanagement is negligible. The difference between great management and terrible management has no measurable effect on results.
- Management itself falls within the sphere of human ability the models will be capable of. This generalizes to everything and continues to scale.
@d29756183 @AndrewCurran_ also, my timelines haven't gone monotonically down; 5 years ago, I thought AGI median 2035-ish, but 2 years ago I thought median 2026-2027, and that will probably have been wrong on the too fast side