Researcher Warns Mythos Accelerates AI Toward Superintelligence

Am I personally annoyed losing access to Fable? Yes, I’m super annoyed!

I’ve been building basically nonstop since it came out. Biggest shift I’ve experienced using AI. And also this doesn’t matter much. Getting superintelligence right, not losing control of AI, is what matters!

Some additional context: In the last few days I’ve joked that I have “AI mania”, because I can’t stop using Claude Code. I’ve been walking around, sending voice notes to direct my Fable cloud agent I built with Fable. I brought out my laptop during date night (sorry @collegraphy)

I think it’s also good to consider the cyber and bio capabilities of the model. If I were running the admin’s AI efforts, I’d greatly expand CAISI and the NSA AI center and task them with evaluating all of these risks in depth and publishing most of the findings.

I don’t think it’s bad for an AI agent to be that useful. I think it’s good in its first-order effects. But it’s scary because people will become dependent. And they will be very upset if they lose access to their agent. If the admin had waited another week…

@JeffLadish Do you mean AGI that can run on very modest hardware resources? Or AI improvements at the software layer allowing labs to train better and better models at equal compute?

I’d also work with third party auditors and the labs, including asking labs to evaluate each other’s safety plans and safeguards.

Transparency here would go a long way!

I think the uproar right now would be much larger. And these are just today’s capabilities. What’s it going to be like a year from now?

I don’t have a clear takeaway or recommendation from this. It’s just wild to notice the feelings in myself and extrapolate.

I walked past my coworker in the hall on Friday and just whispered “agents agents agents”. I started rethinking how to manage my company’s task and document systems. I found using Fable incredibly useful, fun, and addictive.

And I find that scary.

@zetalyrae I think a software only intelligence explosion is likely possible

@JeffLadish You know I've been thinking about this, what do you think are the big vectors of RSI? It seems the capital expenditure and operating costs of the datacenters is a huge limiting factor the AI can't move.

@JeffLadish on the bright side, think of how many "but babe what if it gets export controlled" excuses you have for claude coding on future date nights ❤️

People’s emotions are valid. It’s super annoying to lose access to such an incredible tool.

And I’m not saying the admin’s actions were good. I don’t understand the threat model they were considering, and I don’t think export controls make sense if the main threat was cyber.

But I try not to lose track of the plot. Superintelligence is not a cyber capability or a bio capability. It’s the end of the game.

Another example: I made this meme 1 shot while driving by sending a voice note to my fable agent (to grab a picture of the original meme and then use Image 2 api to make the new version)

@collegraphy Pair programming date night? 🥺

@zetalyrae Yeah, both. I think the latter will lead to the former

@JeffLadish Yeah, more means to do more stuff faster will help everybody, including bad actors. Then they can spend more time on logic AI refuses to do.

Then of course smarter AI without bio abilities could still be unpredictable enough.

We'll have to manage.

@zetalyrae @JeffLadish Both seem possible.

@JeffLadish @collegraphy I think this is exactly the problem. When a tool feels this useful, speed-up risk stops looking abstract very quickly.

@JeffLadish I'll play devil's advocate. Near-term harms are concrete. But the asymptote's the real problem. Slowing global R&D takes coordination no one's built.