Some users supported critiques of AI 2040 scenarios for unrealistic regulatory assumptions while others objected to rules that could block personal open-model research such as cancer treatments.
Based on 2 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@sriramk @krishnanrohit So if someone does “research” to find a cancer treatment for a family member on their own laptop with an open weight model, who sends the SWAT team to their house to kick down the door?
> If any of the even *underlying problems* don't play out (we aren't in fact getting personal Dyson spheres), then the entire approach dramatically consolidates power and creates a much worse world. Why do you think this? It seems like even if AI stalls out, total research transparency + access reduce concentration of power. Maybe you just think it's better to avoid approximately any government involvement and you think this is plausible/feasible? Also, it seems to me like any one of the underlying problems (Can people maliciously secretly align the AIs to their interests? Can anyone align the AIs? How do we handle extremely rapid tech progress? How do we handle a world where human labor is mostly obsolete? How do we avoid WW3 due to AI?) makes Plan A look decent (though other options open up as you eliminate some of these problems). > AI 2040 presumes a level of state capacity and sagacity For this recommendation to happen, I agree that goverments would need to be more reasonable than we expect. However, I think that if USG in practice commited hard to making Plan A happen, this would be good (though it might result in something different happen as they adapt to the situation). I agree the state capacity issue is rough, but sometimes to do good stuff, you need capacity. And it doesn't seem impossible if USG thinks of AI as by far the most important issue. > "Prove that this ASI will behave the way we want" is not a coherent sentence, at least to me, its "colourless green ideas sleep furiously". I don't understand? We just think that it's good to gain as much assurance as we can (within available time/constrants) that the AIs aren't seriously misaligned and will do a good job managing the situation. What we say is "Whereas before the burden was on the skeptic to explain why something might fail, now the burden is on the companies to explain why their development is safe. Governments require companies to write up detailed arguments for why their new AIs won’t cause an irreversible catastrophe. These arguments, known as “safety cases,” need to withstand criticism from the public, the scientific community, government auditors, and rival AI companies." > Also, we've had quasi-pauses a few times now, for one reason or another. It's been several years now that there have been extraordinarily well funded firms. Do we have a "provably good way to control Mythos"? We don't. Safety isn't something you figure out in abstract, iterative deployment is the only way it could be done, which is pretty similar to now! The plan does involve interative deployment and experimenting on increasingly capable AIs. (So long as we have reasonable assurance further scaling can be done without causing existential catastrophe.) We keep scaling up capabilities until AIs are matching or exceeding top human experts and stop there for a while so we can get good enough assurance about the alignment and properties of these AIs. We don't talk about "provably good way"? > What's "Total Research Transparency"? Do I get to see all of how Anthropic's research works? [...] The details here are what matters, because those are the agreements that get drawn up and signed. We explain this in the Transparency Plan (https://ai-2040.com/supplements/transparency-plan). More generally, we give a ton of detail on the exact proposals in the supplements. Obviously we don't figure out everything (this is impossible to do in advance) but there is a quite a bit of work on what the details of the deal could look like. > Why would govt choosing who can train frontier models get to a better place? Our regulatory proposal isn't this, maybe you should read the section on frontier AI regulation? > I would really really like a perspective on "what if we're all wrong". I know that's not the point, but if you're making policy recommendations at least one such balanced view would be useful. Have you read the Plan A Assumption supplement? (https://ai-2040.com/supplements/plan-a-assumptions) It goes into a bunch of this. More generally, we discuss a bunch of failure modes and ways we could be wrong in other supplements (and alternatives etc). > We actually do need more rigour, and operationalisation, and concreteness in scenarios. That would be great. This just isn't it. To achieve rigour it's not enough to show more numbers, but we'd be better served to work through one, specific, scenario in enough detail that it passes muster - whether that's geopolitical negotiations or HW chip tracking and impact on industry or any of the hundred other specific levers that are being discussed in the report. As in, you wish the main body of the scenario had more detail? Or that we had a supplement going through a detailed scenario of how the geopolitical negotiations for Plan A might go? We might be able to do this. A lot of the rigour, operationalization, and concreteness lives in the supplements. The scenario main text is kept relatively small so it's possible to understand quickly and then there are a bunch of ways to dive into more detail.
> you're left with an extremely centralised panopticon Huh? It's an extremely decentralized situation (anyone can look), though I agree transparency is very high? > State capacity isn't scalar. USG can quite easily be good at export controls and bad at tech evals. Agree with this; on the specific example, due to transparency they shouldn't **need** to be good at tech evals and can depend on third parties (see the example regulatory proposal for instance). (It still helps if there is better state capacity here ofc.) > I'm not sure what we're learning even today from the existing model cards beyond building the muscle to write model cards For current models, it's still straightforward to give reasonable assurance based on limited capabilities. (So I agree largely just pactice now.) I don't think model cards are the type of artifact that this would be, more like Anthropic sabotage risk reports or the METR frontier risk report. >Which I think its unworkable, extreme, and dangerous, and will only serve to kill AI industry or put it in a sort of soviet/china style grip. I don't get why this would be desirable if the goal is to have competition to get more frontier models and/or especially risk from RSI if it's even more centralised? It will be possible to run an AI company without needing a bunch of private IP. In fact, it will probably be much easier to enter the industry. I agree current AI companies would lose most of their valuation from this proposal and that might suffice to make it politically infeasible in practice. Not sure why you think this differentially benefits China; seems just as easy for US companies to release great products and there will be institutional practices and other institutional knowledge to compete on even if code is available. Generally, it seems like private IP causes centralization, not the opposite? Note that in practice, we think that keeping code private is likely very difficult, so China will have the algos/etc by default (if they want them). The main delta would just be access for the public and other US companies. BTW, you might prefer the Filtered Transparency proposal we discuss in that supplement. We initially went with a proposal where we tried to prevent algorithmic secrets from reaching covert projects, but then we judged this was (1) probably infeasible given realistic execution and (2) made the deal less stable because the public etc wouldn't be able to see what was going on. And deal collapse risk seemed more worrying than algo exfiltration. > create a whole bunch of political problems alongside which aren't addressed. We might do some future work discussing this sort of thing, if there was a concrete list that people agreed were the biggest problems to investigate in more detail, doing this is feasible.
@krishnanrohit Agree with a bunch of this.
Critics argue the superintelligence blueprint relies on implausible international cooperation.
Some users supported critiques of AI 2040 scenarios for unrealistic regulatory assumptions while others objected to rules that could block personal open-model research such as cancer treatments.
Based on 2 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.