/Tech10h ago

Google DeepMind's Andreas Kirsch argues open-source AI models must face regulation as they approach frontier capabilities

Other researchers proposed upstream data regulation and security benchmarks.

1643122.7K

#271

Original post

Andreas Kirsch 🇺🇦@BlackHC#271inTech

A clarification: I'm not against open-source AI at all, yet I do think regulation will be necessary as open-source capabilities approach those of frontier models such as Mythos and beyond (sorry!)

Open-source models have a valuable place to democratize access to automation and create value for everyone

But that doesn't mean that I'm not concerned about abuse and related severe risks, or that I'm naive about the alternatives to regulating these models: more general surveillance and monitoring - see eg the recent open letter calling for regulating/tracking synthetic DNA. (If open and local models that are available to everyone aren't safe and can't easily be monitored, monitoring and surveillance will move to a different layer in society)

There is also nice existential risk argument in *favor* of open-source models: they can slow down the frontier by squeezing the margins and reducing the willingness for additional $$$ research compute. If open-source models are good enough for value creation in sufficiently many use cases, they will become favored over closed models. This might slow down frontier research by itself slightly and give us a needed breather

Medium term to prepare for regulation, we should make sure that open-source models are not trained on data that is related to risk areas. If a model can still reason it's way to risks using a ton of test-time compute/token and experiments, that is a trade-off and might well be okay if it takes long enough and is expensive enough

Interesting research questions are about creating filtered datasets and researching on how to make sure that models cannot be abliterated and don't have plasticity in areas we don't want to (eg if the model bad at biorisiks, it should stay that way even if we try to pre-train it some more or fine-tune it). Can we do that?

(As always: my own opinion and not on behalf of GDM or G...)

9:13 AM · Jun 29, 2026 · 1.8K Views

Sentiment

Users sympathize with regulating open-source AI nearing frontier capabilities because it aligns with impressive advances like DNA applications.

Pos

100.0%

Neg

0.0%

1 comments with sentiment.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Posts from X

Most Activity

VIEWS373LIKES10REPLIES3

Florian Brand@xeophon

@BlackHC Just train no model on risk-related data, why restrict it to open ones?

Closed models can similarly be jailbroken and misused when they are trained on such data

Andreas Kirsch 🇺🇦@BlackHC

A clarification: I'm not against open-source AI at all, yet I do think regulation will be necessary as open-source capabilities approach those of frontier models such as Mythos and beyond (sorry!)

Open-source models have a valuable place to democratize access to automation and create value for everyone

(As always: my own opinion and not on behalf of GDM or G...)

5h373100

will brown@willccbb

@xeophon @BlackHC benchmark idea: jailbreak an API model vs finetune a smaller OSS model

Florian Brand@xeophon

@BlackHC Just train no model on risk-related data, why restrict it to open ones?

Closed models can similarly be jailbroken and misused when they are trained on such data

4h21760

Florian Brand@xeophon

@BlackHC (cause that’s basically impossible)

I am very sympathetic for increasing the regulation up the chain, the DNA thing is amazing and fits right in with other regulation! You can trivially find out how to build a bomb on the open internet. Order parts and your door gets kicked in

Florian Brand@xeophon

@BlackHC Just train no model on risk-related data, why restrict it to open ones?

Closed models can similarly be jailbroken and misused when they are trained on such data

4h10630

Florian Brand@xeophon

@BlackHC iirc the sandbagging was also triggered by a classifier, but yes, they can post-train the model to be worse in a lot of areas (and they did + do so, see system cards)

problem with dual use (esp cyber) is that being good at attacking helps defense a ton, so it’s hard to balance

Andreas Kirsch 🇺🇦@BlackHC

@xeophon True. Did they do the consequential thing with Fable then by making it worse on capabilities they don't want to have in public?

Imo that's better than having classifiers potentially

The only risk is that the model gets overall worse depending how big the gaps are

4h3630

solipsistic toaster@toasteratemycat

@BlackHC Doubt that anyone could force non-Western companies to abide by those regulations

9h572

Florian Brand@xeophon

@willccbb @BlackHC I guess depending on what misuse entails, I know the results already. I’d argue that decompiling / cracking open binaries is misuse and the very frontier gave me 0 pushbacks when having the right setup

4h411

Andreas Kirsch 🇺🇦@BlackHC

@xeophon True. Did they do the consequential thing with Fable then by making it worse on capabilities they don't want to have in public?

Imo that's better than having classifiers potentially

The only risk is that the model gets overall worse depending how big the gaps are

Florian Brand@xeophon

@BlackHC Just train no model on risk-related data, why restrict it to open ones?

Closed models can similarly be jailbroken and misused when they are trained on such data

4h6800

Andreas Kirsch 🇺🇦@BlackHC

@toasteratemycat Yeah but this is mostly about local regulation. Ie similar to many other things

9h39

Andreas Kirsch 🇺🇦@BlackHC

@xeophon It depends what sort of surveillance you are comfortable with I guess

Florian Brand@xeophon

@BlackHC (cause that’s basically impossible)

4h2800

Aykut Uz@aykutuz

@BlackHC Can’t they be ‘neutered’ to make them unlearn certain subjects , by doing opposite of RL?

https://neurips.cc/virtual/2024/poster/93326

9h521

Florian Brand@xeophon

@willccbb @BlackHC Or 2d ago, when I asked a closed model to reproduce a load-related bug

“Uhm sorry I can’t DDOS that random endpoint” - “Yeah dw it’s mine” “Okay ^_^“

4h201

Andreas Kirsch 🇺🇦@BlackHC

@xeophon I think that is prob an area where some gatekeeping is okay: could imagine that for the most stringent vulnerability discovery, you have licensed providers that are monitored (or at least KYC) whereas for other stuff regular models will be fine

Florian Brand@xeophon

@BlackHC iirc the sandbagging was also triggered by a classifier, but yes, they can post-train the model to be worse in a lot of areas (and they did + do so, see system cards)

problem with dual use (esp cyber) is that being good at attacking helps defense a ton, so it’s hard to balance

3h710

solipsistic toaster@toasteratemycat

@BlackHC The AI Act is a step in the described direction and an acceptable framework to build on. Though I'd argue that, at the moment, the greatest danger is posed by threat actors who can just use highly capable, non-regulated models from Deepseek/Qwen/Z.ai, etc.

8h19

Sahil@sahildarz

@BlackHC That's a realistic perspective. The key'll be determining where regulation starts and ends – protecting consumers from misuse while preserving the innovation that open-source AI can bring.

10h19

Strata@ChainZenit

@BlackHC that’s a fair point, how do you think we regulate that?

9h17

Leyna Music@LeynaMusicx

@BlackHC question: how do you filter training data without just pushing bad actors to proprietary unlabeled datasets? feels like the incentives flip

7h2