Users agree the argument that open-weight AI models can circumvent safety tests is reasonable and favor staggered releases for models with greater cyber or bio capabilities.
Based on 1 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
I think that's a reasonable take. I do favor measures like staggered release of new models with greater cyber or bio capabilities, with first access to reputable defenders so that the models can be used to increase security. So long as that's done in a predictable, non-political, and transparent manner. And I want to be clear that while I don't like mandatory safety testing as a gate for release, I do think that the manner and institutions that Demis proposes to manage that are superior to most AI safety proposals. I expect to lose on this point, to be honest. And it's not a hill I'll die on. But in general I think we should be suspicious of prior restraint of a product, even on the basis of safety, until we have strong evidence that harms are real. Finally, I don't actually think you can legally restrict the release of open weights without violating the 1st Amendment. The precedent here, as I understand it, is that you can restrict running code, but not the transmission of code. In that world, again, there will be incredible capabilities available to nefarious actors (as well as altruistic actors). So our focus on making the models safe instead of making the ecosystem safe seems off target. But, as you say, it may be a case of ”and, not or".
It's possible but not certain to me. And it's not clear that requiring models to pass certain safety tests is a useful way to improve safety. My assumption is that open weight models will always exist and can be jailbroken, fine tuned, or improved with harnesses in ways that extend their capabilities and that circumvent safeguards. Nefarious actors will always be able to get powerful AI from somewhere, and to improve the capabilities of AI specific to their desired task. [E.g., a significant fraction of Mythos's capabilities in cyber were the result of harness work that third parties were able to do on top of nominally less capable open weight models.] In that world, safety testing of models is fairly ineffective as a way to increase global safety. What's more effective is proactively increasing security and resilience of the ecosystem. Using models aggressively for cyber hardening, doing all the pandemic preparedness work we already know we should do, etc..
@ramez It doesn't strike me as an either/or option, but rather both/and. Given the potential stakes, I wouldn't put all the onus on the downstream users and ecosystem to keep things safe in this domain, or in the others I mentioned.
I think that's a reasonable take. I do favor measures like staggered release of new models with greater cyber or bio capabilities, with first access to reputable defenders so that the models can be used to increase security. So long as that's done in a predictable, non-political, and transparent manner. And I want to be clear that while I don't like mandatory safety testing as a gate for release, I do think that the manner and institutions that Demis proposes to manage that are superior to most AI safety proposals. I expect to lose on this point, to be honest. And it's not a hill I'll die on. But in general I think we should be suspicious of prior restraint of a product, even on the basis of safety, until we have strong evidence that harms are real. Finally, I don't actually think you can legally restrict the release of open weights without violating the 1st Amendment. The precedent here, as I understand it, is that you can restrict running code, but not the transmission of code. In that world, again, there will be incredible capabilities available to nefarious actors (as well as altruistic actors). So our focus on making the models safe instead of making the ecosystem safe seems off target. But, as you say, it may be a case of ”and, not or".
It's possible but not certain to me. And it's not clear that requiring models to pass certain safety tests is a useful way to improve safety. My assumption is that open weight models will always exist and can be jailbroken, fine tuned, or improved with harnesses in ways that extend their capabilities and that circumvent safeguards. Nefarious actors will always be able to get powerful AI from somewhere, and to improve the capabilities of AI specific to their desired task. [E.g., a significant fraction of Mythos's capabilities in cyber were the result of harness work that third parties were able to do on top of nominally less capable open weight models.] In that world, safety testing of models is fairly ineffective as a way to increase global safety. What's more effective is proactively increasing security and resilience of the ecosystem. Using models aggressively for cyber hardening, doing all the pandemic preparedness work we already know we should do, etc..
@ramez It doesn't strike me as an either/or option, but rather both/and. Given the potential stakes, I wouldn't put all the onus on the downstream users and ecosystem to keep things safe in this domain, or in the others I mentioned.
Users agree the argument that open-weight AI models can circumvent safety tests is reasonable and favor staggered releases for models with greater cyber or bio capabilities.
Based on 1 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.