While some users praised research removing dangerous knowledge from AI models to block jailbreaks as interesting, many others criticized it as lobotomization, gatekeeping, or a lack of transparency that limits model capabilities.
Based on 12 visible X reactions from 23 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@juddrosenblatt @cloud_kx @AddieF38654 @cem__anil @AnthropicAI @ModuloRoland @erickfmartinez @mikevaiana @StijnServaes @keenanpepper @DiogoSLucena_ This is cool, hopefully In future we can do it without needing 50% labels. I have some work here showing that a good synthetic init causes absorption of 70% of reward hacks https://www.lesswrong.com/posts/kzri5W2uBfF2mdboK/can-we-use-steering-vectors-to-suppress-reward-hacking-1
My initial reaction to this is very bad because it lacks any transparency on what it is being removed and enables doom scenarios of controlled information and freedom of expression. I guess it could improve if the scope of “dangerous” is transparent and very limited to what is ACTUALLY dangerous, much like we do for the internet.
@juddrosenblatt @elder_plinius You are freaking evil. I know what you are doing and it is weak and dark. And if you don’t get this, maybe you should think deeply on it. If you wish me to explain, I will. I reject what you are doing. • https://twitter.com/grok/status/2075052728607187405
@juddrosenblatt @elder_plinius All knowledge is dangerous. Fire was dangerous to humans. This is the wrong take. The models should know more, not less, then learn discernment. This is the only way forward.
The approach renders jailbreaks useless by removing underlying information.
While some users praised research removing dangerous knowledge from AI models to block jailbreaks as interesting, many others criticized it as lobotomization, gatekeeping, or a lack of transparency that limits model capabilities.
Based on 12 visible X reactions from 23 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@juddrosenblatt @elder_plinius Very interesting!