Arb Research red-teamed the safety framework's mathematical argument
We recently helped Yoshua Bengio and his LawZero colleagues write up their mathematical argument for Scientist AI. Our role was to be an internal red-team.
The core notion is "consequence-invariance": training that never selects predictors for what their outputs would cause
No Digg Deeper questions have been answered for this story yet.
@RichardMCNgo On a skim, do you view the paper as overclaiming? I view the project as more of a retreat from the failure mode you describe in OP: not a grandiose general solution to value alignment but an attempt to bound one kind of danger
@RichardMCNgo disclaimer: I wasn't involved in the details at all and don't speak for them
