Jiaxin Wen, a CS PhD student at UC Berkeley and part-time researcher at Anthropic, says pre-training fixes for advanced AI may improve alignment via coherence or instead enable coherent but dangerous goals.
Richard Ngo argues automation framing excuses labs to accelerate before solving foundational questions.
@jiaxinwen22 IMO the concept of automating alignment research is being used as an excuse for what the labs want to do anyway (accelerate as fast as possible).
Alignment researchers used to focus on questions like “how can we verify AGIs’ answers?”, which seem important to answer before RSI.
Jiaxin Wen@jiaxinwen22in this specific case, whether fixing pre-training would lead to worse alignment is also unclear to me. for example, the model could become much more coherent, has cleaner internals, robust characters, etc. However, the model might also be capable enough to develop coherent goals which feel dangerous to me. anyway, before deploying the model, I think labs should definitely try it
@RichardMCNgo I don't know the intent of labs. I think it's perhaps useless to know as long as they do the right thing.
I'd be happy if they invest enough on fuzzy tasks like improving legibilty / conceptual reasoning. This ensures all the capability efforts would help alignment eventually.
Richard Ngo@RichardMCNgo@jiaxinwen22 IMO the concept of automating alignment research is being used as an excuse for what the labs want to do anyway (accelerate as fast as possible). Alignment researchers used to focus on questions like “how can we verify AGIs’ answers?”, which seem important to answer before RSI.
@RichardMCNgo and I do think technically we should automate alignment to solve it
Richard Ngo@RichardMCNgo@jiaxinwen22 IMO the concept of automating alignment research is being used as an excuse for what the labs want to do anyway (accelerate as fast as possible). Alignment researchers used to focus on questions like “how can we verify AGIs’ answers?”, which seem important to answer before RSI.