Reaction
The role of human oversight in automated pretraining research
A user argues that close human supervision helps steer automated pretraining experiments toward production constraints.
TLDR
A user says automated pretraining research yields more when people steer it toward a model’s production needs. They offer a hypothetical switch from softmax to sigmoid: even if loss falls faster, its effects on long context, post-training and other changes still need checking. They stress that autonomous pretraining research is theoretically solvable.
Combined views
1.3K
1 Source, first seen ago