Reducing activation bias improved late-stage learning in preprint experiments

Subtracting fixed values from activation outputs slowed early learning but produced lower final loss in a controlled experiment described in a preprint. Its authors also report comparable or better late-stage performance in larger vision and language models. In one post, @mufan_li disputes their explanation, arguing that the shift may improve signal propagation instead of correcting the positive output mean itself.