Zirui Peng, Heyi Zheng and Ziming Liu report that reducing activation bias slowed early learning but yielded a lower final loss in a controlled experiment. Their preprint also reports comparable or better late-stage performance in larger vision and language models.
The paper calls the average output of an activation function its activation bias. Common functions including ReLU, GELU, Swish and Sigmoid favor positive outputs while suppressing or excluding negative ones, yielding a nonzero post-activation mean. In an Oct. 31 post, Liu explained that ReLU, SiLU and GeLU can have a positive output mean even when their inputs are centered at zero. He called the nonzero mean a “bug” and argued that it contributes to training problems, including loss spikes and overshooting.
To test the shift, the authors compared students with the same initial parameters, training data and optimization settings. One student had a fixed amount subtracted from each layer’s activation output. The preprint reports that the unshifted student’s hidden state overshot during early training and that the shifted student had a flatter measured loss landscape. Its shift for ReLU was 0.230; Liu’s post gives the rounded figure 0.2.
In a post, @mufan_li disputes Liu’s explanation, arguing that the positive output mean is not the issue: After the outputs are multiplied by weights, the next pre-activation is more or less zero-mean. The post instead describes the shift as shaping the activation for better signal propagation.
Comments