The case for gradient descent over difference of means in AI steering

The blog argues that interpretability work often mixes up a method’s form, such as steering or neuron analysis, with its estimator, such as difference of means or gradient descent on log-loss. Its author says gradient descent is usually preferable when available. A reply praises a paper on contrastive losses for steering; the paper’s authors report that their RePS approach outperformed existing steering methods trained with a language-modeling objective on Gemma models.