“Induction heads” reportedly explain a bump in Kaplan’s AI scaling plot

A post says Anthropic researchers traced a bump in Kaplan and colleagues’ scaling-law plot to the emergence of induction heads. It describes their behavior as “[A][B] … [A] → [B]”: encountering A again prompts the head to copy the token that previously followed it. The author connects increasingly abstract versions of this behavior to in-context learning—learning from the context a model is given. According to the post, one-layer transformers do not form induction heads and therefore skip this particular phase transition, leaving their loss curve less steep; induction heads can emerge with two or more layers.