Thanks @tilderesearch for making this blog post! A few future directions for Parallax I find interesting: - Optimizer: understanding why optimizer interacts so strongly with the Parallax correction, and what that implies for attention more broadly. - Architecture: developing the nonparametric counterpart of DeltaNet, a mechanism sitting between Parallax and LLA. - System: Parallax keeps the structure of standard attention, so it should compose with attention sparsity optimizations. - Post-training: with W_R = 0, Parallax is standard attention, so it can be initialized from a pretrained checkpoint and adapted. I'm curious whether W_R could serve as a steering parameter for RL.