Xavier Gonzalez releases a paper mapping Newton iterations to recurrent neural networks, Picard iterations to diffusion models, and Jacobi iterations to parallel LLM decoding

Overall, we unify the major fixed-point techniques for parallelizing sequential computation.

We provide the following guidance: pick the fixed-point method that best approximates the dynamics you care about.

paper: https://alphaxiv.org/abs/2509.21716 code: https://github.com/lindermanlab/parallelizing_with_lds

We show that all these techniques for parallelizing nonlinear dynamics reduce to repeating the following 2 steps: 1. Linearize the dynamics 2. Evaluate them in parallel. Newton, Picard, Jacobi etc just use different approximate linearizations.

Fixed point iterations for parallelizing nonlinear dynamics is all the rage: - Newton for RNNs - Picard for diffusion models - Jacobi for parallel decode of LLMs

But how do these techniques relate, and when should you use them?

We show you how in our new paper 🧵

Why does linearizing the dynamics make them parallelizable?

Through the 🚀 parallel associative scan🚀

The pscan powers modern SSMs like S5 and mamba. It's also what lets us parallelize nonlinear dynamics.

We provide a gentle intro & tutorial on the parallel scan in Appendix B

For example, Picard iterations approximate the dynamics by the identity. So they are great for diffusion models, which use Euler discretization.

But, for the same reason, Picard iterations struggle with contractive systems, where Newton/Jacobi excel.

We prove theoretically how the error of the "approximate linearization" of the dynamics impacts how many fixed-point iterations are needed for convergence.

We show that better approximations converge faster. So, you should use fixed-point methods that fit your target problem.

A Unifying Framework for Parallelizing Sequential Models with Linear Dynamical Systems.

co-lead with @ekellbuch, supervised by @scott_linderman, and on the whole a great team effort with @hyundongleee @jerrywliu @heyyalexwang @dmzoltowski @Leokoz8 and Chris Re.

@xavierjgonzalez Great work Xavier!! Quick question, I heard that people are having trouble with accuracies at the inference when it comes to PDEs when they are trained with PINN style. Is that ever an issue here, or maybe I am making a spurious connection?

@xavierjgonzalez I love this work. It seems like Newton generally requires very few iterations which means in theory could provide massive speedup. Do you find the main obstacle there is extra memory requirements or time spent on the extra model calls to construct the Jacobians?

@xavierjgonzalez I’ve been poking around with similar math this week and fixed point iteration has a whole associated operator calculus. Really fruitful ground for studying optimizer behavior and convergence