Flow matching technique controls pretrained generative models by shifting endpoint means of deterministic interpolants using imperfect reference examples

9/9 The broader idea is that flow models expose a data-level control interface: change the examples, and the posterior endpoint estimate changes with them. Adaptation becomes possible through reference sets, without retraining the generator. 📃 http://arxiv.org/abs/2605.10302 🌐 http://pedrocurvo.com/follow-the-mean 💻 http://github.com/pedrocurvo/follow-the-mean

Joint work with @maxxxzdn, @FEijkelboom, and @jwvdm.

8/9 We also introduce Semi-Parametric Guidance (SPG), which amortizes the same idea with a learned cross-attention anchor and residual refiner. On AFHQv2, it preserves unconditional DiT-B/4 quality while letting class proportions track the reference set provided at inference.

1/9 Follow the Mean: Reference-Guided Flow Matching

Adapting a pretrained generative model to a new target is still expensive, whether through fine-tuning, an auxiliary network, or test-time search. We show flow matching offers a cleaner interface: adaptation through examples.

2/9 For deterministic interpolants, the velocity field is governed entirely by a single object: the endpoint mean. Given an intermediate state, this mean estimates where the trajectory is likely to end under the data distribution. The learned flow points toward that estimate, so changing the endpoint mean changes the dynamics.

3/9 Under a linear bridge, this endpoint mean has an explicit form: a softmax-weighted average over data samples. Algebraically, this is the same operation as cross-attention, with the current noisy state defining the query and the data samples providing keys and values. In this sense, attention appears as the posterior estimator of the endpoint.

5/9 Reference-Mean Guidance (RMG) applies this at inference time. At each denoising step, we compute the reference endpoint mean in closed form and correct the model velocity relative to its own predicted endpoint. The base model remains frozen.

4/9 This gives a simple way to steer a pretrained flow model. Instead of changing the weights, we change the reference distribution used to estimate the endpoint mean. The velocity field is then biased toward the visual attributes represented in that set.

6/9 This is useful because the references do not have to be exact targets. A bank can contain partial evidence: color, texture, pose, object type, or layout cues. The resulting mean shift can still move the trajectory toward the desired composition.

7/9 On GenEval, we use reference banks that contain simpler visual components rather than exact target images. This improves spatial relations by +28.75 and two-object generation by +8.08, suggesting that reference guidance can compose attributes rather than simply copy examples.

@FEijkelboom @pmpcurvo Semiparametric guidance 👍

@FEijkelboom @pmpcurvo https://arxiv.org/pdf/1910.02720

@pmpcurvo Very cool :)