Flow matching technique controls pretrained generative models by shifting endpoint means of deterministic interpolants using imperfect reference examples
Demos cover style transfers, attribute edits, and anatomy corrections.
#263Andrew Carr 🤸@ANDREW_N_CARROhhhhhh, this is great. I'm excited about reference set guidance. It gives lots of benefits (reward hacking) over lots of standard reward forcing.
Most people do have several examples they want to match, either in images, video, or even motion.
Pedro@pmpcurvoGuide with examples, not rewards 🐘 Controlling what a pretrained generative model produces is still mostly a choice between three slow options: fine-tune it, attach a reward network, or search at inference. We found flow matching allows a fourth, and it costs almost nothing. In deterministic interpolants, the velocity of the flow is determined by where the trajectory is headed: the endpoint mean. Shift that mean, and the entire flow shifts with it. This turns control into a matter of reference. Change the examples that define the endpoint, and you change the direction the model follows. The examples need not be perfect. They only need to point the flow toward the attribute you want. Color, identity, style, and structure, all controllable through examples. 🧵👇
#1677Jan-Willem van de Meent@JWVDMSuper excited about this new paper with Pedro Curvo. To guide a flow model at sampling time, just steer toward the empirical mean of a set of reference samples. It's almost embarrassingly simple, and it works much better than it has any right to. Worth a read!
Pedro@pmpcurvoGuide with examples, not rewards 🐘 Controlling what a pretrained generative model produces is still mostly a choice between three slow options: fine-tune it, attach a reward network, or search at inference. We found flow matching allows a fourth, and it costs almost nothing. In deterministic interpolants, the velocity of the flow is determined by where the trajectory is headed: the endpoint mean. Shift that mean, and the entire flow shifts with it. This turns control into a matter of reference. Change the examples that define the endpoint, and you change the direction the model follows. The examples need not be perfect. They only need to point the flow toward the attribute you want. Color, identity, style, and structure, all controllable through examples. 🧵👇