Generalist robot policies learn many useful skills, but struggle to select good behaviors for new tasks. To solve this, we introduce Flow Reversal Steering (FRS), a method to refine coarse semantic guidance into precise, in-distribution motions. http://flow-reversal-steering.github.io
1/N
Users express excitement about Flow Reversal Steering because it lets semantic knowledge guide robots in adjusting fine-grained behaviors.
FRS passes coarse reference actions through flow generalists in reverse, mapping them to noise. When denoised, this yields *similar* actions that are effective and precise. FRS thus takes rough sketches of robot behaviors and "projects" it into the generalist's distribution. 2/N
We hope FRS enables a paradigm wherein semantic knowledge guides robot learning. Very fun leading this w/ @tangerinecoder, in collaboration w/ @ajwagenmaker, @chelseabfinn, @svlevine.
Generalist robot policies learn many useful skills. How can we elicit relevant behaviors when faced with new tasks? We introduce Flow Reversal Steering (FRS): a way to refine coarse actions produced by semantic reasoning into similar precise ones! http://flow-reversal-steering.github.io 1/N
Also check out Will's thread here:
This helps humans + VLMs -- which are often bad at emitting low-level actions, but good at scalably making high-level inferences -- steer the generalist towards semantically-reasonable behaviors. We show how this can lead to gains, even without any training! 3/N
FRS can be distilled into noise policies. Past works use expensive RL for this, but FRS gets around this by finding noises from reference actions, letting us train with supervised learning instead! We call this Diffusion Steering via Behavioral Cloning (DSBC). 4/N
DSBC gets high performance training for 1 min on 10 trajs using ~1GB of GPU memory. It can also use standard offline robot data (without noises), as flow reversal can augment demos w/ noise for the policy to copy. In this data regime, standard BC completely fails. 5/N
Last, when used *with* noise-space RL, FRS can speed up + improve learning by teaching the policy to explore semantically-reasonable behaviors. Providing just one FRS success as prior data allows RL to learn tasks that the base VLA (and thus, standard RL) completely fails on! 6/N
@tangerinecoder @ajwagenmaker @chelseabfinn @svlevine Check out Andy's thread here ...
FRS also can speed up and improve noise space reinforcement learning, by focusing policy exploration on semantically sensible strategies. Adding one FRS success in the replay buffer allows our method to attain high performance on tasks that standard RL completely fails on.
5/N
FRS maps coarse reference actions to noise by passing them through flow generalists in reverse. The resulting "noise seeds" can then be denoised by the same policy to yield actions which *retain semantic relevance with increased precision*.
2/N
FRS pairs especialy well with VLMs, as they excel at making high-level inference but struggle with fine motor skills. We find that FRS can improve *zero-shot* performance of generalist policies on a broad range of tasks by integrating VLM feedback into the model.
3/N
FRS can be distilled into noise policies for inference without high-level planning. Compared to past RL work, we dramatically improve efficiency via supervised learning on noise from FRS-steered trajectories, which we call Diffusion Steering via Behavioral Cloning (DSBC)!
4/N
@tangerinecoder @ajwagenmaker @chelseabfinn @svlevine ... and Sergey's thread here!
And check out Sergey's thread here:
We hope that FRS can be a stepping stone to a world where robots can adjust fine-grained behavior by integrating all kinds of semantic knowledge and reasoning!
Had a blast on this project with co-lead @verityw_, and amazing collaborators @ajwagenmaker, @chelseabfinn, @svlevine.
