Patch Policy Feeds Frozen Pretrained ViT Tokens Into Policy Models
New method feeds frozen pretrained ViT patch tokens into transformer policy heads for robotics tasks.
Patch Policy is a new robotics approach that extracts patch tokens from an internet-pretrained frozen Vision Transformer and feeds them into models such as VQ-BeT or Diffusion Policy. Researchers report it delivers massive gains in training efficiency, sharply cuts model size and latency, and outperforms OpenVLA-OFT while using only 0.7 percent of the parameters. The method was highlighted by academics Furong Huang and Lerrel Pinto as a simple switch from standard visual tokens to dense features. Status remains at the research announcement stage with no deployment details confirmed.
Combined views
4K
2 posts, first seen 18h ago
Reactions from ranked influencers
2 posts
Cool!
The recipe for Patch Policy is embarrassingly simple: a frozen internet-pretrained ViT passes all of its patch tokens into any transformer-based policy head (e.g. VQ-BeT or Diffusion Policy), with an added block-causal mask that maintains causal attention across time.
So it turns out that simply switching the visual tokens in ViTs to dense features can give us MASSIVE improvements in training efficiency, reducing model size, latency. Check the thread below for more details!
Your policy doesn't need 7B params. It simply needs dense features. Introducing Patch Policy: pretrained ViT + small transformer beats OpenVLA-OFT with 0.7% of its params, and trains on a 5090. Here it inserts a cable (~2mm tol), and does it again as we unplug mid-rollout. 🧵