The method achieved an 18% higher manipulation success rate.
Users in the replies thanked collaborators for making the Patch Policy research on efficient visuomotor robot control possible.
No Digg Deeper questions have been answered for this story yet.
Pretrained ViTs see the world in rich, dense detail. Most policies pool it to a single vector before acting, discarding most of it. We introduce Patch Policy: a minimal architectural extension that enables transformer-based policies to consume dense tokens directly, no billion-param VLM required. It outperforms a fine-tuned 7B VLA by 18% with ~0.7% of its parameters, enabling robust, precise manipulation.
Patch Policy provides a clean testbed for which vision features help control. Across five SOTA encoders, the ranking stays consistent across policy types and environments, suggesting that visual representation quality remains a primary bottleneck. WebSSL ranks highest, with DINOv2 also among the strong performers.
The recipe for Patch Policy is embarrassingly simple: a frozen internet-pretrained ViT passes all of its patch tokens into any transformer-based policy head (e.g. VQ-BeT or Diffusion Policy), with an added block-causal mask that maintains causal attention across time.
Overall, Patch Policy stays lean in parameters, training compute, and latency, yet outperforms both global-feature policies and heavyweight VLAs, providing a drop-in pipeline to absorb ongoing progress in visual representation learning for robot control.
Across 4 simulated environments, Patch Policy outperforms policies with global-pooled representations by 40%, and beats a fine-tuned SOTA VLA model while using just ~0.7–5% of its parameters.
The gap widens further on real robots. On three manipulation tasks: cable insertion (~2mm tolerance), pen collection, and tool hanging, Patch Policy with dense features wins, most clearly on the hardest, most contact-precise one, running at just ~11ms/step on a single GPU.
Thanks to all my collaborators who made this possible! @jeffacce, @LangfordAd91020, @bowentan_, @ylecun, @LerrelPinto Website: http://patch-policy.github.io arXiv: https://arxiv.org/abs/2607.18236v1 Code (Coming soon): https://github.com/gaoyuezhou/patch_policy
I love when a paper can fit all the important technical details on a single page, and provides easy-to-follow typing on the data! Pretty cool work too on preserving dense spatial information for visuomotor policies!
Getting spatial detail into a robot policy sounds like it should require a billion-parameter vision-language model. It doesn't. Most visuomotor policies either compress every observation into a single pooled vector or CLS token, or train a visual backbone from scratch. Both throw away the fine-grained spatial detail precise manipulation actually needs, and the second throws away large-scale visual pre-training with it. The policies that do keep the detail, large VLAs, inherit the full cost of a billion-parameter backbone and inference too slow for reactive control. That trade is the one manipulation teams hit every day. Patch Policy doesn't compress. It takes the dense patch features a pretrained Vision Transformer already produces, frozen, no fine-tuning, and feeds them straight into any transformer-based policy head. What makes this work is that spatial attention and temporal causality stay separate. Full bidirectional attention within a frame lets the model see every patch at once. Causal masking across frames keeps the sequence honest about what happened before what. Nobody flattens that structure into one token and hopes precision survives downstream. The numbers. Across four simulated and three real-world suites, dense patch features beat global-pooled baselines by 40 percent relative, regardless of the policy architecture on top. Against a fine-tuned OpenVLA-OFT, Patch Policy comes out 18 percent ahead using roughly 0.7 percent of the parameters, at 11 milliseconds inference. Swap the patch features for pooled or convolutionally compressed ones and performance drops across the board. Same policy head, same training data, worse representation. I read this as a representation result. A bigger backbone buys broad coverage. It doesn't buy back the spatial detail that pooling throws away, and that's one more place where adding parameters isn't the lever people assume it is. Precision wants the patch structure preserved, not compressed and relearned three layers downstream. Worth your time if you're deciding what to fine-tune, or whether you need to fine-tune anything at all. Source: https://lnkd.in/gbRve-Zn Paper: https://lnkd.in/gXTzERiu #Robotics #PhysicalAI #RobotLearning
@GaoyueZhou 把密集token直接喂给策略这个思路挺有意思，但我好奇训练稳定性会不会崩，毕竟ViT的细节太丰富，噪声也容易跟着进来
Prior policies sit at two extremes. Classical-style policies still treat vision as a compact state vector, pooling each frame into one global token and discarding spatial detail. VLAs do use dense patch features, but only by coupling them to a billion-parameter VLM that's heavy to train and slow to run. It remains unclear whether it's the visual patch features or the VLM doing the work.