Computer vision expert Jitendra Malik argues robotics researchers should prioritize physical sensorimotor manipulation over vision-language-action models

Appreciate Jitendra's takes on world models/VLMs. His word below is why back in 2019-2021, instead of VLAs for simple pick-and-place, we chose assembly.

Dexterity = mutual info between your intent and forces/torques on objects via contacts.

I want to offer some unsolicited advice to computer vision researchers jumping into robotics. Don't focus too much on VLMs, VLAs etc. That's fine, but the real action is at the sensorimotor level. Most of the open problems in robotics are in manipulation, which is about hand-object interaction, and contacts and forces are central. Proprioception and tactile sensing are as important as vision. Don't get seduced by cherry-picked demos. You can't do robotics without doing robotics.

There are two core "software" problems worth solving in robotics in my opinion: - end to end learning of dexterous manipulation skills - dynamic, long horizon spatial memory which can interact with the above

As a field we're currently very focused on the first because, well, it's what work the best with current techniques and it produces flashier demos. And for assembly lines it's really the one you need.

But the real long tail of robotics work will actually require both, and I really know very few teams that have strong expertise in both things.

My impression is that the SECOND problem is the harder one, because if it was easy we'd have useful AR glasses on the market

w/ Satoshi, @IMordatch , @coolboi95 (founding engineer at Generalist), Michael, Luke Metz, Dan Freeman, originally w @Vikashplus. great memories from back in research days at Google Brain Robotics.

Real robot link: https://sites.google.com/view/u-shape-block-assembly Sim link: https://sites.google.com/view/learning-direct-assembly

Such a good advice!

I agree. Let's first figure out what necessary modalities are required to make the policy work, focusing on improving the sensors first so we can obtain high-quality data.

By then, LLMs will be strong enough (or are already strong enough) to figure out the best model architecture for a robotics foundation model recurisively.

From first principles, robotics data shouldn't consist solely of vision-action data; it isn't like autonomous driving. It only looks that way right now because the field is currently dominated by computer vision and autonomous driving researchers.

@JitendraMalikCV 💯💯 "Most of the open problems in robotics are in manipulation, which is about hand-object interaction, and contacts and forces are central. Proprioception and tactile sensing are as important as vision."

@JitendraMalikCV @jon_barron 💯

Strong take. VLMs get all the attention because they're easy to benchmark, but the sensorimotor gap is where robotics actually breaks down. Understanding a scene is unsolved; reliably *touching* the right part of it with the right force at the right moment is a different problem entirely.

@JitendraMalikCV @CSProfKGD VLM are computationally expensive and slow for action recognition, yet i don’t know why the vision community is obsessed with them rather than improving the vision models.

@JitendraMalikCV Well, VLAs are actually working quite well in terms of generalization, most of them are also integrating flow matching/diffusion for trajectory/controls as well, not to mention the amount of work going on in representing proprioceptive states and other sensing modalities.

@JitendraMalikCV i've seen so many VLA demos that look incredible and then ship absolutely nothing. meanwhile Figure at BMW: 90k parts, 30k cars, 99%+ placement. -that's years of boring sensorimotor data and brutal evals. the unsexy stuff is the stuff that works, every time, it's almost annoying

@JitendraMalikCV Computer vision already is a flawed terminology. It should be called computer perception.

@JitendraMalikCV @et_tu_deux Does extreme robot arm lightweighting help with this at all?

@JitendraMalikCV @jon_barron very helpful thank you. curious how you think about interoception in relation to tactile sensing and proprioception?

@JitendraMalikCV What textbook would you recommend for diving deeper into the sensorimotor side of robotics?

@JitendraMalikCV You need all of them. Vision included.

@JitendraMalikCV ive simplified the 40 steps PDE cluster fk of a system

and worst i dont do robotics lol

@JitendraMalikCV I think there are more AMRs out there in comparison to manipulators. Real action might be there.

@JitendraMalikCV without a language model in the loop, how will the robot know to say "oops" when it drops something?