/Tech4h ago

Microsoft's Nando de Freitas argues next-step sensor prediction forces robots to build latent physical representations without vision

Story Overview

Nando de Freitas highlights how framing robot control as next-step sensor prediction pushes models to form compressed internal maps of physical objects and forces, even absent visual input, mirroring the way language models acquire structure through token prediction alone.

8126121138.8K

#22

Original post

Nando de Freitas@NandoDF#22inTech

A model trained for next-token prediction is forced to build compressed representations of latent structure in text. Ilya Sutskever correctly refers to this phenomenon as understanding. Here, a model trained for next-step sensor prediction, with a robot that has proprioception and touch sensors but no vision, is forced to build compressed representations of latent structure in the physical world. The robot becomes aware of the shape of external objects. That is, it understands the physical properties of the external world that enable it to make better next-step sensor predictions.

This research was previously done by a diverse team of expert engineers at DeepMind over a month - including stars like @notmisha and @yuvaltassa. Remarkably, this reproduction with a completely different robot took only a few hours to implement using Codex.

The automatic creation of physical environments by AI will likely lead to huge advances in areas of science and engineering that use physical simulators or twin models.

The paper and notebook are available at ❤️∀ https://love4all.ai/

9:53 AM · Jun 27, 2026 · 8.3K Views

Open Question

Prediction forces latent structure

Training on proprioceptive sequences alone leads models to encode external properties like shape and mass because those details improve future-state accuracy; whether this scales reliably beyond controlled lab settings remains an Open Question.

Developer Impact

Zero-shot walking on city streets

A causal transformer trained autoregressively on mixed trajectory data lets a full-size humanoid follow velocity commands across varied real-world terrain without task-specific fine-tuning, though the exact contribution of vision-free sensor streams versus other modalities is still unclear.

Sentiment

Users describe Codex reproducing DeepMind robot research as fascinating and relevant because of the proprioception-only training constraint and paper's applicability.

Pos

100.0%

Neg

0.0%

2 comments with sentiment.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Related links

❤️ 4 ∀.ai

LOVE4ALL.AIVia

#22

Nando de Freitas@NandoDF

Hi @JitendraMalikCV Nice to hear from you. The original paper is from 2018. @notmisha was ahead of the game.

My team was always interested in sequence prediction as a path to intelligence. In 2010, we developed spatio-temporal convnets for video and wrote: future improvements in image representations will, instead, be driven by efficient and robust algorithms that learn to extract hierarchical, distributed feature representations from images and video in a fully unsupervised manner. https://www.cs.ubc.ca/~nando/papers/nipsworkshop2010.pdf - I presented this work at Neurips and I still remember you in the audience stating that we needed more serious data. You were right. We needed the GPUs too. Soon enough we were doing GPU summer schools with @CIFAR_News in @UofTCompSci and students in the audience figured out how to code convnets on GPUs so they could run them on bigger datasets like imagenet.

Back then, we had already tried multimodal language models in https://arxiv.org/pdf/1108.3298 — there we wrote: Detecting temporal patterns and predicting into the future is a fundamental problem in machine learning. It has gained great interest recently in the areas of nonparametric Bayesian statistics (Wood et al., 2009) and deep learning (Sutskever et al., 2011), with applications to several domains including language modeling and unsupervised learning of audio and video sequences. Some researchers have argued that sequence prediction is key to understanding human intelligence (Hawkins and Blakeslee, 2005). The close connections between sequence prediction and data compression are perhaps under appreciated within the machine learning community.

Later at @GoogleDeepMind @scott_e_reed proposed Few-shot Autoregressive Density Estimation https://arxiv.org/pdf/1710.10304 which @OpenAI executed much much effectively later when they wrote the GPT3 paper.

1h29