Users praise LingBot-VA 2.0 for pretraining the full robot control stack from scratch, calling it the harder but better approach that optimizes directly for control instead of bolting action heads onto video models.
Based on 3 visible X reactions from 3 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Very impressed with this video-action foundation model built natively for robot control, from @robbyant_brain, an embodied AI company under Ant Group. Project page. http://technology.robbyant.com/lingbot-va-v2 Paper: https://github.com/Robbyant/lingbot-va/blob/main/LingBot_VA2_paper.pdf
@omarsar0 This is the key distinction — most "robot models" just pipe a video model into a motor command. Pre-training the entire stack for control from scratch means the latent space is actually optimized for physics, not pixels.
@omarsar0 Pretraining the full control stack instead of bolting on an action head is the harder but better approach.
It puts world states and latent actions in one semantic latent space, a visual-action tokenizer aligned to a frozen vision foundation model. Latent actions are learned self-supervised from unlabeled video. So web video, not just scarce robot demos, carries a control signal. https://x.com/omarsar0/status/2075955187034775982/photo/1
Most video-action robot models are a video generator built for content, with an action head bolted on. LingBot-VA 2.0 pretrains the whole stack for control from scratch. Lots of technical improvements here that stand out: https://x.com/omarsar0/status/2075955181640892823/photo/1
Foresight Reasoning keeps it closed-loop. Draft the next action chunk while the current one executes, then re-ground every prediction on the latest real observation. Predict, then correct, so the look-ahead does not drift. https://x.com/omarsar0/status/2075955195863839150/photo/1
Getting this to real time is as much a systems problem as a modeling one. Distill both experts to two steps per chunk, then FP8 TensorRT, paged KV-cache, and runtime cleanup. 927 ms down to 142 ms per chunk. Up to a peak asynchronous execution frequency of 225 Hz. 6.5x end to end.
The capacity trick is a sparse Mixture-of-Experts video stream. 128 experts, top-8, one shared. About 13B total params, about 1.9B active per token. The action stream stays dense. Capacity for hard visual dynamics, low per-step compute for high-frequency control.
Causal from the start. Video generators use bidirectional attention over all frames. Control runs forward in time. Retrofitting one into the other forgets the priors you paid for in pretraining. Training causal from scratch avoids that.
Users praise LingBot-VA 2.0 for pretraining the full robot control stack from scratch, calling it the harder but better approach that optimizes directly for control instead of bolting action heads onto video models.
Based on 3 visible X reactions from 3 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Foresight Reasoning keeps it closed-loop. Draft the next action chunk while the current one executes, then re-ground every prediction on the latest real observation. Predict, then correct, so the look-ahead does not drift. https://x.com/omarsar0/status/2075955195863839150/photo/1