Blog: https://ginwind.github.io/VLA-JEPA/ Docs: https://huggingface.co/docs/lerobot/main/en/vla_jep Models: https://huggingface.co/collections/lerobot/vla-jepa

VLA-JEPA (3B total) combines two components: 1. a standard VLA (Qwen3-VL-2B-Instruct backbone + action head) 2. a V-JEPA2 world model used only during training.

V-JEPA2 predictor takes current frame encodings s_t + the latent action tokens z_t from Qwen, predicts the next latent state s_{t+1}.

Two losses: Action Head FM loss + World Modeling alignment loss.

At inference, the V-JEPA2 part is dropped. the VLA runs at 10hz on a consumer grade RTX 3080 and uses <6GB of VRAM.

TLDR: VLA-JEPA has a promising architecture leveraging V-JEPA2, enabling pre-training using human videos. Shows greater robustness to perturbations at inference, yet runs quite fast on consumer grade GPUs.

Thanks to everyone involved in the implementation, especially the authors of VLA-JEPA: Jingwen Sun & @zhang_weny92997

Paper: https://arxiv.org/pdf/2602.10098 Docs: https://huggingface.co/docs/lerobot/main/en/vla_jepa

Any further question? We will answer them in the comments!

As many of you were interested in the technical details of the model, here is a followup thread to go more into technical details about VLA-JEPA.

1. Architecture 2. Training 3. Recipe for the demo 4. TLDR

🧵below

Training phases: 1. Pretraining on human videos (Qwen + V-JEPA2) 2. Training on robot videos (Qwen + Action Head + V-JEPA2) 3. Finetuning (Qwen + Action Head + V-JEPA2)

=> Pre-training on human videos help especially on LIBERO-Plus benchmark across perturbations. => Varying the horizon of the world model has an effect on short / long tasks.

@LeRobotHF So I record nearly 800 duck episodes only to find there is a model that could have done it with 13 examples? 😭

@LeRobotHF What's blows my mind is that it's a 3B parameter, 6 GB model! It should run on a $200 GPU!

Actions are cheaper than words..! 🤯

@LeRobotHF The 13 example fine tune is the most interesting part.

V-JEPA2 helps the robot learn how actions affect the world before training on the real task. Then during use the world model is removed so the system stays simple and fast.

That is a smart setup for real time robotics.

@LeRobotHF Only 13 demos is great

For the demo, we collected 13 examples of a screwing task available here: https://huggingface.co/datasets/maximellerbach/omx_screwing

Then started the training from the https://huggingface.co/lerobot/VLA-JEPA-Pretrain pretrained weights, 5k steps with a small 16 batch-size.

This task is a good mix of picking + dexterity, but still quite simple enough not to require too many examples.

jepa makes sense: skip pixel reconstruction, learn latent dynamics directly. but i keep wondering how much of the gain is better feature alignment vs actual world model reasoning. libero is controlled lighting, objects, camera. the interesting failure mode is whether the latent world model helps in messy ood scenarios where standard vlAs hallucinate

@LeRobotHF Fascinating, many practical Qs: Action representation? FPS? Horizon? Normalization? Absolute/relative?Cartesian or joints space? Proprioceptive input? How many epochs for demo? True robustness against lighting or source nut orientation/position changes?

@SAldwais @pepijn2233 @LeRobotHF Here’s the link for the ROBOTIS OMX (Dynamixel-X) arms: https://www.robotis.us/omx-ai-us/

OMX-AI bundle (leader + follower) ~$344. Built for teleop, imitation learning & Physical AI.

@LeRobotHF Amazing!! This is "merely" a way to make use of VJEPA, not to fine tune it, right?

@LeRobotHF 🙏 Honored to have our VLA‑JEPA work included — this kind of open, well‑engineered plumbing is exactly what the embodied AI community needs right now. Looking forward to seeing what people build with it. 🤗📦

@LeRobotHF I need the same arms. Were to order it ?

@LeRobotHF Congrats to the LeRobot team! Now the real question: will JEPA actually deliver in robotics?

@KuphDev @LeRobotHF it's a bit more complicated than that 😅 the demo we made is quite simple: pick up from mostly the same area a red nut, and then screwing at mostly the same place aswell. So there is not a lot of variety in the 13 episodes

@LeRobotHF Ah, see most is answered in the tables on https://huggingface.co/docs/lerobot/vla_jepa - fair comparison with Fast-WAM and with Dino latens eg LDA-1B would be interesting https://arxiv.org/html/2602.12215v1