LeRobot introduces a streaming loader designed for robot datasets of 100 TB and beyond

LeRobot says its new loader streams selected robot-training episodes from a Hugging Face dataset or Bucket, keeping a limited pool of compressed video in memory. In an eight-hour training test on four H100 GPUs using RealOmni, a dataset with about 10.7 TB of source data, LeRobot says the model spent 0.52% of step time waiting for data. Streaming is available on LeRobot’s main branch.