Rerun’s Pablo Vela describes Specialist as a recently open-sourced UC Berkeley system for collecting robot-training demonstrations with handheld grippers instead of a robot. In the write-up, which he shared on November 20, Vela says robot-control models trained only on those demonstrations can run on a real robot arm. The robot still has to show up for work eventually. SourceSource
According to Vela, Specialist uses Rerun recordings from capture through model training—not just to display the results, but to hold raw sensor data, reconstructed movements and training inputs. Source
A handheld demonstration, then an offline reconstruction
Vela describes a gripper with six printed parts, taking about five hours to print and six minutes to assemble. Each uses an Insta360 X3 camera and weighs 315 grams with the camera attached. The operator moves the gripper’s fingers directly with a thumb and index finger. Source
The camera’s two fisheye lenses keep the workspace and AprilTags—visual markers on the grippers and around the room—in view. After collection, an offline solver uses camera and motion-sensor data, along with those tags, to estimate each gripper’s movement and finger angle. Source
Rerun isn’t just the replay screen
Writing for Rerun about a system built on its tools, Vela describes separate .rrd recording layers for raw captures, estimated movements, visual debugging information and training-ready data. The layers share a data model. Source
The pipeline converts camera files, estimates motion and prepares the training data, with an optional step to register sessions in a Rerun catalog. Vela explains that a revised motion estimate can be regenerated without rebuilding the other layers, while intermediate results remain available for inspection. Source
Inspecting the data and training with it use the same recording format. The dataset doesn’t need a costume change before its next job. Source
The printer doesn’t do the reconstruction
Vela says reconstruction of real recordings requires Linux and an NVIDIA GPU, and runs at close to real time—a potential bottleneck as datasets grow. The gripper also measures finger position, not force. Source
For a route that doesn’t require building a gripper, Vela points to specialist_sim, a MuJoCo simulation with LEGO sorting and pick-and-place tasks that produces the same kind of training-data layers. He reports running it on his MacBook’s CPU, but rendering at the default 240-pixel image size took a few minutes per episode. He also notes that the simulator’s README asks for a CUDA GPU.
Comments