Back to Guides

Robot Training Data: Collection, Formats, Datasets and Quality

Where to start with robot learning data: how demonstrations are collected, the HDF5, RLDS and LeRobot formats, public datasets like DROID and Open X-Embodiment, quality control, and the collection hardware we sell.

A robot policy is only as good as the demonstrations it learned from. Most teams lose weeks on the data side, not the model side: a format that their training code cannot read, cameras that drift out of sync, or a few hundred episodes that turn out to be too similar to generalize. This page is the map. It sends you to the one guide that answers each question, in the order most projects hit them.

Start here if you are still deciding:

1. Collecting demonstrations

What matters: a collection interface that matches the task (leader-follower arms for contact-rich and bimanual work, handheld grippers for fast in-the-wild data, gloves for multi-finger hands), cameras that are calibrated and time-synchronized, and an episode protocol that every operator follows the same way.

Hardware Price What it is for
OpenArm $7,000.00 Open-source arm built for teleoperation and imitation learning. Pair a leader and a follower for leader-follower collection
Trossen ALOHA Stationary $27,999.00 Two leader and two follower WidowX arms on a stationary frame, for bimanual demonstrations. See the ALOHA guide
RC UMI $1,250.00 Handheld gripper built to the open-source Universal Manipulation Interface design, with a GoPro wrist camera. Collects demonstrations without a robot
AgileX PiKA Data Gripper Kit From $5,949.96 Handheld collectors plus positioning stations that record synchronized 6D pose, gripper angle, RGB, depth and IMU
Wuji Glove $4,500.00 Per-finger tracking glove for teleoperating dexterous hands
RC Dex-UMI $9,800.00 UMI-style system for multi-finger hand data, with a LEAP-hand-compatible end effector

The data collection guide compares the four teleoperation methods on quality, operator training time and throughput. If you collect with LeRobot, record a LeRobot dataset walks through the lerobot record workflow end to end. For a two-arm setup, the ALOHA guide covers the leader-follower rig and its HDF5 episode files.

2. Formats

What matters: what your training code reads natively, how large the dataset will get, and whether you will share it.

  • HDF5 is one file per episode with arrays for joint states, actions and camera frames. It is what the original ALOHA and ACT code write and read.
  • RLDS is Google's episode format built on TensorFlow Datasets (google-research/rlds). Open X-Embodiment and DROID are distributed in it, so you will meet it as soon as you pretrain on public data.
  • LeRobotDataset stores tabular signals as Parquet, video as MP4 and the schema and normalization statistics as metadata, and loads straight from the Hugging Face Hub (format documentation).

Pick one format for training and convert at the edges. The format guide covers the trade-offs and the conversion paths between all three.

3. Public datasets

Public data rarely replaces your own demonstrations, but it is the usual starting point for pretraining and for benchmarking a pipeline before your rig is ready.

  • DROID: about 76,000 teleoperated trajectories (350 hours) across 564 scenes and 84 tasks, all on Franka Panda arms (paper). The reference for diverse, in-the-wild single-arm manipulation.
  • Open X-Embodiment: more than a million trajectories from 22 robot embodiments, pooled from 21 institutions (paper). The pretraining mix behind RT-X, Octo and OpenVLA.
  • ALOHA datasets and the LeRobot collection: bimanual and low-cost-arm data in the formats LeRobot trains on directly.

The dataset directory lists the rest by task, robot and modality.

4. Quality control and curation

What matters: catching bad episodes before they reach training, labeling what each episode contains, and keeping enough variation in the dataset that the policy generalizes.

Check every episode for the basics: all camera streams present and in sync with the joint data, no frozen or black frames, the task completed, and no long idle stretches at the start or end. Label success and failure explicitly rather than deleting failures silently, because failed attempts are useful for evaluation and for training recovery behavior. The annotation guide covers success flags, language labels, contact events and task segmentation, including how to keep timestamps aligned across sensors.

Curation is mostly about coverage. A thousand nearly identical demonstrations teach less than three hundred that vary object position, lighting and object instance. The data collection guide gives episode-count targets by task complexity and a QA checklist. Once a policy is deployed, the data flywheel guide shows how to turn its failures into the next round of collection.

5. Build in-house or buy

Collecting in-house means buying stations, training operators and owning QA. That is the right call when the task, the robot or the data are core to your product. Buying collection as a service is faster when you need volume on a deadline or want to test a task before committing to hardware. The service buyer's guide lists the quality metrics, delivery formats and contract terms to ask any provider about, us included. For what we offer, see data services.

Related guides