Robot Training Data: Collection, Formats, Datasets and Quality
Where to start with robot learning data: how demonstrations are collected, the HDF5, RLDS and LeRobot formats, public datasets like DROID and Open X-Embodiment, quality control, and the collection hardware we sell.
A robot policy is only as good as the demonstrations it learned from. Most teams lose weeks on the data side, not the model side: a format that their training code cannot read, cameras that drift out of sync, or a few hundred episodes that turn out to be too similar to generalize. This page is the map. It sends you to the one guide that answers each question, in the order most projects hit them.
Start here if you are still deciding:
- First project, no data yet? Read how to collect robot training data, then record a LeRobot dataset.
- Choosing a collection method or planning scale? Read the robot data collection guide: teleoperation methods, episode counts and cost.
- Picking a file format? Read HDF5 vs RLDS vs LeRobot.
- Want to pretrain or benchmark on public data? See DROID, Open X-Embodiment and the dataset directory.
- Would rather buy the data? Read the data collection service buyer's guide or see our data services.
1. Collecting demonstrations
What matters: a collection interface that matches the task (leader-follower arms for contact-rich and bimanual work, handheld grippers for fast in-the-wild data, gloves for multi-finger hands), cameras that are calibrated and time-synchronized, and an episode protocol that every operator follows the same way.
| Hardware | Price | What it is for |
|---|---|---|
| OpenArm | $7,000.00 | Open-source arm built for teleoperation and imitation learning. Pair a leader and a follower for leader-follower collection |
| Trossen ALOHA Stationary | $27,999.00 | Two leader and two follower WidowX arms on a stationary frame, for bimanual demonstrations. See the ALOHA guide |
| RC UMI | $1,250.00 | Handheld gripper built to the open-source Universal Manipulation Interface design, with a GoPro wrist camera. Collects demonstrations without a robot |
| AgileX PiKA Data Gripper Kit | From $5,949.96 | Handheld collectors plus positioning stations that record synchronized 6D pose, gripper angle, RGB, depth and IMU |
| Wuji Glove | $4,500.00 | Per-finger tracking glove for teleoperating dexterous hands |
| RC Dex-UMI | $9,800.00 | UMI-style system for multi-finger hand data, with a LEAP-hand-compatible end effector |
The data collection guide compares the four teleoperation methods on quality, operator training time and throughput. If you collect with LeRobot, record a LeRobot dataset walks through the lerobot record workflow end to end. For a two-arm setup, the ALOHA guide covers the leader-follower rig and its HDF5 episode files.
2. Formats
What matters: what your training code reads natively, how large the dataset will get, and whether you will share it.
- HDF5 is one file per episode with arrays for joint states, actions and camera frames. It is what the original ALOHA and ACT code write and read.
- RLDS is Google's episode format built on TensorFlow Datasets (google-research/rlds). Open X-Embodiment and DROID are distributed in it, so you will meet it as soon as you pretrain on public data.
- LeRobotDataset stores tabular signals as Parquet, video as MP4 and the schema and normalization statistics as metadata, and loads straight from the Hugging Face Hub (format documentation).
Pick one format for training and convert at the edges. The format guide covers the trade-offs and the conversion paths between all three.
3. Public datasets
Public data rarely replaces your own demonstrations, but it is the usual starting point for pretraining and for benchmarking a pipeline before your rig is ready.
- DROID: about 76,000 teleoperated trajectories (350 hours) across 564 scenes and 84 tasks, all on Franka Panda arms (paper). The reference for diverse, in-the-wild single-arm manipulation.
- Open X-Embodiment: more than a million trajectories from 22 robot embodiments, pooled from 21 institutions (paper). The pretraining mix behind RT-X, Octo and OpenVLA.
- ALOHA datasets and the LeRobot collection: bimanual and low-cost-arm data in the formats LeRobot trains on directly.
The dataset directory lists the rest by task, robot and modality.
4. Quality control and curation
What matters: catching bad episodes before they reach training, labeling what each episode contains, and keeping enough variation in the dataset that the policy generalizes.
Check every episode for the basics: all camera streams present and in sync with the joint data, no frozen or black frames, the task completed, and no long idle stretches at the start or end. Label success and failure explicitly rather than deleting failures silently, because failed attempts are useful for evaluation and for training recovery behavior. The annotation guide covers success flags, language labels, contact events and task segmentation, including how to keep timestamps aligned across sensors.
Curation is mostly about coverage. A thousand nearly identical demonstrations teach less than three hundred that vary object position, lighting and object instance. The data collection guide gives episode-count targets by task complexity and a QA checklist. Once a policy is deployed, the data flywheel guide shows how to turn its failures into the next round of collection.
5. Build in-house or buy
Collecting in-house means buying stations, training operators and owning QA. That is the right call when the task, the robot or the data are core to your product. Buying collection as a service is faster when you need volume on a deadline or want to test a task before committing to hardware. The service buyer's guide lists the quality metrics, delivery formats and contract terms to ask any provider about, us included. For what we offer, see data services.







