How to Collect Robot Training Data

Complete guide to collecting high-quality robot training data via teleoperation — hardware setup, LeRobot recording, RLDS format, quality scoring, and dataset preparation for VLA fine-tuning.

Intermediate ⏱ 2–4 hours setup Updated April 2026

1. Hardware 2. Install 3. Configure 4. Teleop Setup 5. Record 6. Review 7. RLDS Convert 8. Upload 9. Quality Checks 10. Prep for VLA

Prerequisites

A robot arm (OpenArm, SO-100, Aloha, Koch, UR5, or similar)
At least one RGB camera (wrist-mounted preferred) + one external camera
A teleoperation input device (leader arm, SpaceMouse, or keyboard)
ROS2 Humble installed (see our ROS2 setup guide)
Python 3.10+ with pip
Ubuntu 22.04 with NVIDIA GPU (for visualization and future training)

What you will build

By the end of this tutorial, you will have a complete data collection pipeline: teleoperation hardware configured, LeRobot recording episodes of your robot performing tasks, quality-scored demonstrations, and a dataset in RLDS format uploaded to HuggingFace Hub — ready for VLA fine-tuning.

Data Collection Pipeline

Teleop Input
Leader arm / SpaceMouse

→

Robot Arm
Joint states + actions

→

Cameras
Wrist + external RGB(D)

→

LeRobot
Record & sync

→

RLDS Dataset
HuggingFace Hub

Hardware Requirements

A complete data collection setup has four components. Here is what you need and our recommended options:

Component	Recommended	Budget
Robot arm	OpenArm$5,400.00 or Aloha ($20K+)	SO-100 ($200 DIY kit)
Wrist camera	Intel RealSense D405 ($450.00)	USB webcam ($30)
External camera	Intel RealSense D435 ($550.00)	Logitech C920 ($70)
Teleop device	Leader arm (matched pair)	3Dconnexion SpaceMouse ($130)

Budget path: An SO-100 arm with two webcams and a SpaceMouse gets you collecting data for under $500 total. Quality is lower than a leader-follower setup, but it is enough to learn the workflow and train simple policies.

Install LeRobot

LeRobot is Hugging Face's open-source toolkit for robot learning. Install it in a virtual environment to avoid dependency conflicts.

Copy# Create and activate virtual environment python3 -m venv ~/lerobot_env source ~/lerobot_env/bin/activate # Install LeRobot with all extras for data collection pip install "lerobot[all]" # Or install from source for latest features git clone https://github.com/huggingface/lerobot.git cd lerobot pip install -e ".[all]" # Verify installation python3 -c "import lerobot; print(f'LeRobot {lerobot.__version__} installed')"

Copy# Install additional dependencies for camera recording pip install opencv-python pyrealsense2 h5py # For HuggingFace Hub uploads pip install huggingface_hub huggingface-cli login

Configure Your Robot in LeRobot

LeRobot uses YAML configuration files to define your robot's properties. Create a config for your specific hardware.

Copy# Create robot config directory mkdir -p ~/lerobot_configs # Example config for OpenArm (save as openarm.yaml) cat > ~/lerobot_configs/openarm.yaml << 'EOF' robot: type: openarm port: /dev/ttyUSB0 baudrate: 1000000 joints: - name: shoulder_pan id: 1 min: -3.14 max: 3.14 - name: shoulder_lift id: 2 min: -1.57 max: 1.57 - name: elbow id: 3 min: -2.35 max: 2.35 - name: wrist_pitch id: 4 min: -1.57 max: 1.57 - name: wrist_roll id: 5 min: -3.14 max: 3.14 - name: gripper id: 6 min: 0.0 max: 1.0 cameras: wrist: index: 0 width: 640 height: 480 fps: 30 external: index: 2 width: 640 height: 480 fps: 30 recording: fps: 30 task: "pick_and_place" EOF

Important: Camera indices (index: 0, index: 2) vary by system. Run v4l2-ctl --list-devices to find the correct indices for your cameras. Get this right before recording — wrong camera indices are the most common setup error.

Set Up Teleoperation Interface

You have two main options for controlling the robot during demonstrations: a leader-follower arm pair (best quality) or a SpaceMouse (more accessible).

Option A: Leader-Follower Arms (recommended)

Copy# Configure leader arm (the one you physically move) lerobot calibrate \ --robot-type=openarm \ --robot-port=/dev/ttyUSB0 \ --leader-port=/dev/ttyUSB1 # This runs a calibration routine — follow the on-screen prompts # to move both arms through their range of motion

Option B: SpaceMouse

Copy# Install SpaceMouse driver pip install pyspacemouse # Test SpaceMouse connection python3 -c "import pyspacemouse; m = pyspacemouse.open(); print('SpaceMouse connected')" # Configure SpaceMouse in LeRobot lerobot calibrate \ --robot-type=openarm \ --robot-port=/dev/ttyUSB0 \ --teleop=spacemouse

Which to choose? Leader-follower arms produce more natural demonstrations because your hand movements map directly to the robot's joints. SpaceMouse is easier to set up but produces less smooth data, which can reduce policy performance by 10–20%. For production data collection, always use leader-follower.

Record First Episodes

An "episode" is one complete demonstration of a task from start to finish. Start with a simple task like picking up a block and placing it in a target zone.

Copy# Record episodes to a local dataset lerobot record \ --robot-type=openarm \ --robot-port=/dev/ttyUSB0 \ --leader-port=/dev/ttyUSB1 \ --fps=30 \ --task="pick_red_block_place_target" \ --num-episodes=10 \ --output-dir=~/datasets/openarm_pick_place # Controls during recording: # Enter — start/stop episode # Space — pause/resume # r — redo current episode # q — quit recording

Each episode is saved with synchronized data:

~/datasets/openarm_pick_place/ ├── episode_000/ │ ├── observations/ │ │ ├── joint_positions.npy # (T, 6) joint angles │ │ ├── joint_velocities.npy # (T, 6) joint velocities │ │ ├── wrist_cam.mp4 # wrist camera video │ │ └── external_cam.mp4 # external camera video │ └── actions.npy # (T, 7) target joint positions + gripper ├── episode_001/ ├── ... └── metadata.json

Episode count guidelines: For a single-task policy (ACT, Diffusion Policy), collect 50–200 episodes. For VLA fine-tuning (OpenVLA, pi0), collect 300–1,200 episodes. More diverse demonstrations yield better generalization. Vary the object position, orientation, and lighting between episodes.

Review and Quality-Score Episodes

Not every episode is usable. Review your recordings, remove failed demonstrations, and compute quality metrics.

Copy# Visualize a specific episode lerobot visualize-dataset \ --dataset-path=~~/datasets/openarm_pick_place \ --episode=0 # Replay the episode on the robot (optional — for verification) lerobot replay \ --robot-type=openarm \ --robot-port=/dev/ttyUSB0 \ --dataset-path=~~/datasets/openarm_pick_place \ --episode=3

Mark failed episodes for removal:

Copy# Compute quality metrics for all episodes python3 -c " import numpy as np import json, os, glob dataset_dir = os.path.expanduser('~/datasets/openarm_pick_place') episodes = sorted(glob.glob(os.path.join(dataset_dir, 'episode_*'))) for ep_dir in episodes: actions = np.load(os.path.join(ep_dir, 'actions.npy')) # Smoothness: lower jerk = better demonstration jerk = np.diff(actions, n=3, axis=0) smoothness = 1.0 / (1.0 + np.mean(np.abs(jerk))) length = len(actions) ep_name = os.path.basename(ep_dir) print(f'{ep_name}: length={length}, smoothness={smoothness:.3f}') "

Quality rule of thumb: Aim for 90%+ task success rate in your final dataset. Discard episodes where the robot dropped the object, collided with the table, or had jerky movements. A smaller set of high-quality demonstrations outperforms a larger noisy dataset.

Convert to RLDS Format

RLDS (Reinforcement Learning Datasets) is the standard format for VLA model training. Convert your LeRobot dataset to RLDS for compatibility with OpenVLA, Octo, and RT-2.

Copy# Install RLDS conversion tools pip install tensorflow tensorflow-datasets rlds # Convert LeRobot dataset to RLDS python3 -m lerobot.scripts.convert_to_rlds \ --input-dir=~~/datasets/openarm_pick_place \ --output-dir=~~/datasets/openarm_pick_place_rlds \ --task-description="Pick up the red block and place it on the green target zone"

The RLDS dataset contains TFRecord files with this structure per timestep:

{ "observation": { "image": (480, 640, 3) uint8, # wrist camera "image_external": (480, 640, 3) uint8, # external camera "state": (6,) float32 # joint positions }, "action": (7,) float32, # target joints + gripper "reward": float32, # 1.0 for success, 0.0 otherwise "is_terminal": bool, "language_instruction": string # task description }

Upload to HuggingFace Hub

Push your dataset to HuggingFace Hub for versioning, easy sharing, and direct loading during training.

Copy# Make sure you're logged in huggingface-cli login # Push LeRobot format dataset lerobot push-to-hub \ --dataset-path=/datasets/openarm_pick_place \ --repo-id="your-username/openarm-pick-place-v1" \ --private # Or push RLDS format dataset python3 -c " from huggingface_hub import HfApi api = HfApi() api.upload_folder( folder_path=os.path.expanduser('/datasets/openarm_pick_place_rlds'), repo_id='your-username/openarm-pick-place-rlds-v1', repo_type='dataset', private=True ) print('Upload complete!') "

Versioning tip: Always include a version suffix (-v1, -v2). When you collect more data or fix quality issues, push as a new version. This makes it easy to reproduce training runs and track improvements.

Run Quality Checks

Before training, run a final set of quality checks to catch issues that could waste GPU hours.

Copypython3 -c " import numpy as np import json, os, glob dataset_dir = os.path.expanduser('~/datasets/openarm_pick_place') episodes = sorted(glob.glob(os.path.join(dataset_dir, 'episode_*'))) lengths = [] smoothness_scores = [] action_ranges = [] for ep_dir in episodes: actions = np.load(os.path.join(ep_dir, 'actions.npy')) lengths.append(len(actions)) jerk = np.diff(actions, n=3, axis=0) smoothness_scores.append(1.0 / (1.0 + np.mean(np.abs(jerk)))) action_ranges.append(actions.max(axis=0) - actions.min(axis=0)) print(f'Episodes: {len(episodes)}') print(f'Avg length: {np.mean(lengths):.0f} steps ({np.mean(lengths)/30:.1f}s at 30fps)') print(f'Length std: {np.std(lengths):.0f} steps') print(f'Avg smoothness: {np.mean(smoothness_scores):.3f}') print(f'Action range diversity: {np.std(np.stack(action_ranges), axis=0).mean():.4f}') # Flag potential issues if np.std(lengths) / np.mean(lengths) > 0.5: print('WARNING: High variance in episode lengths — check for stuck episodes') if np.mean(smoothness_scores) < 0.3: print('WARNING: Low smoothness — consider re-collecting with slower movements') if len(episodes) < 50: print('WARNING: Fewer than 50 episodes — may not be enough for robust training') print('Quality checks complete.') "

Prepare for VLA Fine-Tuning

Structure your dataset for training by creating proper splits and task descriptions.

Copy# Create train/val split (90/10) python3 -c " import os, shutil, random, glob dataset_dir = os.path.expanduser('~/datasets/openarm_pick_place') episodes = sorted(glob.glob(os.path.join(dataset_dir, 'episode_*'))) random.seed(42) random.shuffle(episodes) split = int(len(episodes) * 0.9) train = episodes[:split] val = episodes[split:] # Write split files with open(os.path.join(dataset_dir, 'train_episodes.txt'), 'w') as f: f.write('\n'.join([os.path.basename(e) for e in train])) with open(os.path.join(dataset_dir, 'val_episodes.txt'), 'w') as f: f.write('\n'.join([os.path.basename(e) for e in val])) print(f'Train: {len(train)} episodes, Val: {len(val)} episodes') "

Ready to fine-tune a VLA model?

Your dataset is prepared. Continue to the VLA fine-tuning guide to train OpenVLA or pi0 on your data. Expect to invest $150–400 in GPU compute for a typical fine-tuning run.

Data Collection Cost Breakdown

Cost Component	Per Hour	Notes
Operator labor	$50–80	Skilled teleoperator, varies by market
Hardware depreciation	$15–30	Amortized over 2,000 hrs of use
Quality review	$20–40	Episode filtering and scoring
Compute and storage	$8–15	Recording, processing, storage
Facility overhead	$25–35	Lab space, lighting, safety
Total	$118–200

Need help collecting data?

RCSV Data Services offers managed robot data collection with professional teleoperators, quality assurance, and RLDS-ready delivery. Starting at $150/hr.

Learn About Data Services

Designing and Scaling a Demonstration Dataset

The steps above get one robot recording. The practices below decide whether the dataset trains a useful policy.

Episode Design

An "episode" is a single demonstration of the task from start to finish. How you design episodes determines the distribution your policy learns from — and therefore what it can and cannot do at deployment time.

Defining the Task

Write a precise task specification before collecting any data. Include:

Task goal: "Pick up the red cube from the table and place it in the blue bowl." Be specific about which object, where it starts, and where it ends.
Success criteria: "The cube is inside the bowl, the gripper is open, and the arm has returned to home position." Define success unambiguously so every operator and every QA reviewer applies the same standard.
Allowed strategies: "Approach the cube from above with a top-down grasp." Or explicitly: "Any grasp approach is acceptable." If you constrain strategy, you get more consistent data (better for ACT). If you allow multiple strategies, you need more data but the policy may be more robust (better for Diffusion Policy).
Episode duration: Define a maximum episode length. Typical tabletop pick-place: 5–15 seconds. If an episode exceeds 2x the median duration, something went wrong — discard it.

Start State Distribution

The start state distribution determines your policy's deployment robustness. If every demonstration starts with the object in the exact same position, the policy will fail when the object is 3 cm to the left.

Phase 1 (first 50 demos): Use a small start-state distribution. Vary object position within a 5 cm radius of a central point. This ensures consistent demonstrations while the operator builds skill.
Phase 2 (demos 51–200): Expand to the full intended deployment distribution. Vary object position across the entire reachable workspace. Vary object orientation if relevant.
Phase 3 (demos 200+): Add distractors, object variants, lighting variation, and background changes. This phase targets generalization.

Reset Protocol

Between episodes, you must reset the scene to a valid start state. Define the reset protocol explicitly:

Return robot to home configuration (predefined joint positions).
Place the task object in a new position within the start-state distribution.
Verify the scene matches the task specification (object visible, no occlusions, gripper open).
Wait 1 second for the scene to settle (no residual object motion).
Begin recording.

Sloppy resets are the number-one source of bad episodes. Automate the reset where possible — at minimum, automate the robot returning to home position.

Quality Control for Each Episode

Every episode should pass this 8-point quality checklist before being added to the training dataset. Reviewing takes 30–60 seconds per episode and prevents weeks of debugging bad policies later.

8-Point Episode QA Checklist

Task completed successfully? The episode ends with the task goal achieved per the success criteria. Discard partial completions unless you are specifically collecting recovery data.
No operator hesitation >2 seconds? Long pauses mid-task create "stationary" data points that confuse the policy. If the operator paused to think, re-record.
Smooth trajectory? Play back the joint position data. Look for sudden jumps (teleoperation glitches), oscillations (controller instability), or unnatural speed changes.
Camera images clear? Check for motion blur, occlusion of the manipulation point, and correct exposure. If the wrist camera image is blurry during the grasp, that episode is useless for visual policy learning.
Correct start state? Verify the episode starts from within the intended start-state distribution. An episode that starts from a weird configuration introduces out-of-distribution data.
Episode duration within bounds? Duration should be within 2 standard deviations of the mean. Anomalously short episodes may indicate incomplete tasks. Anomalously long episodes indicate operator difficulty.
Data integrity? HDF5 file is not corrupted. All expected data fields are present. Timestamps are monotonically increasing. No NaN values in joint data.
Consistent strategy? If you are constraining strategy (recommended for ACT), verify the episode follows the designated approach. Mixed strategies in a small dataset cause mode averaging failures.

Common Data Collection Mistakes

1. Inconsistent Camera Position Between Sessions

What happens: You collect 100 episodes on Monday, then bump the overhead camera while cleaning up. Tuesday's 100 episodes have a slightly different viewpoint. The mixed dataset trains a policy that performs poorly from both viewpoints.

Fix: Rigidly mount cameras. Photograph the camera positions. At the start of each session, verify camera extrinsics against a reference ArUco board calibration. If calibration has shifted, re-mount before collecting.

2. Operator Fatigue Degrading Data Quality

What happens: The first 50 episodes of the day are smooth and precise. By episode 150, the operator is tired — demonstrations are jerkier, slower, and less consistent. The policy learns an average of good and bad demonstrations.

Fix: Enforce 10-minute breaks every 45 minutes. Limit collection sessions to 4 hours per operator per day. Track quality metrics (episode duration, trajectory smoothness) over time and stop collection when quality degrades.

3. Narrow Start-State Distribution

What happens: All demonstrations start with the object in roughly the same position. The policy achieves 90% success at that position but 10% success at any other position.

Fix: Systematically randomize object positions during collection. Use a grid pattern: divide the workspace into a 4x4 grid and collect at least 3 episodes starting from each grid cell. Track start-state coverage as a dataset metric.

4. Not Validating the Recording Pipeline Before Scaling

What happens: You collect 500 episodes over a week, then discover the action data was recorded at 25 Hz instead of 50 Hz due to a configuration error. The entire dataset is unusable for your target policy.

Fix: Always validate with 10 test episodes first. Run the complete training pipeline on those 10 episodes. Verify data dimensions, frequencies, and format compatibility before investing in large-scale collection.

5. Mixed Strategies in a Small Dataset

What happens: Three operators each use a different grasp approach. With only 150 total episodes (50 per approach), no single approach has enough data. ACT averages the approaches and produces an invalid middle trajectory.

Fix: For datasets under 300 episodes, standardize on a single strategy. Train operators to use the same approach. For larger datasets (300+), multiple strategies are fine — consider Diffusion Policy which handles multi-modality natively.

6. Ignoring Lighting Variation

What happens: All data is collected under consistent lab lighting. The policy fails when deployed in a slightly different lighting condition (afternoon sun, different overhead light).

Fix: During Phase 2 and 3 collection, deliberately vary lighting. Turn off overhead lights for some episodes. Add a desk lamp from different angles. Close/open window blinds. This forces the visual encoder to learn lighting-invariant features.

Scaling from 50 to 500 Episodes

Scaling data collection introduces organizational challenges that do not exist at small scale. Here is how to manage the transition:

Operator Training

When adding new operators beyond the original 1–2:

Have new operators watch 10 example episodes before they start.
Require 20 practice episodes that pass QA before counting toward the dataset.
Pair new operators with an experienced operator for the first session.
Track per-operator quality metrics: mean episode duration, trajectory smoothness score, QA rejection rate.

Shift Scheduling

For large-scale collection (500+ episodes):

Schedule 2–3 operators per day in 3-hour shifts with 30-minute overlap for handoff.
Each shift produces approximately 40–60 episodes (at 3–4 minutes per episode including reset and QA).
Designate one person as the "data lead" who reviews quality metrics at end of each day and flags degradation.

Quality Degradation Detection

Monitor these metrics daily during large-scale collection:

Mean episode duration: Should remain stable (±10%). Increasing duration indicates operator fatigue or increasing task difficulty.
QA rejection rate: Should stay below 15%. If it rises above 20%, stop collection and investigate.
Trajectory smoothness: Compute the jerk (third derivative of position) of each episode. Flag episodes with mean jerk >2x the dataset average.
Start-state coverage: Plot start positions on a 2D heatmap. Verify uniform coverage across the intended distribution. Redirect operators to under-sampled regions.

Troubleshooting

Camera feed is black or frozen

Check camera index with v4l2-ctl --list-devices. Try unplugging and replugging the USB cable. If using RealSense, verify with realsense-viewer. USB 3.0 ports are required for RealSense cameras.

Robot arm not responding during recording

Verify the serial port with ls /dev/ttyUSB*. Check permissions: sudo chmod 666 /dev/ttyUSB0. Make sure no other process is using the port (close RViz, other terminals).

Frame drops during recording (inconsistent fps)

Lower camera resolution to 480x360. Close background applications. Use a dedicated USB bus for each camera (check with lsusb -t). Record to an SSD, not an HDD.

RLDS conversion fails with shape mismatch

This usually means episodes have different observation dimensions. Check that all cameras used the same resolution across all episodes. Re-record any episodes with mismatched dimensions.

HuggingFace upload times out

Large datasets (50+ GB) can time out on slow connections. Use huggingface-cli upload with --revision to upload in parts, or compress videos before upload. Consider uploading from a cloud VM with faster bandwidth.

Frequently Asked Questions

How many demonstration episodes do I need to train a robot policy?

For most single-task manipulation policies (pick and place, pouring, etc.), 50–200 high-quality episodes are enough for ACT or Diffusion Policy. For VLA fine-tuning (OpenVLA, pi0), plan for 300–1,200 episodes depending on task complexity and required generalization.

What is the cost of collecting robot training data?

The fully-loaded cost of robot teleoperation data collection ranges from $118–200 per hour in 2026, depending on operator skill level and hardware. This includes operator time, hardware depreciation, compute, and quality review. RCSV Data Services offers managed collection starting at $150/hr.

What cameras should I use for robot data collection?

At minimum, use one wrist-mounted RGB camera (640x480, 30fps) and one external third-person view camera. For best results, add an RGBD camera like the Intel RealSense D435 for depth data. Many VLA models expect both wrist and external camera views.

What is the RLDS format and why does it matter?

RLDS (Reinforcement Learning Datasets) is a standardized format from Google DeepMind for storing robot learning data. It stores episodes as sequences of observations, actions, and rewards in TFRecord files. Most VLA models (RT-2, OpenVLA, Octo) expect RLDS format, making it the de facto standard for interoperability.

How do I ensure my collected data is high quality?

Track three quality metrics: (1) Task success rate — only keep episodes where the task was completed successfully. (2) Trajectory smoothness — filter out jerky or erratic demonstrations. (3) Diversity — vary object positions, orientations, and lighting across episodes. Aim for 90%+ success rate in your final dataset.

Was this tutorial helpful?

👍 Yes 👎 No

Stay Ahead in Robotics

Get the latest on robot deployments, data collection, and physical AI — delivered to your inbox.