RoboNet: The Original Cross-Robot Video Corpus

A practitioner's guide to RoboNet — the 15-million-frame, 7-robot, 4-lab dataset that established the template for every large-scale multi-robot release that followed.

TL;DR

Metric	Value
Frame count	~15,000,000 RGB video frames
Robots	7 (Sawyer, Baxter, WidowX, Kuka iiwa, Franka, Fetch, Widow200)
Modalities	RGB video (1-4 cameras), Cartesian delta action, proprioception, gripper
License	MIT
Institutions	4 labs (Penn, Stanford, UC Berkeley, CMU)
Original paper	CoRL 2019 (Dasari et al.)

What is RoboNet?

RoboNet was the first serious attempt to answer a very specific question: if several labs pool their autonomously-collected robot video, can a single video prediction model generalize across robots and environments? Sudeep Dasari and collaborators at Penn, Stanford, UC Berkeley, and CMU pooled 15 million video frames from 7 different robot platforms and released them with a unified HDF5 schema — a remarkable achievement in 2019, when robot datasets were usually lab-private and protocol-specific.

Critically, RoboNet is a video prediction dataset, not a language-conditioned manipulation dataset. Most episodes are autonomously collected (random or scripted exploration in front of varied objects), not teleoperated expert demonstrations. That design choice made RoboNet the reference benchmark for stochastic video models (SVG, SVP, SV2P), visual foresight planners, and world-model-based control for years, and it shaped the way the field now thinks about cross-robot action-conditioned prediction.

Although RoboNet has since been dwarfed by Open X-Embodiment and DROID in raw size, it remains the cleanest, smallest, and most pedagogically useful cross-robot dataset. Its schema — Cartesian delta actions, per-robot embodiment tag, 64x64 or 128x128 RGB — is still the easiest one to stand up a cross-robot video prediction baseline on a single GPU.

How to download & load

The canonical distribution is a set of HDF5 shards on the RoboNet wiki, with a TFRecord mirror shipped via the reference GitHub repo:

# Pull the reference code
git clone https://github.com/SudeepDasari/RoboNet.git
cd RoboNet && pip install -e .

# Download the HDF5 shards (see https://www.robonet.wiki/ for bucket URLs)
python robonet/datasets/download_robonet.py --output_dir ./data

# Load episodes with the built-in reader
from robonet.datasets.robonet_dataset import RoboNetDataset
ds = RoboNetDataset(
    batch_size=32,
    dataset_files=["./data/*.hdf5"],
    hparams={"img_size": [64, 64], "load_random_cam": True},
)
for batch in ds:
    print(batch["images"].shape, batch["actions"].shape, batch["states"].shape)

Because frames are small (64x64 or 128x128) and action vectors are short, RoboNet is one of the rare robot datasets where you can run a meaningful experiment on a single consumer GPU.

Common use cases & model pairings

• Video prediction baselines. SVG, SVP, FitVid, and the original Visual Foresight stack all use RoboNet as their cross-robot reference.
• World-model-based control. The paired action/observation sequences make it a natural testbed for Dreamer-style world models in a multi-embodiment setting.
• Embodiment encoder research. Because every shard is tagged with a robot ID, RoboNet is a clean target for learning embodiment embeddings that transfer to new arms.
• Teaching / coursework. The small frame size and clean HDF5 schema make RoboNet the default dataset in graduate robot learning classes.

Benchmarks & leaderboards

There is no closed leaderboard, but the standard evaluation is PSNR / SSIM / LPIPS on 10-step future-frame prediction conditioned on a held-out action sequence, reported separately for seen and unseen robots. See the Papers with Code RoboNet entry, the official wiki, and the Dasari et al. CoRL 2019 paper for reference numbers.

Technical deep dive: Cartesian delta actions and embodiment IDs

RoboNet's action representation was unusually ambitious for 2019. Rather than record raw joint targets or motor commands (which would differ between every robot), the authors mapped every action onto a 5-dimensional Cartesian delta: xyz translation plus yaw rotation plus a binary gripper bit. Pitch and roll were held fixed because most contributing robots only supported top-down grasping in the collection setup. This normalization is what enables a single video prediction model to be conditioned on actions from seven different arms.

Each trajectory is tagged with an integer embodiment ID that policies can either ignore (to learn a generic cross-robot model) or condition on (to exploit robot-specific dynamics). The latter approach — learned embodiment embeddings — became one of the dominant patterns in the follow-on literature and directly inspired the per-dataset embeddings used in Octo and the action normalization scheme in OpenVLA.

Video frames are stored at low resolution (64x64 or 128x128 depending on the shard) specifically so that video prediction baselines can run on commodity hardware. If you need higher-resolution footage, Open X-Embodiment and DROID both ship full-resolution RGB and should be preferred for modern vision-heavy pipelines.

Known limitations

• Low resolution. 64x64 or 128x128 frames are unusable for modern pixel-based policies. RoboNet is a historical reference, not a training target for production systems.
• Autonomous (not expert) data. Most trajectories are random exploration, so RoboNet is not suitable for imitation learning — it was never meant to be.
• Top-down action space only. The 5-DoF normalization drops pitch and roll, which limits the manipulation repertoire to planar grasping.
• No language annotations. Episodes are tagged with robot and environment IDs but no natural language, so RoboNet cannot pretrain language-conditioned policies.

FAQ

Is RoboNet still worth using in 2026? Yes, for two reasons: it is still the best teaching dataset for cross-embodiment video prediction, and its small size makes it ideal for ablation studies that would be too expensive on Open X-Embodiment.

How does RoboNet compare to Open X-Embodiment? OXE is 50-100x larger, includes expert demonstrations, and ships language annotations. RoboNet is smaller, video-only, and autonomous-exploration-based. They are complementary.

Can I combine RoboNet with modern action-labeled datasets? Yes — the embodiment-tagged action schema is compatible with most modern VLA training stacks with minor adapter code.