Open X-Embodiment: What It Is and Why It Matters for Robot Learning

What Is Open X-Embodiment?

Open X-Embodiment is a unified dataset of robot manipulation demonstrations collected across more than 22 different robot embodiments — spanning arms from Franka, WidowX, UR5, Kuka, and others — and across dozens of research labs worldwide. The dataset totals over one million episodes covering hundreds of distinct manipulation tasks: picking, placing, opening drawers, pouring liquids, wiping surfaces, and more.

The "X" in the name stands for cross-embodiment: the defining ambition of OXE is to train policies that transfer knowledge across robot bodies. A policy pre-trained on the full OXE dataset has seen manipulation behavior from a wide range of arm geometries, gripper types, camera configurations, and task domains, giving it a rich prior that can be fine-tuned to a new robot with far fewer demonstrations than training from scratch.

Participating Institutions and Dataset Composition

Contributing institutions include Stanford, UC Berkeley, Google DeepMind, Carnegie Mellon, MIT, ETH Zurich, and many others. Each lab contributed their existing demonstration datasets in a standardized format. The dataset is hosted on Google Cloud Storage and is freely available for research use. Sub-datasets vary significantly in size: some labs contributed tens of thousands of episodes, others a few hundred. Task distribution is skewed toward tabletop pick-and-place, reflecting the most common experimental setup, but the diversity of objects, lighting conditions, and arm configurations is genuinely broad.

The Robotics Transformer 2 (RT-2) and subsequent models from Google were trained on OXE data and demonstrated that cross-embodiment pre-training produces policies with meaningfully better zero-shot generalization than single-robot training. This result validated the core OXE hypothesis and accelerated adoption of cross-embodiment datasets across the field.

Dataset Format and RLDS

OXE uses the RLDS (Robot Learning Dataset Specification) format, a TensorFlow Datasets-based schema for storing robot trajectories. Each episode in RLDS is a sequence of steps, where each step contains an observation dictionary (images, joint states, gripper state), an action vector, a reward signal, and a language annotation describing the task. The schema is flexible enough to accommodate different observation modalities and action spaces across embodiments.

Working with RLDS requires TensorFlow or the rlds_creator library. LeRobot from Hugging Face provides conversion utilities to transform OXE data into its own format, making it accessible to researchers who prefer PyTorch. SVRC's data platform exports datasets in a format compatible with both RLDS and LeRobot, enabling straightforward contribution to future OXE releases.

How to Contribute to OXE

Contributing your dataset to OXE requires formatting your demonstrations in RLDS, adding per-step language annotations, and submitting a pull request to the OXE GitHub repository with your dataset documentation. The submission process includes a review for data quality and format compliance. If your demonstrations were collected with SVRC data services, the platform can generate RLDS-compatible exports with standardized metadata, simplifying the contribution process significantly. Contact the SVRC team for guidance on preparing your data for OXE submission.

Using OXE for Pre-Training

The most practically valuable use of OXE is as a pre-training dataset. Download a subset of OXE relevant to your task domain and robot, train a general policy backbone, then fine-tune on your own task-specific demonstrations. This approach consistently requires fewer task-specific demonstrations than training from scratch — often 5–10x fewer — while achieving higher final performance.

Pre-training on OXE is most beneficial when your fine-tuning data is limited (under 100 episodes), when your tasks are conceptually similar to tasks in OXE, and when you are using an architecture designed for cross-embodiment transfer such as Octo, OpenVLA, or RT-2-X. Pure task-specific fine-tuning from scratch remains competitive when you have abundant high-quality demonstrations collected in deployment conditions.

SVRC Compatibility and How We Help

SVRC's data collection standard is designed to be OXE-compatible from the ground up: standardized camera placement, consistent annotation schema, quality-gated success labeling, and RLDS-ready export. Data collected through SVRC's data services can be used directly for OXE fine-tuning or contributed to future dataset releases. For teams that want to leverage OXE pre-trained models on their specific hardware, SVRC offers engineering support to set up the fine-tuning pipeline and evaluate deployment-ready policies.