MimicGen: Synthetic Demonstration Generation

TL;DR

Metric	Value
Task coverage	18 manipulation tasks across 4 embodiments
Robots	Franka Panda, Sawyer, UR5e, bimanual Panda (robosuite & Isaac Lab)
Modalities	Low-dim state, RGB (agent + wrist), object poses, subtask segmentation
License	NVIDIA Source Code License
Size	50,000+ synthetic demos generated from ~200 human seeds
Built on	robomimic + robosuite + Isaac Lab

What is MimicGen?

MimicGen is a system, not just a dataset. Introduced at CoRL 2023 by Ajay Mandlekar and collaborators at NVIDIA and UT Austin, it tackles the single largest bottleneck in imitation learning: human teleoperation time. The insight is that a manipulation demonstration can be decomposed into object-centric subtask segments — for example, "approach cup," "grasp cup," "lift cup," "place on saucer" — and each segment can be re-anchored to a newly sampled object pose and replayed in simulation. Given a small seed of expert demonstrations (roughly 10 per task), MimicGen generates hundreds of additional successful trajectories per task with only a small filter on simulation rollout success.

The released MimicGen datasets ship ~50,000 synthetic demonstrations across 18 tasks and 4 embodiments, covering contact-rich manipulation such as nut assembly, coffee preparation, kitchen cleanup, and bimanual hammer swing. Policies trained on MimicGen data reach the same success rates as policies trained on 10x more hand-collected human data, and the compute cost is trivial: a single workstation can generate 10,000 demos overnight.

MimicGen has become the default data-scaling recipe for the robosuite ecosystem. LIBERO, RoboCasa, and many follow-on papers use MimicGen-style generation to extend their benchmarks beyond what human teleoperation alone could support, and the method is being ported to Isaac Lab for large-scale GPU-parallel generation.

How to download & load

The released synthetic datasets are HDF5 files in the standard robomimic format, which means any robomimic-compatible algorithm can train on them without modification:

# Install MimicGen and its dependencies
git clone https://github.com/NVlabs/mimicgen.git
cd mimicgen && pip install -e .
# Also install robomimic and robosuite
pip install robomimic robosuite

# Download the pre-generated datasets (~60 GB)
python mimicgen/scripts/download_datasets.py --dataset_type core --tasks all

# Train Diffusion Policy on a MimicGen dataset
python -m robomimic.scripts.train \
  --config configs/diffusion_policy.json \
  --dataset datasets/mg_square_d1/image_v141.hdf5

# Or generate your own synthetic demos from a seed
python mimicgen/scripts/generate_dataset.py \
  --config mimicgen/exps/templates/robosuite/square.json \
  --n_trajectories 1000 \
  --seed_data datasets/square/ph/low_dim_v141.hdf5

The generate_dataset.py script is the interesting one — it runs the subtask decomposition, resamples object poses, replays the adapted trajectories in simulation, filters by success, and saves a fresh HDF5 that is ready for robomimic training.

Common use cases & model pairings

• Scaling imitation learning. MimicGen is the fastest way to 50x your demonstration count without additional human teleoperation.
• Diffusion Policy augmentation. Diffusion Policy on MimicGen data consistently outperforms Diffusion Policy on the original robomimic seed by 15-25 points.
• Bimanual task research. The bimanual Panda tasks are some of the only large-scale bimanual imitation datasets in simulation.
• Benchmark extension. LIBERO and RoboCasa both use MimicGen-style generation internally to scale their evaluation suites.

Benchmarks & leaderboards

MimicGen is usually evaluated by holding the generation recipe fixed and comparing downstream behavioral cloning performance across data scales (10, 100, 1K, 10K demos). The CoRL 2023 paper and follow-ons report 20-30 point improvements over human-only baselines at the same eval protocol. See the Papers with Code MimicGen entry, the official project page, and the Mandlekar et al. CoRL 2023 paper for canonical numbers.

Technical deep dive: subtask decomposition

The core algorithmic idea behind MimicGen is that a demonstration can be cut into object-centric subtask segments, where each segment is parameterized by the pose of a specific object in the scene. Given a source demonstration, MimicGen identifies the active object for each subtask, transforms the end-effector trajectory into that object's frame, and re-applies it to a new sampled object pose in a new scene. A physics replay in robosuite or Isaac Lab then checks whether the adapted trajectory is still successful.

This trick works because most manipulation subtasks really are object-centric: the motion you execute to grasp a cup depends on the pose of the cup, not on the pose of the table. It breaks down when subtasks are hand-centric (moving your own wrist to a specific pose regardless of objects) or when multiple objects interact in a non-separable way. For those cases, MimicGen exposes hooks for custom subtask definitions.

The generation pipeline is embarrassingly parallel — every synthetic demo is independent of every other — so you can scale to 10,000+ demos overnight on a single workstation with 32 CPU cores. For even larger scales, the team has been porting MimicGen to Isaac Lab to exploit GPU-parallel physics, with reported speedups of 10-20x.

Known limitations

• Simulation only. MimicGen generates demos inside a physics simulator. Sim-to-real transfer is not handled by MimicGen itself and requires a separate randomization or domain-adaptation step.
• Requires a task XML. Every target task must be defined as a robosuite or Isaac Lab scene. Real-world-only tasks are out of reach.
• Subtask segmentation is manual. Seed demos must be annotated with subtask boundaries. This is usually a 30-minute task per new benchmark but is not automatic.
• Failure modes under extreme domain shift. When the resampled object pose is too far from the seed distribution, the replay fails and the demo is discarded. Success rate drops sharply at pose distributions beyond roughly 2x the seed range.

FAQ

Can I use MimicGen for real-world data? Not directly. The replay mechanism requires a physics simulator. You can, however, use MimicGen-generated sim data as a pretraining prior and then fine-tune on a small real-world dataset.

How does MimicGen compare to DAgger or behavior cloning with noise injection? MimicGen produces larger volumes of diverse but on-policy-equivalent data, whereas DAgger requires an online expert. For offline imitation learning, MimicGen is strictly more data-efficient.

Does MimicGen work with bimanual tasks? Yes — the released datasets include bimanual Panda tasks, and the subtask decomposition extends naturally to dual-arm episodes.

Related datasets

• Robomimic — the seed dataset and training framework MimicGen is built on
• LIBERO — lifelong benchmark that uses MimicGen-style generation internally
• CALVIN — language-conditioned long-horizon simulation
• Open X-Embodiment — real-world cross-embodiment corpus
• ALOHA — bimanual real-world complement for MimicGen bimanual sim tasks