Real-to-Sim Transfer

Real-to-sim transfer (the complement of sim-to-real) involves constructing or calibrating a simulation to match the real world as closely as possible — essentially building a digital twin of real conditions. This is used to replay real failure cases in simulation, generate additional synthetic training data matched to real sensor characteristics, and test policy updates safely before deployment. Techniques include photogrammetric scene reconstruction, physics parameter identification (system identification), and neural rendering methods (NeRF, 3D Gaussian Splatting) to match camera appearance. Accurate real-to-sim pipelines dramatically reduce the number of physical experiments needed for policy iteration.