Introduction
Sim-to-real transfer is still one of the hardest problems in applied robotics. The gap between how physics behaves in simulation and how it behaves in the real world is real, persistent, and task-dependent. But the teams that consistently transfer policies successfully share a set of practices that dramatically reduce the gap. This post compiles the most reliable ones.
Tip 1: Use Depth Images, Not RGB
Photorealistic RGB rendering in simulation still does not match the real world. Lighting models, material shaders, and shadow algorithms produce images that look similar to a human but activate neural network feature detectors differently than real images. Depth images (structured light or LiDAR-based) have a much smaller sim-to-real gap because depth is a more direct representation of geometry, which simulation renders accurately.
Teams that switched from RGB to depth-primary representations (using RGB only as a secondary input or for semantic information) consistently report 20-40% improvement in zero-shot sim-to-real transfer on object-grasping tasks.
Tip 2: Aggressive Domain Randomization on Camera Parameters
Domain randomization works, but only if the randomization range covers the real-world variation. Most teams under-randomize. Effective ranges for successful sim-to-real: ±10cm camera position offset from nominal, ±10 degrees viewing angle, ±30% brightness, ±20% hue shift, ±15% saturation, Gaussian blur with sigma 0-2px, and random occlusions covering 5-20% of the image.
This level of randomization makes sim images look unrealistically bad — but that is the point. The policy has to learn features that are invariant to all of these variations, which means it learns features that are also invariant to the real-world differences between sim and real.
Tip 3: Add Action Noise During Sim Training
Policies trained with perfect action execution in simulation learn to rely on that perfection — then fail when real actuators add 5-10ms latency and 0.5-2% position error. Fix: inject Gaussian noise on joint position commands during training (σ = 0.01 rad for most 7-DOF arms). This forces the policy to be robust to small execution errors, which directly translates to robustness to actuator noise in the real world.
Tip 4: Measure and Match Your Robot's Actual Physics
Default simulation parameters (joint stiffness, damping, friction, inertia tensors) often differ from the real robot by 10-50%. Before training, spend 2-4 hours doing system identification: move each joint through its range and measure the actual torque-position relationship. Use these measured values in your simulation model. This step alone often reduces sim-to-real error by 30-50% on contact tasks.
Tip 5: Fine-Tune on 100 Real Demonstrations After Sim Pre-Training
Simulation pre-training + real fine-tuning is not a compromise — it is often the best-performing approach, outperforming both sim-only and real-only training when data is limited. The sim policy learns the broad structure of the task (approach, pre-grasp, grasp initiation). The real fine-tuning data corrects the contact model and calibrates for the specific robot and environment.
100 real demonstrations after sim pre-training typically outperforms 10,000 sim-only demonstrations on contact-rich tasks. This ratio makes the hybrid approach economically attractive: the sim training is nearly free, and 100 real demos is an achievable collection target.
Tip 6: Use Systematic Perturbation Testing, Not Random Evaluation
When evaluating sim-to-real transfer, random evaluation trials obscure structure in the failure modes. Instead, test at 5-10 specific challenging positions: extreme workspace corners, objects near the edge of reachable space, objects at atypical heights or orientations. This structured evaluation reveals whether failures are concentrated at specific conditions (diagnosable) or randomly distributed (much harder to fix).
Tip 7: Add Tactile or Force Sensing in Real Even Without Sim Support
If your real robot has a wrist force-torque sensor or tactile fingertips, use them — even if your simulation does not model them. Train the policy to use these sensors through a residual module: the base policy from simulation provides a nominal action, and a small force/torque-conditioned network provides a correction term. This architecture means the base policy still works without the sensor signal, but the real deployment gets the additional robustness of contact sensing.
Tip 8: Log Sim Failures and Match to Real Failure Modes
Before deploying to real hardware, run 1,000 evaluation trials in simulation and categorize failures: grasp failure, approach angle error, object slip post-grasp, etc. Then run 100 trials on real hardware and categorize failures the same way. If the failure mode distributions match, your sim model is well-calibrated. If they diverge significantly, the mismatch points to the specific physics component that needs better modeling.
SVRC's RL environment service provides managed simulation environments with system identification support — helping teams match their simulation physics to their specific hardware before starting large-scale training runs.