The Generalization Gap
A policy trained to pick up red plastic cups achieves 95% success. The same policy presented with a blue cup of identical geometry fails with 0% success. This is not a hypothetical — it's a documented failure mode that recurs across vision-based manipulation systems, and it illustrates why generalization testing is a required step before any deployment.
The underlying cause is that neural policies learn correlations that work on training data, not causal features that generalize. A policy "knows" how to grasp a red cup because it has learned a representation that fires on the combination of: cylindrical depth profile, red RGB values, specific lighting conditions in the collection environment, and the precise offset from the center of the camera frame that the cup appeared at during collection. Change any of these — especially the color — and the policy sees something outside its learned manifold.
Root Causes by Modality
- Visual Overfitting (color/texture): Policies trained on RGB or RGB-D input are highly susceptible to color and texture correlation. If all training grasps were on red cups, the policy's visual encoder may have learned hue histograms as a key feature rather than shape. This is detectable: run your policy on grayscale input. If performance drops more than ~10%, you have a color-dependence problem.
- Spatial Overfitting (position dependence): Policies trained with objects consistently placed in a narrow region of the workspace learn an implicit "this is where objects appear" prior. A ±3cm displacement from the training distribution center can cause failure in policies that have learned a direct mapping from pixel coordinates to joint angles, because the object appears in a different image location.
- Temporal Overfitting (contact timing): Some policies learn to time their gripper close based on the elapsed time since the grasp phase started, rather than on sensory feedback. Works perfectly on training objects with a specific size and compliance; fails on harder or softer objects where the contact dynamics differ.
Measuring the Generalization Gap
Generalization is not one thing — it's at least three independent dimensions that must be tested separately. Testing only one and reporting "generalizes well" is a common error in robot learning papers and product demos.
| Generalization Axis | Test Protocol | Typical Gap (vision policies) |
|---|---|---|
| Novel objects | 10 held-out objects, same task | 20–50% success drop |
| Novel positions | Uniform random in workspace | 10–30% success drop |
| Novel environments | Different lighting, background, table | 30–60% success drop |
What Actually Improves Generalization
In rough order of impact, based on ablation studies across multiple manipulation learning papers:
- Object Diversity in Collection (highest impact): Training on 10+ diverse objects per task is the single most effective intervention. Not 10 variations of the same object — 10 genuinely different objects with different shapes, materials, and colors. Each additional object in training reduces the slope of the generalization gap for novel objects. This is also the most expensive intervention since it requires more data collection.
- Image Augmentation: Random color jitter (hue ±0.4, saturation ±0.5), random crop (crop and resize to original), and Gaussian blur are the most effective augmentations for manipulation policies. Applied aggressively during training, these consistently deliver +10–15% success on novel-object and novel-environment evaluations at zero marginal data cost.
- Language Conditioning: Conditioning the policy on a natural language instruction that describes the task in object-agnostic terms ("pick up the container") rather than training on visual demonstration alone encourages the policy to learn features relevant to the described action rather than the specific visual appearance of training objects.
- Foundation Model Fine-Tuning: Starting from a pre-trained visual encoder (CLIP, DINOv2) rather than training from scratch gives the policy visual features that generalize across thousands of object categories. Fine-tuning these models on manipulation data while retaining the broad visual representations consistently outperforms training encoders from scratch on task-specific data.
The most practical near-term recommendation: before adding model complexity or more training compute, invest in object diversity in your data collection. SVRC's data collection service can structure collection protocols specifically around generalization — using diverse object sets, varied placements, and multiple lighting conditions from the start.