Sim-to-Real and Evaluation: Does Any of This Actually Work?

⬅️ Main Hub: The End-to-End Robot Learning Pipeline ⬅️ Previous: World Models: How Robots Learn to Practice in Their Own Imagination

Introduction

This section asks the question every other section is building toward: does any of this actually work outside the lab, and how do you know? The three sub-sections answer that question at increasing levels of cost and realism. Sim-Evaluation tests policies in software, cheap and reproducible, but only as trustworthy as the simulation’s ability to predict real outcomes. Sim-to-Real covers the techniques for closing the gap between where training happens and where deployment happens. Real-world evaluation is the expensive ground truth that everything else is trying to approximate.

What This Section Covers

• Sim-evaluation: how the field proved that simulation rankings reliably predict real-world outcomes, then scaled that proof into richer metrics, larger environments, and finally generative video models that serve as the evaluator.
• Sim-to-Real: how to close the gap between simulated training and real deployment, from understanding what causes the gap to building accurate simulations from real footage to adapting policies across the divide.
• Real-world evaluation: one paper did this rigorously as its primary contribution, and the shortness of this section is itself the message.

8A: Sim-Evaluation

Testing a real robot is slow, expensive, and fragile. Simulation exists to fix this, but for years, the field had no proof that simulated results actually correlated with real-world ones. The papers in this section trace the arc from establishing that proof, to scaling it, to building richer signals than a binary pass-fail, to the current frontier where generative video models serve as the evaluator.

**CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks** (Mees et al., 2022)

• Tests whether a robot can follow five language commands in a row across four visually different environments, including one never seen during training.
• Scoring 53.9% on single tasks but 0.08% on five-task chains even in familiar environments; now the standard benchmark for generalist robot policies.

**PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks** (Grotz et al., 2024)

• Introduces 18 two-arm manipulation tasks and a single network that reads the full scene and predicts both arms’ actions simultaneously, replacing fixed leader-follower setups.
• Trains in 54 hours versus 89 for the closest two-network baseline; qualitative real-world tests on a humanoid showed the policy transfers across robot morphologies.

**Evaluating Real-World Robot Manipulation Policies in Simulation (SIMPLER)** (Li et al., 2024)

• Automatically tunes simulated joint stiffness and damping to match real robot movement, then replaces simulated backgrounds with photographs of the actual environment.
• Simulated rankings matched real-world outcomes across 1,500 episodes: at or above 0.85 on Google Robot tasks, 0.89 average on Bridge tasks, 7 times faster.

**PolaRiS: Scalable Real-to-Sim Evaluations for Generalist Robot Policies** (Jain et al., 2025)

• Instead of manually building environments, PolaRiS converts a short room video into a 3D simulation and populates it with AI-generated objects.
• Creates a new evaluation environment in under 20 minutes of human effort, with simulation rankings predicting real-world rankings at 0.9 correlation across six environments.

**vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models** (Choi et al., 2026)

• Wraps fourteen robot simulation benchmarks in isolated software containers so any model connects once and runs across all benchmarks without dependency conflicts.
• Parallel episode execution achieves 47x evaluation speedup. One undocumented setting in X-VLA changed task success by 55 percentage points.

**RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation** (Wang et al., 2026)

• Tracks trajectory smoothness, collision counts, and arm coordination alongside pass-fail scores, revealing the top-success policy was best on only 33% of metrics.
• On a tray-lifting task, three policies with zero overall success still completed 35 to 48% of task stages, progress that binary scores erase entirely.

**RoboPlayground: Democratizing Robotic Evaluation through Structured Physical Domains** (Wang et al., 2026)

• Type a plain-English task description, and the system generates a runnable simulation with randomized positions, a success check, and a family of variations.
• All six tested policies scored under 2% on tasks requiring multi-step sequencing or undoing prior actions, versus 60 to 96% on training tasks.

**KinDER: A Physical Reasoning Benchmark for Robot Learning and Planning** (Huang et al., 2026)

• Tests five physical reasoning challenges across 25 environments that regenerate randomly, blocking memorization, with 13 methods from classical planners to AI models compared.
• Bilevel planning achieved the highest success at 0.57 while model-free reinforcement learning reached only 0.02; real-robot tests confirmed the simulated challenges reflect physical reality.

**MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation** (Kim et al., 2026)

• Packages 230,000 environments, 130,000 objects, and 42 million pre-computed grasps into one robot evaluation platform; simulation benchmark scores predict real-world success at 0.96 correlation.
• Blocking the wrist camera alone cut pick success from 55% to 2%; the overhead camera mattered far less, dropping it only to 20%.

**RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translation **(Jangir et al., 2026)

• Automatically turns real robot videos into physics simulations, then scores policies using an AI judge and crowdsourced humans who agreed exactly across 8,749 comparisons.
• All six robot policies scored substantially higher on SIMPLER’s four environments than on 70 BridgeSim scenes from the same dataset, suggesting SIMPLER inflates performance.

**RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies** (Yang et al., 2026)

• Policies trained on real-world data are tested in simulation environments they have never seen, making high scores proof of genuine generalization across 120 tasks.
• Simulated rankings matched real-world results exactly across four policies, and wrist camera misalignment hurt performance far more than lighting variation.

**SC3-Eval: Evaluating Robot Foundation Models via Self-Consistent Video Generation** (Tseng et al., 2026)

• Simulates robot behavior in video, verifies generated frames can reproduce the commands that drove them, and trains all camera views to stay coherent.
• Predicted policy success rates matched real robot outcomes at 0.929 correlation across seven tested checkpoints, outperforming all three competing video-based evaluators.

8B: Sim-to-Real

Every paper in 8A described a way to test policies without running them on a physical robot. This section covers a different problem: training policies in simulation and making them work on real hardware. The gap between those two places comes from four sources: physics that simulation approximates wrong, sensors that produce subtly different outputs, motors that respond differently than their simulated counterparts, and system-level timing mismatches.

**The Reality Gap in Robotics: Challenges, Solutions, and Best Practices** (Aljalbout et al., 2025)

• Classifies simulation-to-hardware failures into four families (physics, sensing, actuation, system design) and divides solutions into narrowing the gap versus training policies to tolerate it.
• Read this first: every other paper in this sim-to-real section targets exactly one of the four failure categories defined here.

**Reconciling Reality through Simulation: A Real-To-Sim-to-Real Approach for Robust Manipulation (RialTo)** (Torne et al., 2024)

• Maps sim-to-real failures into four causes (physics, sensors, motors, system timing) and two fixes: close the gap or build policies that tolerate it.
• Gives practitioners a shared vocabulary for diagnosing why simulation-trained robots fail in reality and which fix strategy to apply.

**ASID: Active Exploration for System Identification in Robotic Manipulation** (Memmel et al., 2024)

• Films the real workspace, builds a physics simulation in 15 minutes, trains a policy inside with randomization, and deploys it on the real robot.
• Across eight tasks, RialTo averaged 91%, 77%, and 75% under pose randomization, visual distractors, and physical disturbances; demonstration-only baselines averaged 25%, 11%, and 5%.

**URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images** (Chen et al., 2024)

• Predicts from a single photo which objects have moving parts, what type of motion each allows, and where joints sit, without manual authoring.
• Achieved 78% zero-shot real-world success across 10 cabinet manipulation tasks, versus 18% for standard domain randomization and 0% for a vision-language baseline.

**DRAWER: Digital Reconstruction and Articulation With Environment Realism** (Xia et al., 2025)

• Converts a room video into two linked 3D representations: one geometrically precise for physics, one photorealistic for visuals, then loads both into a simulator.
• A manipulation policy trained purely on simulation data from this reconstruction transferred directly to a real robot with no manual scene setup.

**Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL** (Wagenmaker et al., 2024)

• Trains policies in simulation not to complete the task but to visit diverse states, then deploys them to collect real-world data for task learning.
• Direct transfer failed all 6 real Franka puck trials; exploration-based transfer succeeded all 6, with theory proving exponential reduction in required real-world samples.

**TAM: Torque Adaptation Module for Robust Motion Transfer in Manipulation** (Son et al., 2026)

• Intercepts motor commands below the policy, reads a long joint motion history, and adds corrective torques to compensate for payload, wear, and hardware variance.
• Reduced mean tracking error from 4.48 to 0.76 degrees across seven robot arms and improved a flip task from 50% to 72% without retraining.

**Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning (SGFT)** (Yin et al., 2025)

• Trains a measure of task-progress in simulation and uses it to steer real-world exploration, even when the simulator’s contact physics are fundamentally wrong.
• Achieved 100% success on hammering and pushing within one hour; baselines without this guidance needed far more real-world trials, and some failed entirely.

**Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation** (Maddukuri et al., 2025)

• Fill training batches with mostly simulation data alongside a few real demonstrations; camera angle matching matters, physics alignment does not.
• Real-only averaged 45% across six tasks; adding purpose-built and general simulation data lifted that to 83%, on a robot arm and humanoid.

**Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training** (Cheng et al., 2026)

• Adds an alignment loss that maps corresponding simulation and real observations to the same internal location, rather than leaving them as two separate clusters.
• Under evaluation on novel object poses and textures never seen in real training, task success improved from 10% to 40% over standard co-training.

**Point Bridge: 3D Representations for Cross Domain Policy Learning** (Haldar et al., 2026)

• Replaces images with 3D coordinates sampled from task-relevant objects only, so the representation stays consistent across simulation and reality despite visual differences.
• Using only synthetic data, it beat image-based transfer by 39–44%; adding 45 real demonstrations extended gains to 61–66%.

**Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation** (Levy et al., 2026)

• Trains a world model in simulation, updates only the dynamics model on real data; reward, value, and perception transfer across the sim-to-real gap.
• Across manipulation and locomotion tasks, roughly doubled baseline success rates in 15 to 35 minutes of real-world data; competing methods made no consistent progress.

8C: Evaluation (Real World)

Real-world testing is expensive, slow, hard to reproduce, and unsafe to run at the scale modern policies need. There is essentially one paper in this entire collection whose primary contribution is doing real-world evaluation rigorously, as the main point of the paper rather than a supporting result appended to an algorithm. The shortness of this section is itself the message.

**A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation** (TRI LBM Team, 2025)

• Trains one policy across 500+ tasks on 1,700 hours of data, then finetunes per task and evaluates across 1,800 real-world trials with blind evaluators.
• Pretraining needs under 30% of the data to match single-task baselines and outperforms with equal data; most robotics papers can’t detect gains this small.

Series Complete

You have reached the end of the End-to-End Robot Learning Pipeline series. Every stage of the chain has been covered, from the first recorded demonstration to the rigorous real-world test that tells you whether any of it actually worked.

The field is moving fast. The papers in this series represent the state of the art as of mid-2026, but the chain reaction continues. New bottlenecks are being found and solved every month. The best way to stay current is to follow the researchers behind the papers you found most useful, and to keep asking the same question this section asked: Does any of this actually work outside the lab, and how do you know?

🚀 You Reached the End

⬅️ Previous: World Models: How Robots Learn to Practice in Their Own Imagination 📌 Series Hub: The End-to-End Robot Learning Pipeline