Sim-to-real manipulation with Gaussian-splat simulation

PhD research · UR5 · MuJoCo-Warp · Gaussian splatting · 2026
The RGB-only policy trained in simulation picking a bottle on the real UR5.
21 / 29
picks on the real UR5 with a policy trained only in simulation
98.9%
success over 5,120 simulated episodes
6.5×
higher training throughput per GPU
~30 min
peg-in-hole teacher training with FastTD3 (PPO: ~8 h)

Overview

I train camera policies in simulation and run them on a real UR5 with an OnRobot RG2 gripper and wrist cameras. The simulator is MuJoCo-Warp, and the camera images are rendered from a Gaussian-splat scan of the lab. No real robot data is used for training.

Videos and images

Simulation: picking a lying bottle, with the rendered wrist camera view.
Real recording and reconstructed simulation side by side for a DROID scene.
First successful pick on the real robot (October 2026).
Novel views of a reconstructed DROID scene.
Repeated attempts on the real robot (4 times faster).
Wrist camera view during a tube picking run on the real robot (2 times faster).

Details

Setup

The robot is a UR5 with an OnRobot RG2 gripper and RealSense cameras on the wrist. The task is to pick a bottle that is either standing or lying on a table. A second task is to pick test tubes from a bin.

Simulation

The physics runs in MuJoCo-Warp through mjlab, with thousands of environments on each GPU. To make the camera images look like the real lab, I scanned the lab and trained a Gaussian-splat model of it. The simulated robot and objects are rendered together with this background. Metric scale comes from the depth camera, so no markers are needed in the scene.

Training

First a teacher policy is trained with PPO using the exact state of the scene. Then student policies that only see camera input (RGB, stereo or point clouds) are trained to copy the teacher with DAgger, followed by RL fine-tuning. For the tube task I also trained ACT imitation policies.

Results on the real robot

The RGB-only student was run on the real UR5 without any real training data. It picked the bottle in 21 of 29 attempts (72%). The success rate was 89% for standing bottles and 45% for lying bottles. The number of real trials is small, so these numbers have a wide uncertainty.

Results in simulation

The point-cloud student reached 98.9% success over 5,120 episodes and 98.5% over 3,072 episodes with conditions not seen in training. The teacher reaches 99.9%.

Training speed

I increased the throughput from 145 to about 950 samples per second per GPU (6.5 times) and the training scales to 7 GPUs with about 97% efficiency. I also compared FastTD3 with PPO for teacher training. Peg-in-hole reached 95% success in 26 to 34 minutes on one GPU with FastTD3, compared with about 8 hours with PPO. The bottle task took about 1 to 2 hours compared with about 6 hours. A nut-threading task was not solved by either method.

Real-to-sim on public data

I applied the same scene reconstruction to recordings from the DROID dataset from three labs. The median depth error was 0.6 mm and the mask overlap (IoU) was 0.96.

Other projects

Stereo-inertial SLAM and monocular VIOContinual learning for manipulationCustom quadruped robotManipulation control and teleoperationHumanoid manipulationHospital delivery and inspection robotSimulation projects