Manipulation control and teleoperation

Kinova Gen3 · xArm6 · Franka · UR5 · RealSense / ZED · Meta Quest 3 · 2024 – 2026
Gravity compensation on the Kinova Gen3 with torque control.
1 kHz
torque control on the Kinova Gen3
5
controllers, including Cartesian impedance
4
metric depth models

Overview

I wrote control and perception software for an xArm6 (250 Hz) and a Kinova Gen3 (1 kHz torque control). It is operated from a browser and has five controllers, including gravity compensation and Cartesian impedance control. It also has four metric depth models and SAM2 segmentation. I also built VR teleoperation with a Meta Quest 3 for collecting demonstrations.

Videos and images

Live point cloud and arm control in the browser (2 times faster).
Kinova Gen3 picking objects from a plate.
Depth, segmentation and point cloud from one camera (2 times faster).
Peg-in-hole on a real Franka arm.
Peg-in-hole in simulation.
Push-T with several Franka arms in simulation (2 times faster).
Pick and place on a UR5.
Collecting demonstrations with an Open Manipulator X (4 times faster).
Unloading a dish rack in simulation.

Details

Control software

The software controls two arms that can be switched from the browser while it is running: an xArm6 with joint position commands at 250 Hz, and a Kinova Gen3 with position and torque commands at 1 kHz. There are five controllers: joint servo, end-effector control with differential inverse kinematics, gravity compensation, compliant joint PD, and Cartesian impedance control. The last three use torque control. Kinematics and dynamics are computed with Pinocchio.

Perception

The browser shows the camera image, the metric depth and a 3D point cloud live. Four depth models can be selected: Depth Anything V2, Apple DepthPro, UniDepth V2 and Metric3D V2. SAM2 gives object masks and 3D bounding boxes from a click. I tested six cameras on the setup and ran five USB cameras at the same time at 1280×720 without dropped frames.

Teleoperation

For collecting demonstrations I built VR teleoperation with a Meta Quest 3 over WebXR. The same software drives a simulated or a real arm, and the headset shows a point cloud of the scene. I also wrote a version of hand teleoperation that runs completely in the browser with MuJoCo WebAssembly and hand tracking from a webcam.

Learning experiments

The videos below also show earlier experiments on the lab arms: push-T and peg-in-hole policies on a Franka arm, pick and place on a UR5, and data collection with an Open Manipulator X.

Other projects

Sim-to-real manipulation with Gaussian-splat simulationStereo-inertial SLAM and monocular VIOContinual learning for manipulationCustom quadruped robotHumanoid manipulationHospital delivery and inspection robotSimulation projects