Continual learning for manipulation

PhD thesis · mjlab · reinforcement learning · 2023 – present
Eight of the 37 benchmark tasks.
37
manipulation tasks in the benchmark
87.6%
average success over 15 tasks with one network
4
tasks run by one policy on a real 7-DOF arm

Overview

My thesis is about adding new manipulation skills to one policy without storing the data of earlier tasks. I built a benchmark of 37 manipulation tasks and trained a separate teacher policy for each task. The teachers are then combined into one network one after another.

Benchmark tasks

Peg Insertion Franka
Open Drawer Franka
Open Door Franka
Stack Cube Franka
Rotate Valve Franka
Place In Container Franka
Turn Lever Franka
Push Button Franka
Flip Switch Franka
Slide Window Franka
Throw To Bin Franka
Tool Pull Franka
Stack Cube Franka Leap
Lift Cube Franka Leap
Reorient Object Franka
Open Lid Franka

Details

Problem

A robot that keeps working for a long time needs to learn new skills. Training on a new task usually makes a neural network forget the earlier tasks. Keeping all old data and retraining on everything solves this, but the amount of data keeps growing. I study the case where the data of earlier tasks is not kept.

Method

For each new task a teacher policy is trained separately. The teacher can be an RL policy, a behaviour-cloning policy or a classical controller. The teacher is then distilled into the single student network. During distillation the weights that are important for earlier tasks are protected with Synaptic Intelligence. After a task is added, its teacher and its data are discarded.

Benchmark

I built a benchmark of 37 manipulation tasks in mjlab (MuJoCo-Warp) with a Franka arm and a LEAP hand. The tasks include lifting, stacking, pushing, peg insertion, opening doors and drawers, turning valves and levers, and pressing buttons. 24 PPO teacher policies were trained for the continual learning experiments.

Results

With 15 tasks learned one after another, the single network reached 87.6% average success (mean of three seeds, standard deviation 0.4). The baseline without consolidation reached 79.5%. In an earlier study with four tasks, all 24 possible task orders were tested with three seeds each. The final success depended on the order of the tasks. The best order reached about the same success as training on all tasks together. One combined policy was run on a real 7-DOF arm and performed all four tasks without fine-tuning on the hardware.

Current work

I am now working on continual reinforcement learning, where the policy is trained directly on a sequence of tasks. Six single-task policies with point-cloud input (FastTD3) each reach 99.8 to 100% success, and these are the starting point for the continual experiments.

Earlier paper

My earlier work on learning manipulation tasks one after another with offline RL was published in Robotica in 2024.

Other projects

Sim-to-real manipulation with Gaussian-splat simulationStereo-inertial SLAM and monocular VIOCustom quadruped robotManipulation control and teleoperationHumanoid manipulationHospital delivery and inspection robotSimulation projects