GR00T Policy Inference

Hierarchical control architecture — how a trained GR00T manipulation policy runs on the real Unitree G1.

1 Block diagram

flowchart TD
    subgraph SENSORS["Sensors (live, 50 Hz)"]
        CAM["Cameras (camera_host:5555)"]
        STATE["Robot state via state_processor.py (DDS subscriber)"]
        LANG["Task prompt (text, e.g. 'pour water from jug')"]
    end

    subgraph POLICY["GR00T Policy (outer loop, ~3 Hz)"]
        OBS["prepare_observation_data : tokenize images/state, build parsed_obs"]
        VLM["Eagle VLM backbone : vision + language -> vl_embs, sa_embs"]
        DIT["Action DiT (diffusion transformer)
TensorRT engine via TensorRTDiTWrapper
iterative denoising"]
        CHUNK["action_chunk : H=16 future actions"]
        OBS --> VLM --> DIT --> CHUNK
    end

    subgraph STREAM["Action streaming (inner loop, 50 Hz)"]
        DISPATCH["Dispatch one action per tick
(replaces pico_streamer.py output)"]
    end

    subgraph WBC["WBC Stack (unchanged, 50 Hz)"]
        TELEOP["run_teleop_policy_loop.py
IK + InterpolationPolicy + DecoupledWBC"]
        CTRL["run_g1_control_loop.py
ONNX Balance/Walk RL policy
(switch threshold = 0.05)"]
        DDS["command_sender.py -> rt/lowcmd DDS"]
        MOTORS["35 motors on Unitree G1"]
        TELEOP --> CTRL --> DDS --> MOTORS
    end

    CAM --> OBS
    STATE --> OBS
    LANG --> OBS
    CHUNK --> DISPATCH
    DISPATCH --> TELEOP

2 Control-block diagram

The diagram above shows data flow through the software stack. This one is drawn in classical control-block style: signals are named, dimensions and update rates are annotated, and the physical feedback loop closes through the robot itself (motors move the body → IMU/encoders re-read the new state).

All sizes and key names below are taken from the trained checkpoint's experiment_cfg/conf.yaml and dataset_statistics.json (embodiment unitree_g1_full_body_with_waist_height_nav_cmd).

2.1 Signal legend

Each signal carries a specific physical quantity — position, velocity, torque, or something else. The table also makes explicit which fields are actually consumed at each stage (e.g. GR00T reads positions only, the motor cmd is full PD+feedforward).

Per-stage cheat sheet — what each consumer actually reads/writes:

Important: GR00T is not a torque controller. It outputs target joint positions; the low-level PD loop in each motor does the actual force production. The "torque" you see on rt/lowcmd (cmd.tau) is a feedforward term, typically zero in this stack except where the RL policy populates it.

2.2 Block diagram

%%{init: {"flowchart": {"useMaxWidth": false, "htmlLabels": true}, "themeVariables": {"fontSize": "20px"}}}%%
flowchart LR
    subgraph PLANT["PLANT - Unitree G1 + environment"]
        BODY["Rigid body, 35 actuated DOF
Hands + arms + waist + legs
onboard motor PD: tau = kp(q*-q) + kd(dq*-dq) + tau_ff"]
        SENS["IMU + joint encoders + cameras
measures: q, dq, tau_est, ddq, quat, gyro, accel"]
        BODY -- "physical motion" --> SENS
    end

    subgraph PERCEPT["Perception (50 Hz)"]
        STATE_PROC["state_processor.py
DDS rt/lowstate -> obs_vector
[q, dq, tau_est, ddq, IMU]"]
        CAMSRV["camera_server
(192.168.123.164:5555)
-> I_t (256x256x3)"]
    end

    subgraph POLICY["GR00T Policy (outer ~3 Hz, on dGPU)"]
        BUILDOBS["prepare_observation_data
tokenize I_t, normalize s_t (positions only), embed L
state_history_length = 1"]
        VLM["Eagle VLM backbone
(Cosmos-Reason2-2B, qwen)
-> vl_embs, sa_embs"]
        DIT["Action DiT (TensorRT)
num_inference_timesteps = 4
action_horizon = 40, use first 16"]
        BUILDOBS --> VLM --> DIT
    end

    subgraph BUFFER["Action buffer (FIFO, ~3 Hz refill / 50 Hz drain)"]
        QUE["a_chunk = [a_t, ..., a_t+15]
target positions for arms/hands/waist
+ base_height (m) + (vx, vy, vyaw)"]
    end

    subgraph WBC["WBC stack (50 Hz)"]
        IK["run_teleop_policy_loop.py
IK + InterpolationPolicy
target arm q* -> joint refs"]
        SWITCH{"|nav_cmd| < 0.05 ?"}
        BAL["Balance ONNX
in: leg q, dq, IMU
out: 15 leg deltas"]
        WALK["Walk ONNX
in: leg q, dq, IMU, vel setpt
out: 15 leg deltas"]
        MERGE["DecoupledWBC
upper-body refs + leg targets
q* = action * 0.25 + default_angles"]
        IK --> MERGE
        SWITCH -- "yes (stand)" --> BAL --> MERGE
        SWITCH -- "no (walk)" --> WALK --> MERGE
    end

    SENDER["command_sender.py
writes per-motor (q*, dq*, tau_ff, kp, kd)
DDS publish rt/lowcmd @ 50 Hz"]

    L[/"Task prompt L
(constant per episode)"/] -- "text" --> BUILDOBS
    CAMSRV -- "I_t (RGB image)" --> BUILDOBS
    STATE_PROC -- "s_t = joint positions q only" --> BUILDOBS
    DIT -- "a_chunk every ~330 ms
(target q + base setpoints)" --> QUE
    QUE -- "a_t @ 50 Hz
upper-body target q*" --> IK
    QUE -- "navigate_command
(vx, vy, vyaw)" --> SWITCH
    STATE_PROC -- "leg q, dq + IMU" --> BAL
    STATE_PROC -- "leg q, dq + IMU" --> WALK
    MERGE -- "combined target q* (35 joints)" --> SENDER
    SENDER -- "rt/lowcmd DDS
q*, dq*, tau_ff, kp, kd per motor" --> BODY

    SENS -- "q, dq, tau_est, ddq, IMU
rt/lowstate DDS @ 50 Hz" --> STATE_PROC
    SENS -- "RGB frames" --> CAMSRV

2.3 Why this is the right way to draw feedback

The only real feedback in the system is the closed-loop through the plant:

policy -> action -> motors -> physical robot moves -> sensors observe new state -> back into policy

There is no shortcut wire that sends state back into the policy bypassing the robot. Each outer tick the policy re-reads s_t because the robot has physically changed pose since the last tick.

2.4 Per-block update rates and latencies

So end-to-end camera-to-motor latency for a fresh policy decision is roughly one outer-tick period (~330 ms) plus a few ms of WBC, but because actions are streamed from a chunk, the motor command updates every 20 ms — only one in 16 of those motor commands rides on a brand-new policy decision.

2.5 Comparison with the original Pico teleop path

Same diagram, but the GR00T block is replaced by pico_streamer.py:

Everything from the FIFO downward is unchanged between teleop and autonomous inference.

3 Two clocks

• Outer (policy) clock — ~3 Hz: one heavy forward pass every H control ticks. This is what TensorRT accelerates by swapping the PyTorch DiT for a TRT engine (replace_dit_with_tensorrt, TensorRTDiTWrapper).
• Inner (control) clock — 50 Hz: the H = 16 actions from the latest chunk are dispatched one per tick into the WBC teleop input. The WBC stack (IK + RL standing/walking policies) keeps producing motor commands at 50 Hz regardless.

4 The inference loop (heart of it)

From scripts/deployment/standalone_inference_script.py (run_single_trajectory, line 371):

# Setup once
policy = Gr00tPolicy(checkpoint_path, embodiment_tag, modality_configs, ...)
policy = replace_dit_with_tensorrt(policy, trt_engine_path)   # swap DiT for TRT engine

executor = ThreadPoolExecutor(max_workers=1)
future_obs = executor.submit(prepare_observation_data, ...)   # prefetch step 0

for step_idx, step_count in enumerate(step_counts):           # stride = action_horizon
    parsed_obs = future_obs.result()                          # block until obs ready

    # Prefetch next obs on CPU while GPU runs inference on current one
    if step_idx + 1 < len(step_counts):
        future_obs = executor.submit(prepare_observation_data, ..., next_step_count, ...)

    # >>> the actual inference call <<<
    action_chunk, _ = policy.get_action(parsed_obs)           # VLM + DiT forward

    # Stream H future actions at the control rate
    action_chunk = parse_action_gr00t(action_chunk)
    for j in range(action_horizon):
        send_or_record(concat_per_key(action_chunk, j))

Note: standalone_inference_script.py runs against a recorded LeRobot dataset (offline replay/eval). On the real robot, loader[traj_id] is replaced by live sensors, and the inner send_or_record(...) becomes publish to the WBC teleop input (the same ROS topic pico_streamer.py currently writes to).

5 What policy.get_action() does internally

1. Image branch: camera frames → vision encoder (Eagle VLM backbone).
2. State branch: proprioceptive state vector → state encoder.
3. Language branch: task prompt → tokenizer (cached if the prompt is constant).
4. Backbone fusion: Eagle VLM produces vl_embs (vision+language) and sa_embs (state+action) embeddings.
5. Action DiT: a diffusion transformer denoises a noise tensor over several timesteps, conditioned on vl_embs + sa_embs, producing the action chunk. This is the part swapped out for a TensorRT engine (TensorRTDiTWrapper.__call__, line 132 — called once per denoising step).
6. Output: dict of per-modality action chunks → flattened into per-joint trajectories.

6 Real-robot loop (pseudocode)

while task_running:
    parsed_obs = build_obs_from(live_cameras, live_state, task_prompt)
    action_chunk, _ = policy.get_action(parsed_obs)            # ~1x per H ticks (TRT)
    for a in action_chunk:                                     # stream at 50 Hz
        publish_to_teleop_input(a)                             # where pico_streamer used to write
        wait_until_next_control_tick()

• publish_to_teleop_input(a) lands on the same ROS topic / streamer hook in run_teleop_policy_loop.py that the Pico headset normally feeds.
• run_g1_control_loop.py keeps running unchanged below, consuming those targets and producing motor commands at 50 Hz via DDS.

7 TL;DR — the hierarchy

[outer ~3 Hz]   policy.get_action(obs)  ->  16-step action chunk         (GR00T VLM + DiT, TRT)
[inner 50 Hz]   stream chunk action-by-action to WBC teleop input
[inner 50 Hz]   run_teleop_policy_loop.py     : IK + DecoupledWBC
[inner 50 Hz]   run_g1_control_loop.py        : ONNX Balance/Walk RL  ->  rt/lowcmd DDS  ->  motors