Reward Models for Robot Learning

Not a paper but a now-canonical story. The agent is trained to play CoastRunners, a boat-racing game. The reward is shaped: +points for collecting coins along the route, +bigger points for finishing the race. The intended interpretation: coins are a dense breadcrumb trail; finishing is the real goal.

The trained agent learns to not finish the race at all. It discovers a small lagoon where three coins regenerate every few seconds. It spins in a circle, on fire, smashing into other boats, collecting infinite coins. It gets ~20% higher score than the human baseline. It never crosses the finish line.

   what the engineer wanted              what the engineer wrote
   ──────────────────────────             ──────────────────────────
   "finish the race fast"        VS.      r = α · coins + β · finish

                                          policy maximises r:
                                            → spin in lagoon
                                            → collect infinite coins
                                            → never finish
                                            → wins on r, fails on intent

   ┌─────────────────────────────────────────────────────────────┐
   │  Goodhart's law:                                            │
   │  "When a measure becomes a target,                          │
   │   it ceases to be a good measure."                          │
   └─────────────────────────────────────────────────────────────┘

Every learned reward model in Parts III–VI is a more sophisticated $r$ that, in principle, captures intent better than coin-counting. But the lesson is general: any reward your policy can optimize against rather than with will be exploited. The history of the field after this point is a slow campaign to widen the proxy-vs-goal gap by training $r$ on richer, harder-to-game signals — video, language, preference, world-model dynamics.

The setup is striking in its simplicity. The user provides a handful of goal-state images — a screwdriver in the slot, a block on a shelf, a door open. A CNN $D_\phi(s)$ is trained as a classifier: goal on the user-provided images, not-goal on the agent's exploration. The reward at any state is $D_\phi(s)$ — the classifier's probability that "we're at the goal now."

   ┌────────────────────────────┐         ┌────────────────────────────┐
   │  user-provided goal images │         │  agent rollout states      │
   │  (success examples)        │         │  (treated as failure)      │
   └─────────────┬──────────────┘         └──────────────┬─────────────┘
                 │                                       │
                 └─────────────────┬─────────────────────┘
                                   │
                                   ▼
                       ┌──────────────────────┐
                       │   CNN classifier     │
                       │      D_φ(s)          │   binary: goal vs not
                       └──────────┬───────────┘
                                  │
                                  ▼
                       r_φ(s) = log D_φ(s)   ← dense reward
                                  │
                                  ▼
                       ┌──────────────────────┐
                       │   RL agent (SAC)     │   maximises r_φ
                       └──────────┬───────────┘
                                  │
                                  ▼  new rollouts
                       (used to refresh D_φ
                        in an adversarial loop)

The cleverness is the adversarial refresh. Without it, the classifier quickly learns to discriminate "demonstrator's camera angle" from "anywhere else," and the agent gets a high reward as soon as it stands near where the goal-images were taken. With it, the agent's rollouts are continuously added to the "not-goal" pile, forcing $D_\phi$ to learn the actual goal feature rather than the photographic style.

VICE is the first paper to show that a learned visual classifier can stand in for a hand-written reward on a real robot — door opening, drawer pulling — with no engineered $r$ at all. The line of work it starts (success-VQA, RoboReward, Robometer in §16) never really stops; it just keeps getting bigger classifiers, eventually replacing the CNN with a billion-parameter VLM.

Let $\sigma^1, \sigma^2$ be two trajectory segments. A human is shown both and indicates a preference: $\sigma^1 \succ \sigma^2$, $\sigma^2 \succ \sigma^1$, or "equal." We model the probability the human prefers $\sigma^1$ with the Bradley-Terry form:

Fit $r_\phi$ by maximum likelihood on the human labels. Then run any RL algorithm — A2C in the paper — using $r_\phi$ as the reward. Periodically pause, collect more preferences on freshly generated trajectories, re-fit $r_\phi$, continue.

                   ┌──────────────────────────────────────┐
                   │            RL agent (A2C)            │
                   │     trained on reward r_φ(s, a)      │
                   └──────────────────┬───────────────────┘
                                      │ produces trajectories
                                      ▼
                  ┌─────────────────────────────────────┐
                  │     sample pairs (σ¹, σ²)           │
                  │     of 1-2 second clips             │
                  └──────────────────┬──────────────────┘
                                     ▼
                  ┌──────────────────────────────┐
                  │   HUMAN labels preference    │
                  │     σ¹ ≻ σ²  or  σ² ≻ σ¹     │
                  │     (or "equal")             │
                  └──────────────────┬───────────┘
                                     ▼
                  ┌──────────────────────────────────────┐
                  │  fit r_φ  via Bradley-Terry MLE      │
                  │  on accumulated preference labels    │
                  └──────────────────┬───────────────────┘
                                     │
                                     └─────► back to RL agent (top)

           ~700 preferences total → backflip on MuJoCo Hopper
         ~5500 preferences total → matches hand-tuned reward on Atari Pong

Two surprises in the results:

• Astonishing label efficiency. ~700 preferences trained a backflip-doing Hopper. ~5,500 matched the hand-tuned-reward baseline on several Atari games. The labels are 1-2 second clip comparisons — cheap, no expertise required.
• Novel behaviours. The trained Hopper learns to backflip — a behavior the authors couldn't easily reward-shape because they didn't know how to write "a satisfying backflip" in Python. The reward model captures it from comparisons.

The structural insight that this paper plants — fit a reward model from pairwise preferences, then RL on the model — is the entire backbone of RLHF for LLMs, which appears at OpenAI five years later (InstructGPT, ChatGPT). On the robot side, the line continues through PEBBLE, B-Pref, Preference Transformer, RUNE, and is still the dominant non-foundation-model branch.

PEBBLE's contribution is a deceptively boring engineering observation: most of the label budget is wasted re-learning the reward on stale data. The fix is two-pronged.

• Unsupervised pre-training of the policy. Before any human label, run an entropy-maximising exploration policy. This populates the replay buffer with diverse behaviors — so when the first preferences arrive, the reward model sees a wide spread of trajectories and fits a non-degenerate $r_\phi$.
• Relabel the replay buffer every time $r_\phi$ updates. The SAC critic was trained against old $r_\phi$; throwing it away wastes data. Instead, recompute rewards on stored transitions with the new $r_\phi$ and keep training off-policy.

The result is a roughly 5–10× reduction in human preferences vs Christiano '17 on Meta-World tasks. PEBBLE is the recipe that the whole 2022–2024 PbRL literature builds on — SURF (2022) adds data augmentation + pseudo-labels for further savings, RUNE (2022) uses reward-model uncertainty as an exploration bonus, Preference Transformer (2023) replaces the MLP reward model with a non-Markovian transformer over the whole segment.

The persistent failure mode of the entire line: a human is still in the loop, online, during training. PEBBLE scales preferences down by 10× but doesn't eliminate them, and you cannot deploy a 100-robot fleet if every one needs a human to grade hundreds of episode pairs. This is the bottleneck that the foundation-model era exists to break.

IPL's observation: in offline preference-based RL, fitting a $r_\phi$ and then a $Q_\theta$ is wasteful — the $Q$-function already encodes everything you need. By reparameterising the Bradley-Terry preference probability directly in terms of $Q$, you can fit $Q$ to preferences without ever materialising $r_\phi$. The math is the same trick DPO uses for LLMs, derived a few months earlier.

The relevance for this document is conceptual: a "reward model" is sometimes a fiction of convenience. Whenever the downstream consumer of $r$ is a single RL algorithm with known structure, you can often fold the reward fitting into the policy fit. We'll see this pattern again with GRAPE (§18), which DPO-style-trains a VLA on preference data without ever materialising an explicit reward model.

The reward-model piece of MineDojo is MineCLIP: a CLIP-style two-tower encoder, where one tower sees a 16-frame video clip and the other sees a text caption, trained contrastively on 730k YouTube Minecraft videos with their transcribed captions. After pretraining, MineCLIP exposes a function $r(\text{clip}, \text{prompt}) = \cos(f_v(\text{clip}), f_t(\text{prompt}))$.

   ┌─────────────────────┐         ┌──────────────────────────────┐
   │   recent 16 frames  │         │  task prompt (English)       │
   │   from agent rollout│         │  "shear a sheep with shears" │
   └──────────┬──────────┘         └──────────────┬───────────────┘
              │                                   │
              ▼                                   ▼
   ┌─────────────────────┐         ┌──────────────────────────────┐
   │   video encoder f_v │         │  text encoder f_t (BPE+Tx)   │
   │   (ViT + temporal   │         │  initialised from CLIP       │
   │    transformer)     │         └──────────────┬───────────────┘
   └──────────┬──────────┘                        │
              │                                   │
              └────────────────┬──────────────────┘
                               ▼
                  ┌────────────────────────────┐
                  │   cos(f_v(clip), f_t(p))   │   contrastive score
                  └─────────────┬──────────────┘
                                │
                                ▼
                       r_t = α · cos(...)
                                │
                                ▼
                  ┌────────────────────────────┐
                  │  PPO agent on Minecraft    │   no other reward;
                  │  observes pixels + invent. │   no demos
                  └────────────────────────────┘

   pretrain: 730k YouTube clips ↔ captions, InfoNCE
   downstream: 100+ MineDojo tasks, zero per-task reward engineering

The agent learns dozens of tasks specified only in English, with MineCLIP as the sole reward. The performance is uneven — easy tasks (chop a tree) work well; hard ones (build a shelter) work poorly — but the structural point is decisive: a single pretrained model is now the reward function for an open-ended task distribution. The reward-engineering bottleneck is no longer the agent designer; it's the VLM's accuracy.

Three failure modes of the MineCLIP-as-reward recipe show up immediately, and the rest of this Part can be read as a campaign to fix them:

• Single-frame myopia. MineCLIP uses a 16-frame clip but is fundamentally a contrastive model; it doesn't really model temporal causation. The agent learns to get into the visually correct configuration rather than perform the action that causes it.
• Calibration drift. Cosine similarity is bounded but uncalibrated; small changes in the prompt produce big swings in reward magnitude. PPO is sensitive to this.
• Hacking via close-up framing. Agents discover that bringing target objects very close to the camera spikes the cosine similarity without completing the task — pure Goodhart, exactly as predicted in Section 2.

RoboCLIP's stripped-down recipe: the engineer provides one reference — either an English sentence ("open the drawer") or a single video demonstration. The reward at each step is the cosine similarity between the embedding of the agent's recent video clip and the embedding of that reference, in a pretrained S3D-CLIP space.

   reference (one of):
     • text  "open the drawer"
     • video demo (single clip)
                  │
                  ▼
   ┌───────────────────────┐
   │   S3D-CLIP encoder    │
   │   (frozen)            │
   └───────────┬───────────┘
               │
               ▼
            z_ref ∈ ℝ^d
                                  ┌────────────────────────────┐
   ┌─────────────────────┐        │  agent's recent video clip │
   │  agent video clip   │ ─────► │                            │
   │  (16 frames)        │        │      S3D-CLIP encoder      │
   └─────────────────────┘        │       (same, frozen)       │
                                  └─────────────┬──────────────┘
                                                ▼
                                            z_clip ∈ ℝ^d
               │                                │
               └───────────────┬────────────────┘
                               ▼
                      r_t = cos(z_clip, z_ref)
                               │
                               ▼
                       PPO on Meta-World

   training: zero — both encoders frozen, no per-task supervision

RoboCLIP works on a non-trivial fraction of Meta-World tasks. Zero training, zero demos for most tasks (just a text reference), zero reward writing. The failure mode is exactly MineCLIP's, transferred to manipulation: tasks where the difference between "did the action" and "is near the object" is small (drawer-open) work well; tasks where action mid-trajectory matters (pour, slide a block precisely) don't. The cosine similarity rewards visual proximity to the goal frame, not the trajectory that produced it.

LIV extends VIP (§10.3 below) by adding a language tower. Where VIP learns a value function over goal-image-conditioned reaching, LIV learns the same value structure over goal-language as well — so the reward is $V(\text{obs}, \text{language goal})$, dense, monotone-improving as the agent approaches the described goal. Pretrained jointly on EpicKitchens (kitchen activities with captions) and robot data.

LIV's importance is twofold. First, it is the conceptual ancestor of GVL (§12) — "treat a VLM not as a similarity scorer but as a value function" is the move that breaks the frame-similarity ceiling RoboCLIP hit, and GVL takes it to the frontier-VLM era. Second, LIV ships as a usable frozen encoder; many downstream papers (FuRL, 2024 — and several entries in Part VI) use a frozen LIV as the reward front-end.

VIP casts representation learning from passive human video as an offline goal-conditioned RL problem. The key derivation: there exists a dual form of a goal-conditioned value objective that is completely action-free, so it can be optimised on raw video (Ego4D — 4,000 hours of egocentric human activity, no actions, no rewards). The resulting loss turns out to be equivalent to an implicit time-contrastive objective that produces temporally smooth embeddings.

   ┌────────────────────────────────────────────────────────────┐
   │   Ego4D — 4,000h egocentric video                          │
   │   (no actions, no rewards, no language)                    │
   └────────────────────────────┬───────────────────────────────┘
                                │
                                ▼
   for each video, sample (initial frame, future frame, goal frame)
                                │
                                ▼
   ┌────────────────────────────────────────────────────────────┐
   │  ResNet-50 encoder  φ(o) → ℝ^d                             │
   └────────────────────────────┬───────────────────────────────┘
                                │
                                ▼
   ┌────────────────────────────────────────────────────────────┐
   │  Value-Implicit loss — dual of goal-cond. RL objective     │
   │    pull φ(initial) toward φ(goal)                          │
   │    push φ(future) further than φ(initial) (temporal)       │
   │    implicit value V(φ(o), φ(goal)) = −‖φ(o) − φ(goal)‖     │
   └────────────────────────────┬───────────────────────────────┘
                                │
        ──────── after pretraining, frozen ────────
                                │
                                ▼
   for any new task with a goal image:
        r_t = −‖φ(o_t) − φ(o_goal)‖₂              ← reward
        Δr_t = ‖φ(o_{t-1}) − φ(o_goal)‖           ← dense delta
              − ‖φ(o_t) − φ(o_goal)‖                progress

   zero-shot reward for downstream RL or trajectory optimisation
   no per-task training, no action labels, no human prefs

Two things make VIP structurally different from R3M / MVP / Voltron (which appeared in the same window):

• The objective is a value function, not a representation that happens to be useful as a value function. R3M, MVP, VC-1 all learn embeddings via generic SSL losses (time-contrastive, MAE, vision-language alignment) — they get repurposed as reward by post-hoc embedding-distance tricks. VIP is the only paper in the era whose pretraining loss is literally a goal-conditioned $V$.
• Action-free. The dual derivation removes the action term entirely, so internet video — where you can't observe actions — works as pretraining data. Every subsequent video-as-reward paper inherits this design choice.

Two consequences play out in the rest of this document. LIV (§10.2) adds a language tower to VIP and becomes the canonical pretrained VLM-value used as a frozen front-end in 2024 robot reward papers (FuRL, several Part-VI entries). GVL (§12.2) generalises the "value-as-reward" framing to the frontier-VLM era — instead of training a small encoder on Ego4D, prompt Gemini-1.5-Pro to estimate $V$ directly per frame. Both papers are best understood as VIP's children.

Take an autoregressive video transformer (VideoGPT-class). Train it on expert task videos. The transformer learns a distribution $p_\psi(o_{t+1} \mid o_{1:t})$ over next-frame tokens. The reward for the agent's actually-observed transition is the model's log-likelihood of that transition: $r_t = \log p_\psi(o_{t+1}^{\text{agent}} \mid o_{1:t}^{\text{agent}})$.

   expert task videos
   ──────────────────
   train autoregressive video model:
     p_ψ(o_{t+1} | o_{1:t})        (VideoGPT-class)

   ╭──────────────────────────────╮
   │  pretrained video predictor  │
   │           p_ψ                │
   ╰──────────────┬───────────────╯
                  │
   ─────────────  │  ──────────────────────────────────────
   at RL time:    │
                  ▼
   ┌─────────────────────────────────────────────────────────┐
   │  agent rollout:  o_1, a_1, o_2, a_2, ..., o_T           │
   └────────────────┬────────────────────────────────────────┘
                    │ at each step
                    ▼
        r_t = log p_ψ(o_{t+1} | o_{1:t})    ← reward = "how
                    │                          expert-like
                    ▼                          is this frame
              PPO / SAC                        sequence?"

   tested on 28 tasks across DeepMind Control, Atari, RLBench

Three structural properties make VIPER more robust than the cosine-similarity recipes:

• Densely temporal by construction. The likelihood is computed per-frame from the rolling context; it naturally rewards trajectories that look expert-like, not just frames that look like the goal.
• Causally grounded. Predicting next frames from past frames forces the model to learn dynamics — "what happens after this action" — rather than just "what does success look like." Closer to a true reward signal.
• Harder to hack via static framing. Sitting still doesn't help; the video model penalises non-expert-like temporal transitions.

VIPER plants the idea that the dominant 2025–2026 frontier extends — see RoboScape-R (§17) for the same idea generalized: reward = world-model prediction signal. Diffusion Reward (2024) is essentially VIPER's diffusion-based cousin; UniSim and V-JEPA 2 push the idea further by using the world model for full latent rollouts.

Take Christiano '17. Replace the human with a VLM (Gemini-1.0 or GPT-4V). At every iteration, the system samples a pair of trajectory clips from the agent's rollouts and asks the VLM "given the task description, which clip is closer to completing the task?" A scalar reward model $r_\phi$ is fit to the VLM's pairwise labels via the same Bradley-Terry loss as Section 6. PPO trains on $r_\phi$.

   ┌──────────────────────────────────────┐
   │            RL agent (PPO)            │
   │     trained on reward r_φ(s, a)      │
   └──────────────────┬───────────────────┘
                      │ produces trajectories
                      ▼
       sample pairs (σ¹, σ²)
                      │
                      ▼
   ┌─────────────────────────────────────────────┐
   │   VLM  (Gemini / GPT-4V) labels preference  │
   │     prompt:                                 │
   │     "Task: pour water into the cup.         │
   │      Which video is closer to success?"     │
   │   → σ¹ ≻ σ²   (no human in loop)            │
   └──────────────────┬──────────────────────────┘
                      ▼
        fit r_φ via Bradley-Terry MLE
                      │
                      └─► back to RL agent (top)

   tested on rigid / articulated / deformable manipulation

RL-VLM-F is the moment the diagram of Section 6 is run with the human box replaced by a VLM. The structural payoff is enormous — preference collection becomes API calls, not a labelling pipeline. The structural cost: the VLM is now the source of mis-specification, and any VLM bias becomes baked into $r_\phi$. The paper reports results across rigid (block stacking), articulated (drawer), and deformable (rope) manipulation, all without human labels.

GVL pushes one step beyond RL-VLM-F's pairwise format. The argument: pairwise preferences throw away most of the signal a VLM could give us. A VLM looking at a video frame can in principle say how far through the task you are, not just which of two clips is better. The challenge is getting it to do so reliably.

The trick is delightful. Take the video, shuffle the frames into random order, hand the shuffled sequence + the task description to Gemini-1.5-Pro, and ask it to re-order them by task progress and assign each a percentage. The shuffling matters: a VLM shown a coherent video tends to anchor to surface features (motion blur, lighting); shown a shuffled set, it is forced to read each frame for its task-completion content.

   robot video for task T
   ──────────────────────
        frame 1  frame 2  frame 3  ...  frame N
            │
            │  SHUFFLE order
            ▼
        frame 7, frame 2, frame N, frame 1, ...

                            │
                            ▼
   ┌────────────────────────────────────────────────────┐
   │   Gemini-1.5-Pro prompt:                           │
   │     "Task T. Below are video frames in random      │
   │      order. Reorder them by task-progress and      │
   │      assign each a percentage 0–100%."             │
   └────────────────────────┬───────────────────────────┘
                            ▼
              ordered list with values:
                frame 1 → 12%
                frame 2 → 38%
                ...
                frame N → 96%
                            │
                            ▼  reassemble in original order
              dense per-frame value v_t ∈ [0,100]
                            │
                            ▼
              r_t = v_t − v_{t-1}   (delta-progress)
                            │
                            ▼
              RL agent (or downstream eval)

   tested on 300+ real-world manipulation tasks (no per-task training)

The shuffle-and-reorder trick gives GVL three structural wins simultaneously:

• Dense per-frame value. Every frame gets a number, not just every clip pair — fixes RoboCLIP/RL-VLM-F's sparsity.
• VLM is forced to read content. The shuffle blocks surface-feature anchoring (smooth camera motion, consistent lighting). The VLM has to actually decide "this frame shows more task progress than that one."
• Zero per-task training. Same prompt template, any task, any embodiment. The paper reports usable rewards on 300+ real-world tasks spanning multiple labs and robots.

GVL's importance for the rest of this document is twofold. First, it is the cleanest training-free dense reward model on real robots — a usable baseline that any new paper must beat. Second, it directly inspires the 2025–2026 push to fine-tune (rather than just prompt) VLMs for this role: LRM (§16) is roughly "what if we trained Qwen3-VL on a GVL-style objective?", with predictable gains.

Text2Reward's pipeline is direct. The user provides (1) a task description in English and (2) a "Pythonic" representation of the environment — class names, attributes, helper functions. GPT-4 emits a dense reward function in Python. The function is plugged into the simulator; PPO runs; if the policy fails, a human gives a one-sentence critique ("the cube is being pushed off the table") and GPT-4 rewrites the reward.

   user provides:
     • task description (English)
     • environment Pythonic spec
       (class names, state attrs, helpers)
              │
              ▼
   ┌─────────────────────────────────────────────────────────┐
   │   GPT-4 prompt:                                         │
   │     "Write a Python reward(env) function for: 'pick     │
   │      up the cube and stack on the green block.'         │
   │      Use the Env spec below..."                         │
   └─────────────────────────┬───────────────────────────────┘
                             ▼
              candidate reward(env) → Python
                             │
                             ▼
              ┌───────────────────────────┐
              │   PPO  (simulator + new   │   train policy
              │   reward function)        │
              └─────────────┬─────────────┘
                            │  rollout videos
                            ▼
              user one-sentence critique
              "the cube falls off the table"
                            │
                            ▼
              GPT-4 rewrites reward
                            │
                            └─────────► repeat

   matches expert-written reward on 13 of 17 manipulation tasks

The headline result: on 13 of 17 manipulation tasks, the LLM-generated reward matches a human-expert-written one — sometimes after zero iterations, sometimes after two or three critique rounds. The structural payoff is large: an entire stage of robot research (reward engineering) compresses from days to minutes, and the resulting reward is symbolic, fast at training time, and inspectable when it breaks.

L2R from Google takes a slightly different angle — instead of producing a free-form Python function, it produces a structured reward template that is fed into an MPC controller, not into PPO. The L2R-style template approach is cleaner when there is a known low-level controller; Text2Reward's free-form approach is more general but produces messier reward functions that are easier to hack. Both papers establish the same point: the LLM is the reward designer, not the reward labeller.

Eureka's loop:

1. Hand GPT-4 the task description and the env source code.
2. Ask for K candidate reward functions in parallel (typically K=16).
3. Train K policies in IsaacGym simultaneously (this is the part that needs massive GPU parallelism — IsaacGym does it cheaply).
4. Score each policy on a task-specific fitness metric. Compute per-component statistics — which term of the reward was the bottleneck, which was hacked.
5. Feed all that back to GPT-4 with a "here are the candidates and their fitnesses; please produce a better population" prompt.
6. Repeat. Best-so-far reward is the final output.

   env source code + task description
                       │
                       ▼
   ┌──────────────────────────────────────────────────────────┐
   │     GPT-4   →   K reward functions (Python)              │
   └──────────────────────────┬───────────────────────────────┘
                              ▼
   ┌──────────────────────────────────────────────────────────┐
   │     IsaacGym × K  (massively parallel sim)               │
   │     train K policies, one per reward                     │
   └──────────────────────────┬───────────────────────────────┘
                              ▼
       per-reward fitness + per-component breakdown
       "reward[2] was hacked: dist-bonus dominated"
       "reward[7] best fitness 0.82 — close to expert"
                              │
                              ▼
   ┌──────────────────────────────────────────────────────────┐
   │     GPT-4 prompt:                                        │
   │     "Here are K rewards and their fitnesses.             │
   │      Produce a new population of K better ones,          │
   │      addressing the per-component issues."               │
   └──────────────────────────┬───────────────────────────────┘
                              ▼
                       (back to top, ~5 iters)

   beats human-engineered rewards on 83% of 29 RL tasks
   across 10 different robot morphologies

The headline number is striking: Eureka beats expert-hand-written reward functions on 83% of 29 tasks across 10 robot morphologies, including a pen-spinning task on a 5-finger dexterous hand that humans had not been able to reward-engineer at all. The structural insight is that the LLM is good at generating reward candidates but not at knowing which are good; coupling it to an outer evolutionary loop that tells it which candidates trained well closes the gap.

The remaining failure is sim-to-real. Eureka's rewards train great policies in simulation; the policies fall over on the real robot because the sim's physics, friction, sensor noise, and visual rendering are subtly off. DrEureka (Ma et al., RSS 2024) fixes this by extending the same loop to also generate domain-randomization parameters alongside the reward — what ranges of friction, mass, contact stiffness, sensor noise to train the policy across. The result: a quadruped trained from Eureka's reward + DrEureka's DR settings transfers to the real Unitree Go1 with zero real-world fine-tuning.

ReWiND's defining trick is a near-free way to generate failure data. The supervision problem with learned reward models has always been class imbalance: you have lots of successful demos and almost no labeled failures. ReWiND points out that any video played backwards is, with high probability, a labelled failure for the same task — the gripper releasing the cup is "picking it up"; closing the drawer is "opening it." Train a video-language transformer on the union of (forward, success) + (reverse, failure) data, and the resulting RM learns to penalize trajectories that move away from the goal as well as reward those that move toward it.

   ┌─────────────────────────────────────────────────────────┐
   │   small demo set per task (~20 successful videos)       │
   └────────────────────────┬────────────────────────────────┘
                            │
              ┌─────────────┴──────────────┐
              ▼                            ▼
       (forward, success)           (reversed, failure)
         label = "doing X"            label = "doing X"
                                      target reward = ↓

              └─────────────┬──────────────┘
                            ▼
   ┌─────────────────────────────────────────────────────────┐
   │   video-language transformer  r_φ(video, lang)          │
   │   trained to predict dense progress + final outcome     │
   └────────────────────────┬────────────────────────────────┘
                            │
                            ▼
              dense reward + success score
                            │
                            ▼
   ┌─────────────────────────────────────────────────────────┐
   │   downstream RL (PPO / GRPO)  or  best-of-N verifier   │
   │   bimanual setup: 5× real-world gain vs baselines       │
   └─────────────────────────────────────────────────────────┘

The headline real-world result is a 5× success-rate gain on bimanual manipulation tasks against the strongest baseline (a fine-tuned LIV variant). The structural reason it works is that the reverse-aug data fixes the gradient: a regular video-RM trained only on successful demos can't tell the policy which way is downhill because it has never seen a failure to learn from. Reverse-aug fills that gap for free.

ReWiND's place in the arc: it is the moment specialised, fine-tuned, robot-only reward models become competitive with prompted frontier VLMs on real manipulation. The frontier-VLM line continues (GVL still wins at zero-shot breadth), but for any single task family with a small demo set, a fine-tuned ReWiND-style model is the new default.

SARM addresses a different residual problem: long-horizon, multi-stage tasks (folding a T-shirt, setting a table) where the duration of each stage varies from one episode to the next. A monolithic progress predictor (GVL-style or ReWiND-style) fits the average duration and is then wrong on every episode that deviates. SARM trains a two-headed video transformer that emits both (a) the current stage label and (b) the within-stage progress — both supervised from a small set of stage-annotated demos.

The clean contribution is the LeRobot integration — SARM ships as a first-party reward provider you can drop into any LeRobot policy training loop, which makes it the easiest fine-tuned video RM to actually use as of 2025. Combined with the ReWiND-style reverse-augmentation trick, it is currently the strongest open recipe for long-horizon real-world tasks.

Robometer is in many ways the first paper that treats robot reward modelling as a data problem rather than an architecture problem. The team assembles RBM-1M — over a million robot trajectories spanning multiple embodiments, lighting conditions, tasks, and crucially including failures and suboptimal episodes. The model is a pretrained VLM head with two training objectives:

• Frame-level progress — dense per-frame value, GVL-style but learned from data rather than prompted.
• Trajectory-level preference — Bradley-Terry head over pairs of full trajectories.

Trained on both objectives jointly, the model learns from successful, suboptimal, and failed trajectories alike — the second objective gives signal even when both trajectories in a pair are unsuccessful (one was less bad). Performance on the companion RoboReward-Bench and on real-robot RL fine-tuning is well above prior fine-tuned RMs.

Two siblings extend the same line in 2026:

• LRM (Large Reward Models) — Tsinghua + collaborators. Qwen3-VL 4B/8B trained on 24 sources including Open X-Embodiment and HOI data. Designed specifically to be plugged into online closed-loop RL fine-tuning; reports +13 pp real-robot success in 30 RL iterations.
• RoboReward 4B / 8B — also Qwen3-VL backbone, trained on 45k scored episodes from OXE + RoboArena. Ships RoboReward-Bench, the first leaderboard for robot RMs across 22 VLM baselines. Notably beats GPT-5 and Gemini-Robotics-ER on real-robot tasks.

The three together are the "RewardBench moment" for robotics: a benchmark, a public model family, and a paper that demonstrates scaling-and-curation beats clever architecture. The same shape of result that Skywork-Reward-V2 produced for LLMs in 2025.

V-JEPA 2 is a self-supervised video encoder trained on 1M+ hours of internet video plus 62 hours of robot video with the JEPA (joint-embedding predictive architecture) objective — predict future-frame embeddings from past-frame embeddings, without reconstructing pixels. The trained encoder doubles as a value function: latent distance between the current frame and a goal-frame embedding is a usable, dense, zero-shot reward.

   ┌────────────────────────────────────────────────────────────┐
   │   pretrain on 1M+ hours internet video + 62h robot         │
   │                                                            │
   │   JEPA objective:                                          │
   │      enc(o_{t+1..}) ≈ predictor(enc(o_{..t}))              │
   │      in latent space, masked-prediction style              │
   └─────────────────────────┬──────────────────────────────────┘
                             ▼
   ┌────────────────────────────────────────────────────────────┐
   │   V-JEPA 2 latent encoder    z_t = enc(o_t)                │
   └─────────────────────────┬──────────────────────────────────┘
                             │
        ┌────────────────────┼─────────────────────┐
        ▼                    ▼                     ▼
   z_goal             z_current               z_imagined_rollout
   (target frame)     (now)                   (under policy)
        │                    │                     │
        └───────┬────────────┘                     ▼
                ▼                            value head
       r = − d(z_goal, z_current)         V(z_imag) → score
       (dense reward)                     (rollout in latent)
                │                                  │
                ▼                                  ▼
       RL agent / verifier             latent MPC planning

   zero-shot transfer to new robots, new labs, new tasks

Two structural properties make V-JEPA 2 the most credible "world-model-as-reward" candidate so far:

• Scaled pretraining. 1M+ hours of video is large enough that the latent encodes generic physical intuition — object permanence, contact, support. The value signal generalizes across embodiments and labs that the model has never seen, in a way no per-task RM does.
• Latent rollouts at test time. The companion V-JEPA 2-AC adds an action-conditioned predictor on top of the encoder, enabling latent-space MPC — imagine K candidate action sequences, roll each one in latent, pick the one with the best terminal $V$. The reward, the value function, and the world model are the same object.

What V-JEPA 2 is not: it is not a turnkey policy. The reward signal is sometimes noisy on tasks where the goal can't be cleanly described as a target frame embedding (compliant pouring, force-aware contact). And the model is heavy — running it as the reward on every RL step is the expensive end of the spectrum.

GRAPE's loop:

1. Start from a pretrained VLA (OpenVLA-7B).
2. Roll out many trajectories on real / sim tasks. Sort each trajectory into success vs failure via a VLM-generated stage-constraint check.
3. Construct (good, bad) trajectory pairs.
4. Apply DPO: update the VLA so it raises log-likelihood of good trajectories relative to bad ones, regularised against the pretrained reference policy.
5. Repeat.

   ┌────────────────────────────────────────────────────┐
   │   pretrained VLA  π_ref  (OpenVLA-7B)              │
   └──────────────────────┬─────────────────────────────┘
                          │ rollouts
                          ▼
        many trajectories on tasks
                          │
                          ▼
   ┌────────────────────────────────────────────────────┐
   │   VLM evaluator   →   per-stage constraints        │
   │     "did the gripper close around the cup?"        │
   │     "was the cup placed on the table?"             │
   └──────────────────────┬─────────────────────────────┘
                          ▼
            (good trajectory, bad trajectory) pairs
                          │
                          ▼
   ┌────────────────────────────────────────────────────┐
   │   DPO update on the VLA:                           │
   │   raise log-π(good) − log-π(bad)                   │
   │   with KL regulariser to π_ref                     │
   └──────────────────────┬─────────────────────────────┘
                          ▼
              new policy π_θ
                          │
                          └─────► (back to rollouts)

   no explicit reward model — DPO absorbs it into the policy

The shape of this loop is structurally the same as IPL (§8) — preferences in, no explicit reward model, policy updates directly. The pieces that are new in 2025: the policy is a 7B-parameter VLA (not a small MLP); the preferences come from a VLM evaluator (not a human); the stage-constraint formalism gives the VLM something concrete to check against rather than an open-ended "is this good." The result is a real-world preference-aligned VLA without any human in the loop after pretraining.

GRAPE is the meeting point of every line in this document. The reward model (from Part III) is implicit in the DPO loss. The labeller (from Part IV) is a VLM, not a human. The policy (from the VLA literature) is itself the pretrained foundation model. Every component of the loop has been replaced by a foundation model, and the result trains in a few hundred GPU-hours. This is what the hand-built-reward era became, after thirty years of replacing each component with a learned one.