Reward Models — Survey
Categorized landscape · newest first within each category · primary focus: robotics · last updated May 2026
Scope. A categorized catalogue of reward models (RMs) across four families: (1) robotics & embodied AI — the deepest section, (2) process reward models (PRMs) for reasoning, (3) LLM / RLHF reward models, (4) vision / multimodal generation RMs. Within each category entries are ordered newest first.
Recency weighting. 2025–2026 entries get the most coverage; 2023–2024 covered well; 2021–2022 kept as low-density reference baselines. Watch-list rows are papers whose results look strongest but have no public code yet.
Legend.
2026 latest
2025
2024
≤2023 ·
★ = author-recommended first pick within the subsection ·
⚠ no code = paper / project only.
1. Robotics & Embodied AI Reward Models
Reward models for robot learning split along two axes: what they consume (RGB frame, video, language, proprio, action) and what they emit (sparse success, dense scalar, preference, executable code, value function). The subsections below group by mechanism. Category chips used in tables:
VLM VLM-as-reward
VID learned video RM
CLS success classifier / verifier
PREF preference-based RL
WM world-model / value
CODE LLM-written reward code
RLHF RLHF on VLA.
1.1 VLM-as-reward — zero-shot or fine-tuned VLMs scoring robot frames/videos.
The dominant 2024–2026 pattern: take a pretrained vision-language model, feed it RGB + language goal, read out a scalar or preference. Sub-split between training-free prompting and lightweight fine-tunes.
| # | Model | Yr | Team | Code / link | Backbone | Reward output | Input | Training source | Notes |
|---|
| ★1 | LRM (Large Reward Models) | 2026 | Tsinghua + collab. | project · arXiv | Qwen3-VL 4B / 8B | Dense per-frame (process + completion + temporal contrastive) | RGB video + language | 24 sources incl. OXE, HOI, sim | Online closed-loop policy refinement; +13 pp real-robot success in 30 RL iters. Strongest open robot RM at time of writing. |
| ★2 | RoboReward 4B / 8B | 2026 | OpenReview team | OpenReview · arXiv | Qwen3-VL 4B / 8B | 5-level end-of-episode progress | RGB rollout + task text | 45 k scored episodes (OXE + RoboArena) | First general-purpose open robot RM that beats GPT-5 and Gemini-Robotics-ER on real-robot tasks. Ships RoboReward-Bench. |
| 3 | MARVL | 2026 | arXiv 2602.15872 | arXiv | Fine-tuned VLM | Dense stage-projected scalar | RGB + language | Meta-World demos | Multi-stage subtask decomposition + task-direction projection; matches oracle dense reward. |
| 4 | TOPReward | 2026 | TOPReward team | GitHub · project | Qwen3-VL (frozen) | Dense scalar via internal token logit probabilities | RGB video + language | Zero-shot, no training | Bypasses text generation — reads progress from VLM logits. 0.947 VOC on 130+ tasks. |
| 5 | T²-VLM (training-free temporal RM) | 2025 | arXiv 2507.04789 | arXiv · ⚠ no code | Frozen VLM + Bayesian tracker | Dense scalar (temporally smoothed) | RGB + language | Training-free | Bayesian smoothing over VLM scores → monotonic rewards. Pure prompting baseline. |
| 6 | GVL (Generative Value Learning) | 2025 | UPenn / DeepMind | project · arXiv | Gemini-1.5-Pro | Dense % task-progress | Shuffled video frames + language | Zero-shot in-context | Reframes value estimation as frame-ordering. Scales to 300+ real-world tasks. ICLR 2025. |
| 7 | PrefVLM | 2025 | Ghosh et al. | arXiv · ⚠ no code | VLM + selective human label | Preference labels | RGB pairs + language | Meta-World + sparse human queries | Cuts human annotation 2× via VLM filtering + inverse-dynamics adaptation. |
| 8 | IKER (Iterative Keypoint Reward) | 2025 | Patel et al. | project · arXiv | GPT-4o + keypoint sampler | Python reward over keypoint geometry | RGB-D + language | Zero-shot | Real→sim→real loop with VLM-generated keypoint rewards. ICRA 2025. |
| 9 | ELEMENTAL | 2025 | Ga Tech | arXiv · ⚠ no code | LLM + VLM | Reward function (mixed code + feature) | Demos + language | Few demos + iterative VLM | Visual demos + NL guidance overcome LLM-only feature balancing. |
| 10 | RL-VLM-F | 2024 | Wang, Sun et al. | arXiv | Gemini / GPT-4V | Preference → learned scalar | Image pairs + text goal | Zero-shot VLM prefs | Pairwise VLM prefs across rigid / articulated / deformable manipulation. ICML 2024. |
| 11 | FuRL | 2024 | Fu et al. | GitHub · arXiv | LIV / VLM | Dense reward (fine-tuned) | RGB + text | Meta-World | Fixes VLM-reward misalignment via reward-alignment fine-tune + relay RL. ICML 2024. |
| Reference baselines — older, kept as historical anchors |
| R1 | VLM-RM (zero-shot) | 2023 | DeepMind | arXiv | CLIP / Flamingo | Sparse scalar | RGB + language | Zero-shot | Reference baseline — VLMs as direct RL reward without any training. |
| R2 | LIV | 2023 | UPenn / FAIR | arXiv | CLIP + VIP | Goal-conditioned value | RGB + language | EpicKitchens + robot | Multimodal value-implicit pretraining; widely cited dense-reward baseline. ICML 2023. |
| R3 | RoboCLIP | 2023 | USC | project · arXiv | S3D-CLIP | Cosine-similarity scalar | RGB video + language / demo | Zero-shot | One demo or text → reward. NeurIPS 2023. |
| R4 | MineCLIP | 2022 | NVIDIA (MineDojo) | MineDojo | CLIP-ViT | Dense shaping scalar | RGB + text | YouTube Minecraft | Contrastive video-text reward in open-world env. NeurIPS 2022. |
First pick: LRM for closed-loop RL fine-tuning, RoboReward for episode-level scoring, TOPReward if you want zero training.
1.2 Learned video / preference reward models.
Encoders trained explicitly to predict reward, progress, or preference from robot video — typically distilled or aligned beyond a frozen VLM.
| # | Model | Yr | Team | Code / link | Backbone | Reward output | Training source | Notes |
|---|
| ★1 | ReWiND | 2025 | USC LIRA | project · arXiv | Video-language transformer | Dense language-conditioned scalar | Small demo set; reverse-aug failures | Splits videos forward / reverse to synthesize failures. 5× real-world bimanual gain. CoRL 2025 Oral + RSS Best Paper. |
| 2 | SARM (Stage-Aware RM) | 2025 | Stanford / Berkeley | LeRobot · project · arXiv | Video transformer | Stage + dense progress | Long-horizon manipulation demos | Joint stage + progress prediction. Native LeRobot integration. Handles T-shirt folding (variable duration). |
| 3 | VLAC (Vision-Language-Action-Critic) | 2025 | arXiv 2509.15937 | arXiv · ⚠ no code | InternVL | Dense progress delta + done | Heterogeneous robot + human | Unified critic + policy; 30% → 90% real-world success in 200 episodes. |
| 4 | GenFlowRL | 2025 | Yu et al. | project · arXiv | Stable Diffusion (flow gen) | Dense shaped reward (delta-flow matching) | Cross-embodiment demos | Object-centric flow as reward proxy. ICCV 2025. |
| 5 | TeViR | 2025 | CAS | arXiv | Text-to-video diffusion | Dense distance-to-goal | Pretrained T2V | Compares predicted vs observed images for dense rewards across 13 sim/real tasks. |
| 6 | REDS | 2025 | Kim et al. | GitHub · project | Video encoder + contrastive | Dense subtask-aligned reward | Action-free video segments | EPIC-distance aligned dense reward. ICLR 2025. |
| 7 | Video-Language Critic | 2024 | Aalto / Manchester / Intel | GitHub · arXiv | CLIP4Clip | Contrastive temporal scalar | Open X-Embodiment | 2× sample efficiency on Meta-World via cross-embodiment RM. CoLLAs 2024. |
| 8 | Diffusion Reward | 2024 | Huang et al. | project · arXiv | Conditional video diffusion | Dense −conditional-entropy | Expert task videos | Rewards expert-like behaviors via diffusion conditional entropy. ECCV 2024. |
| 9 | Video2Reward | 2024 | arXiv 2412.05515 | arXiv · ⚠ no code | VLM + iterative refine | Reward code | Internet videos | Iterative VLM refinement of reward for legged-robot tasks from animal video. |
| R1 | VIPER | 2023 | Berkeley + DeepMind | project · arXiv | Autoregressive video transformer | Log-likelihood scalar | Expert videos | Reference baseline — video-prediction log-prob as reward across 28 tasks. NeurIPS 2023. |
First pick: ReWiND (RSS Best Paper, ships failure-aware data). Strong alternative: SARM for long-horizon tasks where stages matter (e.g., pour-place).
1.3 Success classifiers & trajectory verifiers.
Score whole trajectories or detect task completion. Increasingly used as VLA verifiers at test time (the “robot ORM” pattern).
| # | Model | Yr | Team | Code / link | Backbone | Reward output | Input | Notes |
|---|
| ★1 | Robometer | 2026 | arXiv 2603.02115 | GitHub · project | Pretrained VLM | Dense per-frame + global preference | RGB + lang | Trained on RBM-1M (1M+ trajectories incl. failures); dual objective learns from suboptimal data. RSS 2026. |
| 2 | RoVer | 2025 | arXiv 2510.10975 | arXiv · ⚠ no code | Robot Process Reward Model | Scalar + action direction | VLA candidate actions | Test-time verifier for VLAs — plug-and-play on OpenVLA, π0, RDT. |
| 3 | WorldGym (scalable video-world policy eval) | 2025 | arXiv 2511.11520 | arXiv · ⚠ no code | Video world model + VLM | Reward + rollout | RGB + actions | MC rollouts inside world model with VLM as reward. Used for policy evaluation. |
| 4 | WorldEval | 2025 | arXiv 2506.00613 | arXiv | Video world model | Eval scoring | RGB + actions | Frames policy eval as world-model rollouts. Companion to WorldGym. |
| R1 | SuccessVQA | 2023 | Berkeley / DeepMind | arXiv | Flamingo | Yes/no scalar | Clip + question | Reference baseline — success detection as VQA. ICML 2023. |
| R2 | Concept2Robot | 2020/21 | Stanford | project | Video action classifier (Sth-Sth) | Binary / score | RGB video | Reference baseline — Sth-Sth-V2 classifier as policy reward. |
| R3 | VICE | 2019 | Berkeley | arXiv | CNN success classifier | Sparse binary | Goal images | Reference baseline — classifier-from-goal-examples as RL reward. |
First pick: Robometer if you need to score real rollouts; RoVer as a test-time VLA verifier you can drop in over an existing policy.
1.4 Inverse RL / preference-based RL for robotics.
Learn a scalar reward from human (or VLM) pairwise preferences over trajectories. The classical PbRL line, with 2024–2025 work adding multimodal / non-Markovian models.
| # | Method | Yr | Team | Code / link | RM architecture | Pref source | Notes |
|---|
| ★1 | PRIMT | 2025 | Purdue / IU / UIUC | project · arXiv | VLM + LLM hybrid (neuro-symbolic) | Foresight + hindsight FM-synthesized | NeurIPS 2025 Oral. Counterfactual trajectory augmentation; cuts human cost dramatically. |
| 2 | ICPL | 2024 | Yu et al. | arXiv · ⚠ no code | GPT-4 in-context | Few in-context human prefs | 30× fewer queries than classical PbRL. NeurIPS 2024. |
| 3 | PrefMMT | 2024 | arXiv 2409.13683 | project · arXiv | Multimodal Transformer | D4RL + Meta-World | Hierarchical multimodal transformer for state-action preferences. |
| 4 | RIME | 2024 | Cheng et al. | arXiv | MLP reward + sample-selection | Noisy prefs | Warm-start discriminator → robust to label noise. ICML 2024. |
| 5 | IPL (Inverse Preference Learning) | 2023 | Stanford | arXiv | Q-function only (no RM) | Offline pref pairs | Eliminates the reward model — Q encodes all reward info. NeurIPS 2023. |
| R1 | Preference Transformer | 2023 | KAIST | arXiv | Transformer over trajectories | Human prefs | Reference baseline — non-Markovian preference reward. ICLR 2023. |
| R2 | SURF | 2022 | Park et al. | OpenReview | MLP | Few labels + pseudo-labels | Reference baseline — semi-supervised PbRL. ICLR 2022. |
| R3 | RUNE | 2022 | Berkeley | GitHub | MLP + uncertainty | Human prefs | Reference baseline — reward uncertainty as exploration bonus. |
| R4 | PEBBLE | 2021 | Berkeley | arXiv 2106.05091 | MLP | Human prefs | Reference baseline — unsupervised pre-training + relabeling. ICML 2021. |
2024–2025 trend: replace human teachers with VLM / LLM judges (PRIMT, ICPL) — the line between PbRL and §1.1 is collapsing.
1.5 World-model / value-function reward.
Reward arrives from a learned world model — either an explicit value head, prediction error, or VLM scoring inside imagined rollouts.
| # | Model | Yr | Team | Code / link | Backbone | Reward output | Notes |
|---|
| ★1 | V-JEPA 2 / 2-AC (v2.1) | 2026 | Meta FAIR | GitHub · arXiv | V-JEPA video encoder | Implicit value via latent distance | 1M+ hours internet video + 62 h robot. Zero-shot to new labs. Most mature open value-head world model. |
| 2 | World-VLA-Loop | 2026 | arXiv 2602.06508 | arXiv · ⚠ no code | State-aware video world model | Joint future obs + reward | Closed-loop joint refinement of world model and VLA. |
| 3 | RoboScape-R | 2025 | arXiv 2512.03556 | arXiv · ⚠ no code | Embodied world model | "Endogenous" intrinsic reward from dynamics | Reward = world-model state-transition disagreement. +37.5% OOD. |
| 4 | VLA-RFT | 2025 | arXiv 2510.00406 | arXiv · ⚠ no code | World model simulator | Dense trajectory reward | RL fine-tuning of VLA inside data-driven world model; beats SFT in 400 steps. |
| 5 | GE-Sim 2.0 (Genie Envisioner) | 2025 | AgiBot | project · arXiv | Action-conditioned video diffusion + judge | World-judge reward | Integrated world-judge scores generated rollouts against task instructions. |
| R1 | UniSim | 2024 | Berkeley / DeepMind | arXiv 2310.06114 | Video diffusion world sim | Reward + obs | Reference baseline — policy trained entirely in video world model; 81% real-world transfer. ICLR 2024 Outstanding. |
1.6 LLM-generated reward code.
Use an LLM to write a Python reward function (and sometimes the environment). The Eureka family. Best fit when you have access to env source code or a structured state.
| # | Method | Yr | Team | Code / link | LLM | Output | Notes |
|---|
| 1 | STRIDE | 2025 | arXiv 2502.04692 | arXiv · ⚠ no code | GPT-4 + agentic engine | Reward code | Agentic reward design for humanoid locomotion; 2.5× over Eureka. |
| 2 | REvolve | 2024 | Örebro | project · arXiv | GPT-4 + evolutionary | Reward code (NL feedback in loop) | Tested on AD + humanoid + dex manipulation. |
| ★3 | EurekaVerse | 2024 | UPenn | project | GPT-4 | Reward + environment code | Generates curriculum + reward together. CoRL 2024. |
| ★4 | DrEureka | 2024 | NVIDIA / UPenn | GitHub · arXiv | GPT-4 | Reward code + DR params | Joint reward + domain-randomization design for sim-to-real. RSS 2024. |
| 5 | Auto-MC-Reward | 2024 | Li et al. | project · arXiv | GPT-4 (3-agent loop) | Dense reward code | Designer / Critic / Analyzer agents close the loop. CVPR 2024. |
| 6 | Text2Reward | 2024 | HKU / NTU | project · arXiv | GPT-4 | Dense Python reward | First free-form dense reward code synthesis; matches expert rewards on 13/17 tasks. ICLR 2024. |
| 7 | Eureka | 2023 | NVIDIA / UPenn / Caltech | project · arXiv | GPT-4 (evolutionary) | Reward code | Beats human-engineered rewards on 83% of 29 RL tasks across 10 morphologies. ICLR 2024 Oral. |
| R1 | L2R (Language to Reward) | 2023 | Google DeepMind | project · arXiv | PaLM-2 | Sparse reward template + MPC | Reference baseline — language → reward → MPC. CoRL 2023. |
First pick depends on env: DrEureka if sim-to-real matters, EurekaVerse if you also want auto-generated environments.
1.7 Foundation-model RLHF for VLAs.
RL post-training of vision-language-action policies — the "RLHF for robots" frontier. Reward is supplied by a learned RM (§1.1–§1.3), verifiable env signal, or implicit preference.
| # | Method | Yr | Team | Code / link | Policy class | Reward signal | Notes |
|---|
| 1 | StARe-VLA | 2025 | arXiv 2512.05107 | arXiv · ⚠ no code | VLA + stage-aware reward | Progressive stage reward | Progressive stage-aware reward for VLA training. |
| 2 | CO-RFT | 2025 | arXiv 2508.02219 | arXiv · ⚠ no code | VLA + critic head | Q-value over chunked actions | Chunked offline RL fine-tuning of VLAs with critic head. |
| ★3 | RL-VLA empirical study | 2025 | arXiv 2505.19789 | project · arXiv | RL on OpenVLA-class | Verifiable env reward | Shows RL > SFT for VLA generalization in Execution / Semantics. Useful as recipe. |
| 4 | ManipLVM-R1 | 2025 | arXiv 2505.16517 | arXiv · ⚠ no code | Large VLM + RL reasoning | Verifiable reward | +144% IoU on affordance tasks via RL fine-tuning. |
| ★5 | GRAPE | 2025 | aiming-lab | GitHub · arXiv | OpenVLA + DPO | Implicit preference reward | Trajectory-level VLA preference alignment with VLM-generated stage constraints. Cleanest open RLHF-for-VLA recipe. |
| R1 | RT-2 | 2023 | Google DeepMind | arXiv | PaLI-X / PaLM-E VLA | (RL-fine-tunable; no explicit RM) | Reference baseline — the VLA paradigm that subsequent RL work builds on. |
1.8 Benchmarks & surveys (robotics).
| Item | Yr | Link | What it measures |
|---|
| RoboReward-Bench | 2026 | OpenReview · arXiv | End-of-episode VLM-reward accuracy across 22 VLMs on real-robot tasks. |
| OpenGVL | 2025 | GitHub · arXiv | VLM temporal progress prediction for data curation (human + robot embodiments). |
| RBM-1M (Robometer dataset) | 2026 | project | 1M+ multi-embodiment trajectories incl. failures for reward training / eval. |
| RoboArena | 2025 | arXiv | Distributed real-world generalist-policy evaluation (used by RoboReward). |
| B-Pref | 2021 | GitHub · arXiv | PbRL benchmark with simulated irrational teachers. NeurIPS 2021. |
| Survey: RL of VLA Models | 2025 | Awesome-RL-VLA · TechRxiv | Comprehensive survey of RL training of VLAs; explicit reward-modeling subsection. |
| Survey: Large VLM-based VLAs for Manipulation | 2025 | arXiv | VLM-based RM shaping covered as core dimension. |
| Survey: Foundation Models for Robot Manipulation | 2024/25 | arXiv | Reward learning treated as one of the key categories. |
| Survey: World Model for Robot Learning | 2026 | project · arXiv | Covers value / reward heads on world models. |
| List: Awesome-Embodied-VLA-VA-VLN | — | GitHub | Continuously updated index — useful tracker. |
Blog / lab pages:
NVIDIA — Eureka ·
RAI Institute — CoRL 2024 round-up ·
Robohub — ReWiND interview ·
ActiveModels — V-JEPA 2 guide.
2. Process Reward Models (PRMs) — reasoning
PRMs score every intermediate step of a reasoning trajectory rather than only the final answer. 2024 was the breakout year (Math-Shepherd, OmegaPRM, ReST-MCTS); 2025 brought generative / verifier-style PRMs that "think before scoring"; 2026 expands to multimodal, code, and agent trajectories.
2.1 Math / reasoning PRMs.
| # | Model | Yr | Team | Code / link | Base | Training | Notes |
|---|
| ★1 | DeepSeekMath-V2 | 2026 | DeepSeek-AI | arXiv | DeepSeek-V3-class | Verifier as central RM; verification-time scaling labels new hard proofs | Gold on IMO 2025 / CMO 2024; 118/120 on Putnam 2024. |
| 2 | R-PRM | 2025 | NJU NLP | GitHub · arXiv | Qwen2.5-Math | Cold-start CoT + self-evolving preference opt | +13.9 F1 on ProcessBench; EMNLP 2025. |
| ★3 | ThinkPRM | 2025 | CMU | GitHub · HF · arXiv | R1-Distill-Qwen 1.5B / 7B | Only 1k synthetic verification CoTs | Extremely data-efficient generative PRM; scales by thinking longer. |
| 4 | GenPRM | 2025 | Tsinghua | arXiv | Qwen2.5-Math | CoT + executes verification code | Scales test-time compute of PRMs themselves. |
| ★5 | Qwen2.5-Math-PRM 7B / 72B | 2025 | Alibaba Qwen | HF 7B · HF 72B · arXiv | Qwen2.5-Math | Hybrid MC + consensus filtering | Strongest open math PRM through most of 2025. |
| 6 | PRIME / Implicit PRM | 2025 | Tsinghua + UIUC | arXiv | Qwen2.5-Math-7B | Implicit PRM from ORM via log-likelihood ratios | 26.7% AIME pass@1 with only 10% of Qwen2.5-Math-Instruct's data. |
| 7 | PURE | 2025 | arXiv 2504.15275 | arXiv | Min-form credit assignment | Min-of-future-rewards | Eliminates reward hacking from summation-form credit; matches verifiable RL with 30% steps. |
| 8 | Skywork-o1-Open-PRM (1.5B / 7B) | 2024 | Skywork AI | HF · GitHub | Qwen2.5-Math-7B-Instruct | — | 7B matches Qwen2.5-Math-RM-72B; 96.7% GSM8K BoN. |
| 9 | AceMath-72B-RM | 2024 | NVIDIA | HF · project | — | BT outcome RM + step data | SOTA on AceMath-RewardBench rm@8. |
| 10 | Math-PSA / OpenR | 2024 | OpenR collab. | arXiv | Qwen2.5-Math-7B-Instruct | PRM800K + Math-Shepherd + MATH-APS | Reference open-source framework for o1-style reasoning. |
| 11 | OmegaPRM | 2024 | Google DeepMind | arXiv | Gemini Pro | Divide-and-conquer MCTS auto-labels 1.5M steps | Lifted Gemini Pro to 69.4% MATH (+36% rel). |
| 12 | ReST-MCTS* | 2024 | THUDM | GitHub · arXiv | — | Tree-search probability of correct answer | Self-bootstrapping policy + PRM. NeurIPS 2024. |
| R1 | Math-Shepherd | 2023 | DeepSeek + PKU | arXiv | Mistral-7B | MC rollout auto-annotation | Reference baseline — first widely-used automatic step-annotation. |
| R2 | PRM800K ("Let's Verify Step by Step") | 2023 | OpenAI | GitHub · arXiv | — | Human-labeled 800K step-correctness | Reference baseline — the dataset that started the field. |
2.2 Generative / verifier-style PRMs.
Emit a verification chain-of-thought instead of a scalar — interpretable and harder to hack.
| # | Model | Yr | Team | Link | Distinctive idea |
|---|
| 1 | PathFinder-PRM | 2025 | arXiv 2505.19706 | arXiv | Error-aware hierarchical PRM — math vs consistency dimensions before scoring. |
| 2 | Dyve (Think Fast and Slow) | 2025 | Awesome-PRM list | list | Adaptive system-1 / system-2 verification per step. |
| 3 | Retrieval-Augmented PRM | 2025 | arXiv 2502.14361 | arXiv | Two-stage retrieval of similar Q / steps → better OOD generalization. |
| 4 | AURORA | 2025 | arXiv 2502.11520 | arXiv | Universal generative PRM; ensemble prompting + reverse verification; ships UniversalBench. |
| 5 | FreePRM | 2025 | arXiv 2506.03570 | arXiv | Label-free PRM from outcome signals; beats supervised Math-Shepherd by +24 F1. |
2.3 Code & agent PRMs.
| # | Model | Yr | Team | Link | Notes |
|---|
| 1 | SecCodePRM | 2026 | CMU | GitHub · arXiv | Step-level security scores; +12% over LLMxCPG static-analysis pipeline. |
| 2 | AgentPRM | 2025 | arXiv 2511.08325 | arXiv | Step-wise promise + progress; TD-based labels. >8× compute-efficient vs baselines. |
| 3 | SWE-PRM ("Act Like You're Paying") | 2025 | IBM | arXiv | Taxonomy of agent inefficiencies. SWE-bench Verified 40.0 → 50.6%, $0.2 added cost/trajectory. |
| 4 | AgentRM | 2025 | THUNLP | GitHub · arXiv | Generalizable RM across 9 agent tasks (web, embodied, games, tools). ACL 2025. |
| 5 | CodePRM | 2025 | ACL Findings | ACL | Reasoning + execution feedback input; pairs with Generate-Verify-Refine decoding. |
2.4 Multimodal PRMs.
Active 2025–2026 frontier.
| # | Model | Yr | Team | Link | Notes |
|---|
| 1 | VisualPRM-8B | 2025 | Shanghai AI Lab / InternVL | arXiv | Ships VisualPRM400K + VisualProcessBench. +5.9 pts on 7 multimodal benches on InternVL2.5-78B. |
| 2 | Athena-PRM | 2025 | arXiv 2506.09532 | arXiv | SOTA on VisualProcessBench with just 5k samples. |
| 3 | MM-PRM | 2025 | arXiv 2505.13427 | arXiv | Multimodal PRM family. |
| 4 | VLPRM | 2025 | arXiv 2509.23250 | arXiv | Vision-language PRM. |
2.5 Benchmarks & survey.
| Item | Yr | Link | Focus |
|---|
| PRMBench | 2025 | GitHub · arXiv | Fine-grained: simplicity / soundness / sensitivity. 83,456 step labels. ACL 2025. |
| ProcessBench (Qwen) | 2024 | arXiv | First-error identification on 3.4k competition / olympiad problems. |
| VisualProcessBench | 2025 | arXiv | Multimodal step verification. |
| Socratic-PRMBench | 2025 | arXiv | Reasoning-pattern coverage for PRMs. |
| MR-Ben | 2024 | arXiv | Meta-reasoning across physics / chem / logic / code. |
| AceMath-RewardBench | 2024 | HF | Math reward-model leaderboard (rm@8). |
| Survey: Survey of Process Reward Models | 2025/26 | arXiv · Awesome-PRMs | Outcome → process supervision; notes length / verbosity hacking. |
| Insight: GRPO is Secretly a PRM | 2025 | arXiv | Shows GRPO+ORM ≡ MC-based PRM-aware RL; proposes λ-GRPO to fix step-imbalance. |
3. LLM / RLHF Reward Models
Reward models for aligning chat / instruction-following LLMs via RLHF, RLAIF, DPO, and best-of-N. 2024 saw open RMs catch up to closed labs; 2025 split into two frontiers — (a) scale-and-curate scalar RMs and (b) reasoning / generative critics — while RLVR (Tülu 3) and rule-based rewards increasingly replace learned RMs for verifiable tasks. Process / step-level RMs are covered separately in §2.
3.1 Scalar / Bradley–Terry preference RMs.
The classifier-style RM that dominates production RLHF pipelines.
| # | Model | Yr | Team | Code / link | Base | Output | Training data | Notes |
|---|
| ★1 | Skywork-Reward-V2 | 2025 | Skywork AI / 2050 Research | HF · GitHub · arXiv | Llama-3.2 / Qwen3 0.6–8B | Scalar BT | SynPref-40M (26M curated pairs) | Top of RewardBench v1/v2, PPE, RM-Bench, RMB, JudgeBench. 1.7B model beats prior 70B SOTA. |
| 2 | Decision-Tree-Reward-Gemma-2-27B | 2025 | RLHFlow | blog · HF | Gemma-2-27B | Multi-objective + tree gate | HelpSteer2 + Skywork-Reward-Gemma | Briefly 95.4% RewardBench SOTA; interpretable decision paths. |
| 3 | INF-ORM-Llama3.1-70B | 2025 | INF.AI | HF | Llama-3.1-70B-Instruct | Scalar (scaled BT) | INF-ORM-Pref-Magnitude-80K | Briefly #1 on RewardBench in early 2025 via magnitude-weighted BT. |
| 4 | Llama-3.1-Nemotron-70B-Reward | 2024 | NVIDIA | HF · arXiv | Llama-3.1-70B-Instruct | BT + SteerLM regression hybrid | HelpSteer2-Preference (10k pairs) | Powered Nemotron-70B-Instruct policy. |
| 5 | Skywork-Reward (v1) | 2024 | Skywork AI | HF · GitHub · arXiv | Llama-3.1-8B / Gemma-2-27B | Scalar BT | Skywork-Reward-80K | "Bag of tricks" paper; canonical open RM baseline. |
| 6 | Athene-RM-70B / -8B | 2024 | Nexusflow | HF 70B | Llama-3-70B/8B-Instruct | Scalar BT | Curated Arena-style | Trained the Athene-70B policy that briefly matched GPT-4 on Arena. |
| ★7 | ArmoRM-Llama3-8B | 2024 | RLHFlow / UIUC | HF · arXiv | Llama-3-8B | Multi-objective + MoE gate | UltraFeedback + HelpSteer + others | 19 fine-grained objectives with context-aware MoE gating. Long-time RewardBench leader; still a strong baseline. |
| 8 | InternLM2-Reward (1.8B / 7B / 20B) | 2024 | Shanghai AI Lab | HF | InternLM2-Chat-SFT | Scalar BT | 2.4M EN+ZH pairs | Strong bilingual RM family. |
| 9 | GRM-Llama3-8B | 2024 | Ray2333 et al. | HF · arXiv | Llama-3-8B | Scalar + generative regularization | UltraFeedback + Unified-Feedback | SFT-regularization mitigates reward hacking. |
| 10 | Eurus-RM-7B | 2024 | OpenBMB / Tsinghua | GitHub · arXiv | Mistral-7B | Scalar BT | UltraInteract + UltraFeedback + UltraSafety | RM specialized for reasoning / code preferences. |
| R1 | Starling-RM-7B-alpha | 2023 | Berkeley NEST | HF · project | Llama-2-7B | K-wise scalar | Nectar (183k prompts, GPT-4) | Reference baseline — popular early open RM. |
| R2 | UltraRM-13B | 2023 | OpenBMB / Tsinghua | HF · arXiv | Llama-2-13B | Scalar BT | UltraFeedback + HH-RLHF + SHP + summarization | Reference baseline — basis for many follow-ups. |
| R3 | OpenAssistant RM (DeBERTa-v3) | 2023 | LAION / OA | HF | DeBERTa-v3-large (435M) | Scalar BT | OA + WebGPT + summary | Reference baseline — first broadly-used small drop-in scorer. |
First pick: Skywork-Reward-V2 if SOTA matters; ArmoRM if multi-objective interpretability matters; Nemotron-70B-Reward if you're already in the NVIDIA stack.
3.2 Generative / LLM-as-a-judge RMs.
Produce a critique + verdict in natural language; scale with inference-time compute.
| # | Model | Yr | Team | Code / link | Base | Output | Notes |
|---|
| 1 | J1 (Llama-8B / 70B) | 2025 | Meta FAIR | arXiv | Llama-3 8B / 70B | Reasoning + pairwise verdict | RL-trained "thinking" judge; beats DeepSeek-GRM & distilled R1 judges on non-verifiable tasks. |
| ★2 | RM-R1 | 2025 | UIUC | GitHub · arXiv | Qwen2.5 / Llama-3 | CoT + Chain-of-Rubrics + verdict | "Reward modeling as reasoning"; +4.9% over much larger RMs on three RM benches. |
| 3 | DeepSeek-GRM-27B | 2025 | DeepSeek + Tsinghua | arXiv | DeepSeek-V2-Lite-27B | Pointwise generative critique + score | Self-Principled Critique Tuning (SPCT); 27B matches 70B+ via inference scaling. |
| 4 | Skywork-Critic-Llama-3.1 (8B / 70B) | 2024 | Skywork AI | HF 70B | Llama-3.1-Instruct | Pairwise verdict | Topped RewardBench among generative judges in Sep 2024. |
| 5 | Self-Taught Evaluator | 2024 | Meta FAIR | arXiv | Llama-3-70B-Instruct | CoT pairwise judge | Self-bootstrapped: RewardBench 75.4 → 88.3 with zero human labels. |
| 6 | GenRM / GenRM-CoT | 2024 | Google DeepMind | arXiv | Gemma / PaLM-2 | Next-token Yes/No + CoT | Foundational "reward modeling as next-token prediction". +20% on reasoning BoN. |
| 7 | Prometheus-2 (7B / 8x7B) | 2024 | KAIST | GitHub · arXiv | Mistral-7B / Mixtral-8x7B | Score + critique (direct & pairwise) | Most-used open evaluator; merges absolute scoring + pairwise. |
| 8 | Themis-8B | 2024 | PKU ONELab | GitHub · arXiv | Llama-3-8B | Aspect-aware score + explanation | 58 NLG datasets, 500k samples; beats GPT-4 on several tasks. |
| R1 | Auto-J-13B | 2023 | GAIR / SJTU | GitHub · arXiv | Llama-2-13B-chat | Pairwise + critique | Reference baseline — early influential open generative judge. |
| R2 | JudgeLM (7B / 13B / 33B) | 2023 | BAAI | GitHub · arXiv | Vicuna | Pairwise score + rationale | Reference baseline — established fine-tuned LLM judges as a viable GPT-4 alternative. |
3.3 Multi-objective / fine-grained RMs.
| # | Model | Yr | Team | Link | Output | Notes |
|---|
| 1 | HelpSteer3 / Llama-3.3-Nemotron-Reward | 2025 | NVIDIA | HF dataset | 5+ attributes (helpful, correct, coherent, complex, verbose) | Newest HelpSteer with multilingual + STEM expansions. |
| 2 | Decision-Tree-Reward | 2025 | RLHFlow | blog | Multi-objective + decision-tree gate | Combines multi-obj regression with small decision tree for explainability. |
| 3 | ArmoRM | 2024 | RLHFlow | arXiv | 19 fine-grained objectives + MoE | Canonical multi-objective RM (also listed in 3.1). |
| 4 | HelpSteer2 SteerLM-RM | 2024 | NVIDIA | arXiv | 5-dim regression | Permissive 10k dataset that drove SteerLM-style regression RMs to SOTA. |
| R1 | FLASK | 2023 | KAIST | arXiv | 12 skill dimensions | Reference baseline — established fine-grained skill-set evaluation. ICLR 2024. |
3.4 Pairwise & reward-free methods.
Included for context — RM-free alternatives that often replace explicit RMs in production.
| # | Method | Yr | Link | Notes |
|---|
| 1 | KTO | 2024 | arXiv | Prospect-theory loss; only binary (good/bad) labels — no pairs. |
| 2 | PairRM (0.4B) | 2023 | HF | DeBERTa pairwise scorer; near-GPT-4 agreement at tiny size. Widely used for DPO data curation. |
| 3 | DPO implied reward | 2023 | arXiv | Reward-free; policy log-ratio with reference encodes implicit reward. |
| 4 | IPO | 2023 | arXiv | Squared-loss fix for DPO's unbounded reward-gap growth. |
| R1 | LLM-Blender | 2023 | GitHub | Reference baseline — pairwise ranking + generative fusion ensemble; parent of PairRM. |
3.5 Constitutional / rule-based / RLAIF reward systems.
| # | System | Yr | Team | Link | Notes |
|---|
| 1 | OpenAI Rule-Based Rewards (RBR) | 2024 | OpenAI | blog · arXiv · GitHub | LLM-graded composable rule prompts as RL reward. F1 91.7 → 97.1 on safety. |
| 2 | Meta-Rewarding LM | 2024 | Meta FAIR | arXiv | Meta-judge role on top of Self-Rewarding — judges the judgments. |
| 3 | Self-Rewarding LM | 2024 | Meta FAIR | arXiv | Single LLM acts as actor + judge; iterative DPO improves both. |
| R1 | Anthropic CAI | 2022 | Anthropic | blog · arXiv | Reference baseline — introduced RLAIF: RM from AI prefs against a written constitution. Basis for Claude HH training. |
3.6 Benchmarks & surveys.
| Item | Yr | Link | What it measures |
|---|
| RewardBench 2 | 2025 | arXiv · HF | Harder multi-skill RM accuracy; correlates with BoN + PPO downstream. ~20 pt lower than v1. |
| JudgeBench | 2024 | arXiv | Verifiable knowledge / reason / math / code pairs. ICLR 2025. |
| PPE (Preference Proxy Evaluations) | 2024 | arXiv · GitHub | First RM bench predictive of post-RLHF Arena scores. 16k Chatbot-Arena pairs + 80k verifiable completions. |
| RM-Bench | 2024 | arXiv | Sensitivity to subtle content edits + style-bias resistance. ICLR 2025 Oral. |
| RMB | 2024 | arXiv | 49 real-world scenarios × helpful / harmless. 18k BoN-aware pairs. |
| MJ-Bench | 2024 | arXiv | Multimodal judges for T2I: alignment / safety / quality / bias. |
| RewardBench (v1) | 2024 | arXiv · GitHub | First standardized RM leaderboard; de-facto baseline. |
| Survey: Comprehensive Survey of Reward Models | 2025 | arXiv | Taxonomy, applications (VL, multilingual, RAG), challenges, future. |
| Survey: Secrets of RLHF Part II — Reward Modeling | 2024 | HF paper | Practitioner guide to RM training tricks. |
| List: Awesome-Reward-Models | — | GitHub | Continuously updated index. |
Blog / lab posts:
AI2 — RewardBench ·
AI2 — Tülu 3 / RLVR ·
HF — Illustrating RLHF ·
Interconnects — Tülu 3 ·
Interconnects — o3 over-optimization ·
RLHFlow — Multi-objective RM & MoE.
4. Vision / Multimodal Generation Reward Models
RMs for aligning image, video, 3D, and motion generation. 2025 shifted from CLIP-based scorers to VLM backbones (Qwen2-VL, InternVL, LLaVA-OneVision), and from scalar outputs to generative / CoT scoring with explicit rationales.
4.1 Text-to-image (T2I) reward models.
| # | Model | Yr | Team | Code / link | Backbone | Output | Notes |
|---|
| 1 | UnifiedReward-Flex | 2026 | CodeGoat24 (Fudan) | GitHub · arXiv | Qwen2.5-VL | Pairwise + pointwise; dynamic hierarchical multi-aspect | Context-adaptive reasoning instantiates fine-grained criteria on-the-fly; tested with FLUX.1-dev, FLUX.2-klein, Wan2.1/2.2. |
| ★2 | RewardDance | 2025 | ByteDance | HF · arXiv | InternVL 1B–26B | Generative "yes-token" prob (pairwise) | First serious scaling-laws study for visual RMs; reward variance stays high (resists hacking) up to 26B. |
| ★3 | HPSv3 | 2025 | MizzenAI | GitHub · HF · arXiv | Qwen2-VL-7B | Scalar + uncertainty-aware ranking | HPDv3 — 1.08M text-image pairs, 1.17M comparisons. Introduces Chain-of-Human-Preference (CoHP). ICCV 2025. |
| 4 | UnifiedReward / -Think | 2025 | CodeGoat24 / Fudan | GitHub · arXiv | LLaVA-OneVision | Pairwise + pointwise across image/video/understanding | First unified RM covering understanding + image + video. Think variant adds GRPO RL on long CoT. |
| 5 | IXC-2.5-Reward | 2025 | Shanghai AI Lab (InternLM) | arXiv | InternLM-XComposer2.5 | Scalar reward head over text / image / video | Top open RM on VL-RewardBench (70.0% macro acc, beats Gemini-1.5-Pro & GPT-4o). ACL Findings 2025. |
| 6 | VisionReward-Image | 2024 | Zhipu AI / Tsinghua | GitHub · arXiv | CogVLM2 backbone | Multi-dimensional fine-grained (interpretable linear weights) | Built for stable RLHF; pairs with MPO algorithm. AAAI 2026. |
| 7 | MPS (Multi-dimensional Preference Score) | 2024 | Zhang et al. | arXiv | CLIP + preference-condition module | 4 dimensions (aesthetics / alignment / detail / overall) | MHP — 918k choices on 607k images. CVPR 2024. |
| 8 | RAHF (Rich Automatic Human Feedback) | 2024 | Google + UCSD/USC/Cambridge | GitHub · arXiv | ViT + T5 | Plausibility / alignment / aesthetic scalars + spatial heatmaps + token-level misalignment | CVPR 2024 Best Paper. Goes beyond a single scalar. |
| 9 | Q-Align | 2024 | Q-Future | GitHub · arXiv | mPLUG-Owl2 LMM | Discrete text-level rating mapped to scalars | Treats quality scoring like a human study via text levels. ICML 2024. |
| 10 | VQAScore | 2024 | CMU / Meta | GitHub · arXiv | CLIP-FlanT5 | P("Yes" | "Does this show {text}?") | Beats CLIPScore / PickScore on compositional prompts; zero-shot image / video / 3D. ECCV 2024. |
| R1 | PickScore + Pick-a-Pic | 2023 | Stability AI / Tel Aviv | arXiv | CLIP-H finetune | Scalar | Reference baseline — 500k+ in-the-wild prefs. NeurIPS 2023. |
| R2 | HPSv2 | 2023 | CUHK MMLab | GitHub · arXiv | CLIP finetune | Scalar | Reference baseline — HPDv2 with 798k pairwise choices. |
| R3 | ImageReward | 2023 | Zhipu / Tsinghua | GitHub · arXiv | BLIP + MLP head | Scalar | Reference baseline — first general-purpose T2I RM; ships ReFL fine-tuning. NeurIPS 2023. |
4.2 Text-to-video (T2V) reward models.
| # | Model | Yr | Team | Code / link | Backbone | Output | Notes |
|---|
| 1 | PhysCorr / PhysicsRM | 2025 | arXiv 2511.03997 | arXiv | — | Dual PhyScore (intra-object stability + inter-object interaction) | Plugs into PhyDPO for physics-grounded video DPO. |
| ★2 | VideoScore2 | 2025 | TIGER-AI-Lab | GitHub · arXiv | Qwen2.5-VL-7B | Visual / text-alignment / physical-commonsense scores + CoT | "Think before scoring" — first generative video RM with structured rationales. VideoFeedback2 (27k annotated). |
| ★3 | VideoReward / VideoAlign | 2025 | Kling team (Kuaishou) | GitHub · HF · arXiv | VLM + Bradley-Terry-with-Ties | Visual Quality / Motion Quality / Text Alignment | 182k prompt-video triplets; ships Flow-DPO / Flow-RWR / Flow-NRG for rectified-flow video models. NeurIPS 2025. |
| 4 | VisionReward-Video | 2024 | Zhipu / Tsinghua | GitHub · arXiv | CogVLM2-Video | Fine-grained multi-aspect (linearly weighted) | Beats VideoScore by +17.2% pair-acc; CogVideoX+MPO yields ~27% win-rate uplift. AAAI 2026. |
| 5 | LiFT / LiFT-Critic | 2024 | CUHK | arXiv | VILA-1.5 VLM | Semantic consistency / motion smoothness / video fidelity + reasoning | First end-to-end T2V RLHF pipeline (RM + DPO on CogVideoX). |
| R1 | VideoScore + VideoFeedback | 2024 | TIGER-AI-Lab | GitHub · arXiv | Mantis-Idefics2-8B | 5 aspects (VQ / TC / Dynamic / TextAlign / Factual) | Reference baseline — first fine-grained learned video RM. EMNLP 2024. |
4.3 3D / motion reward models.
| # | Model | Yr | Team | Link | Output | Notes |
|---|
| 1 | PhyMotion | 2026 | phy-motion.github.io | project | Three physics-grounded axes (kinematic / contact-balance / dynamic) | SMPL recovery → MuJoCo retarget → physics scoring; beats 2D-image rewards for motion realism. |
| 2 | DreamCS / RewardCS | 2025 | arXiv 2506.09814 | arXiv | Geometry-aware mesh quality | First 3D geometry-aware RM trained on unpaired data; beats 2D RMs for text-to-3D. |
| 3 | MotionCritic | 2025 | Ou et al. | GitHub · arXiv | Plug-and-play critic; perceptual-alignment loss | MotionPercept — 52k human pref pairs over generated motions. ICLR 2025. |
| R1 | DreamReward / Reward3D | 2024 | THU | GitHub · arXiv | Scalar | Reference baseline — first general-purpose 3D RM. 25k expert comparisons. ECCV 2024. |
4.4 MLLM-as-judge for generation.
Training-free; relies on a strong MLLM.
| Method | Yr | Link | Notes |
|---|
| GPT-4o / GPT-4V as judge | 2024/25 | — | Standard prompted scoring; backbone for VIEScore and many VBench/T2V-CompBench variants. Matches human-human correlation on generation, weaker on editing. |
| Gemini-1.5/2.5-Pro as judge | 2024/25 | — | Long-context advantage for video judging; used as generalist scorer in VBench-2.0. |
| VIEScore | 2024 | GitHub · arXiv | Semantic + perceptual scores with rationales; GPT-4o variant reaches ~0.40 Spearman vs human 0.45. ACL 2024. |
| VideoGen-Eval | 2025 | arXiv | Agent-based judge — LLM structuring + MLLM judgment + temporal patch tools over 700 prompts / 12k videos. |
4.5 Benchmarks for evaluating vision-generation RMs.
| Benchmark | Yr | Link | Focus |
|---|
| VideoGen-RewardBench | 2025 | HF | 26.5k (prompt, vidA, vidB) triplets; expert pairwise; 12 SOTA T2V models. |
| VBench-2.0 | 2025 | arXiv · project | 5 "intrinsic faithfulness" dimensions (Human / Controllability / Creativity / Physics / Commonsense). |
| VideoPhy / VideoPhy-2 | 2024/25 | arXiv | Physical-commonsense; ships VideoPhy-2-AutoEval auto-judge. ICLR 2026. |
| T2V-CompBench | 2025 | arXiv | 1,400 prompts × 7 compositional axes across 23 T2V models. CVPR 2025. |
| VBench / VBench++ | 2024 | GitHub | 16-dim T2V / I2V benchmark; dominant T2V leaderboard. CVPR 2024 Highlight. |
| GenAI-Bench | 2024 | arXiv | 1,600 compositional T2I/T2V prompts + 80k human ratings; default for VQAScore. |
| ImagenHub | 2024 | arXiv | Unified library + benchmark over 7 conditional image-generation tasks. ICLR 2024. |
| EvalCrafter | 2024 | — | 17 sub-dimensions complementary to VBench. CVPR 2024. |
5. Cross-cutting trends & reading list
Trends across all four families (2025 → 2026)
- From scalar to generative. Every category is shifting from a single-number reward to a generated critique + verdict — RM-R1, J1, GenRM (LLM); ThinkPRM, GenPRM, R-PRM (PRMs); RewardDance, VideoScore2, UnifiedReward-Think (vision); SARM, ReWiND (robotics). More interpretable, harder to hack, scales with inference compute.
- VLM backbones replace CLIP. Across vision RMs and robot RMs alike, Qwen2-VL / InternVL / LLaVA-OneVision dominate. CLIP-era scorers (HPSv2, PickScore, ImageReward) are now baselines.
- Label-free / weakly-labeled RMs. PRIME, FreePRM, Implicit-PRM, Self-Taught Evaluator, GRAPE — derive step or trajectory rewards from outcomes or self-bootstrapping rather than human labels.
- RMs as test-time verifiers. RoVer (robots), SWE-PRM (agents), ThinkPRM (math) — the RM moves from training-time signal to inference-time best-of-N / search rerank.
- Reward hacking awareness. Style-bias resistance (RM-Bench), min-form credit (PURE), temporal smoothing (T²-VLM), physics grounding (PhysCorr, PhyMotion, RoboScape-R) — explicit mitigations are now table stakes.
- RLVR / rule-based rewards replace RMs where verifiability exists. Tülu 3, OpenAI RBR, RL-VLA empirical study — when env signal is verifiable, learned RMs are increasingly skipped.
- Robotics catches up fast. The 2025–2026 wave (LRM, RoboReward, ReWiND, Robometer, SARM) finally brings RewardBench-style benchmarks (RoboReward-Bench, OpenGVL) and scaled multi-source datasets to the robot side.
Top reading list