This survey organizes the fragmented literature on progress reward models for robotics: models that score how far along a task is at each step, rather than only whether it eventually succeeded. It proposes a three-part frame (interface, construction method, data/benchmarks) and argues that most reported gains don’t cleanly separate whether a reward is faithful from whether it’s useful.
Suppose you’re training a robot policy to load a dishwasher, or evaluating rollouts from a Vision-Language-Action model before shipping. Today you mostly have one bit of feedback per episode: did it finish? That’s fine for short tasks, terrible for long ones. You can’t tell whether step 40 of a 60-step rollout is making things better, stalling, or actively undoing step 20. Progress rewards try to fill that gap by emitting a dense signal during execution, and they can double as evaluators for rollout reranking, data filtering, and failure detection.
The dominant baseline for the last few years has been VIP (Value-Implicit Pre-training)-style embedding methods: learn a representation where distance-to-goal drops monotonically along successful demos, and use that distance as reward. The survey’s argument is that this is one point in a much larger design space that nobody has been mapping consistently.
The survey is a taxonomy, not a new method. It cuts the field along three axes.
Axis 1: the interface. What does the model see, and what does it emit? Inputs range from a single frame, to a trajectory prefix, to before/after comparisons, to full simulator state. Goal specification is language, a goal image, a demonstration, or a structured predicate. Outputs are scalar scores, transition-level deltas (did this action help?), rankings between rollouts, or executable reward code.
Axis 2: how the reward is built. Four families:
•
Frozen foundation models used zero-shot, e.g. CLIP image-text similarity or TOPReward reading the probability of a “yes-completed” token from a VLM.
•
Learning from temporal or relative supervision: assume later frames in a successful demo are closer to done, or collect pairwise preferences.
•
Instruction-tuned progress prediction: fine-tune a VLM to answer “what percent done is this?” against rubric-scored trajectories.
•
Programmatic construction, e.g. Eureka generating and refining reward code with LLM feedback from training runs.
Axis 3: data and evaluation. Supervision comes from human labels, human-in-the-loop (annotate keyframes, interpolate the rest), or fully automated pipelines that exploit temporal order in demos or ground-truth simulator predicates. Evaluations split into progress fidelity (does the score match ground truth?), robustness/generalization (does it survive new tasks, viewpoints, embodiments, and failed rollouts?), and downstream utility (does it actually improve policy learning or rollout selection?).
The schematic pipeline the survey implicitly critiques looks like this:
# typical progress-reward setup
for obs in rollout:
r = progress_model(obs, goal_spec) # scalar, delta, or rank
buffer.add(obs, action, r)
# and then evaluate the model by:
# (a) correlation with labeled progress, OR
# (b) does RL with r train a better policy?
# these two evals validate different things and are often conflated
The prevailing assumption in the reward-model literature is that if your reward improves downstream policy success, the reward itself is good. This survey pushes back: a reward can improve a policy without faithfully tracking progress, and a faithful progress estimator can still be useless for closed-loop control if it’s sparse, laggy, or brittle. Fidelity and utility are separate axes and need separate benchmarks. The load-bearing evidence is the survey’s inventory of evaluation protocols showing that most papers report only one of the two, and that near-monotonic “progress curves” on successful demos can be produced by a model that has learned elapsed time rather than task grounding.
As a survey, the findings are structural rather than numerical.
•
The literature confounds different tasks under the label “reward model.” One paper predicts binary success from a still image; another predicts a progress percentage from video; another emits executable code. These are not comparable, but they cite each other as baselines.
•
Temporal-order supervision has a specific failure mode. If you train by assuming frame t has more progress than frame t-k in successful demos, the model can learn “time elapsed” as a shortcut. It then fails on non-monotonic executions: retries, reversals, plateaus, and partial failures, which are exactly the cases where a progress signal would be most valuable.
•
Goal grounding is undertested. Task-grounding benchmarks use counterfactual instructions or mismatched video-language pairs to check that the same frame gets different scores under different goals. Most methods aren’t evaluated this way, so smooth-looking reward curves may reflect generic scene change rather than task-conditional progress.
•
Answerability is nearly absent. Almost no method is evaluated on whether it knows when it can’t tell (occlusion, missing history, ambiguous goal). A confidently wrong dense reward is worse than a sparse honest one.
•
Programmatic rewards shift, not solve, the problem. LLM-generated reward code is interpretable and easy to edit, but the difficulty moves from “predict progress from pixels” to “decompose the task and expose the right state variables,” which is often the harder half.
If you’re building an agent evaluator or a rollout reranker, the practical takeaway is to pick your interface deliberately. Before/after delta models (e.g. VLAC) fit naturally into RL credit assignment. Ranking models fit rollout selection and best-of-N. Scalar-progress models fit dashboards and monitoring. Frozen-VLM scorers like TOPReward give you a zero-shot starting point with no labels, at the cost of needing normalization and prompt tuning before the numbers are trustworthy as rewards.
The survey itself is the artifact: it’s a map of ~40 methods with their interfaces, supervision sources, and evaluation protocols laid out consistently. No code repo or dataset is released, and the paper doesn’t publish a leaderboard or head-to-head numbers, because its point is that existing head-to-head comparisons don’t mean what people think they mean.
A reward that trains a better policy is not, by itself, a reward that measures progress. Decide which one you actually need. If you need progress (for monitoring, filtering, or planning), evaluate calibration and task grounding directly. If you need training signal, admit that and stop citing correlation-with-progress as if it settled the question.
•
It’s a survey with no new method and no empirical head-to-head, so the taxonomy is a lens, not a verdict. Anyone claiming a specific method is best on this frame is doing work the paper deliberately doesn’t do.
•
The critique of temporal-order supervision assumes you care about non-monotonic executions. If your deployment only ever runs near-optimal policies (e.g. after heavy behavior cloning), the shortcut-learning concern is muted.
•
The survey is written from a robotics-learning stance. Analogous ideas exist in LLM agent evaluation (process reward models, step-level verifiers), but the paper doesn’t bridge to that literature, so translating its taxonomy across domains is on the reader.