September 4, 2026 — If you evaluate a manipulation policy today, you probably report one number: the fraction of trials that reached the goal. It is a convenient summary, and three recent benchmarks each attack it from a different side. One asks what happens after the policy makes a mistake mid-episode. One asks whether a completed contact action touched the right place while staying below the benchmark’s force gate. One asks whether the model that assigned the success label was itself correct. Use them to ask three questions: can the robot recover, does it meet the physical constraints, and can you trust the success label?
Recovery is a different capability from completion
Start with the conventional setup. A policy gets a language instruction and an initial state, emits actions, and you check the goal condition at the end. A final success score does not show how the policy recovered from a mistake, or why it failed to recover.
LIBERO-Recover reverses the framing. It collects failure states that embodied models actually produced during execution, then restarts evaluation from those states. Not every mid-episode state change counts. The benchmark’s operational definition is narrow: a state is a failure state when the preceding action induced a task-relevant change that invalidates the current plan, yet the original goal is still reachable through further interaction. If continuing the plan would have worked anyway, that is a normal variation, not a failure. From there the authors sort recoverable failures into four levels of increasing difficulty: retrying the same action when nothing really moved (L1), adapting the action to a shifted but still usable object (L2), first restoring an object’s state before resuming the task (L3), and clearing an environmental obstruction that the original task never touched (L4). LIBERO-Recover
The distinction is whether the robot can adjust its next action or needs to change its plan. Suppose a grasp shifts an upright bowl but leaves it directly usable. An L2-style response adapts the next action to the pose the camera now shows. If the bowl is tipped onto its side instead, an L3 response needs the policy to represent that state change, insert a righting sub-goal, and only then return to the instructed manipulation — a plan edit, not a servo correction.
The results follow that split. Across six model configurations — four vision-language-action policies and two world-action models, each task run for ten trials with slight object-position perturbations under a difficulty-scaled time limit — every model lost more than half its performance when moved to naturally occurring failures, and success rates fell consistently from L1/L2 to L3/L4. Rankings also moved: Wan2-Policy beats GR00T-N1.5 by 14.4 points on LIBERO-100 but sits 5.0 points behind it on the corresponding failure set. Averaged over models, LIBERO-100 success fell from 26.2% to 0.3% after failure. That last figure is worth reading precisely — it is the average over these six configurations on LIBERO-100, not a general claim about robot recovery rates.
The authors also report that finer-grained action execution and temporal context help recovery, and that recovery training transferred only limitedly back to failures encountered during standard execution. So the fix is not simply “add recovery data.” The practical consequence is narrower and firmer: a leaderboard position on clean rollouts tells you little about behavior after the first mistake, and if your deployment involves long horizons where mistakes compound, recovery needs its own evaluation.
“Completed” does not mean “physically valid”
The second paper moves to a setting where the physics of contact is the point. A physics-consistent benchmark for contact-rich assistive care evaluates seven micro-tasks — touch, push, scrub variants — on three soft body regions of a compliant medical-care manikin lying on a nursing bed, with a Franka Panda under Cartesian impedance control. Twenty initializations times seven tasks gives 140 runs per method.
What makes it instructive is that it scores several properties that a single completion rate fuses together: whether the original evaluator called the run a success, whether contact landed in the instructed body region, whether the action made sustained progress, and whether peak force stayed under a safety gate. The gates come from published region-wise pain-onset limits (100 N forearm muscle, 100 N deltoid, 60 N abdomen) with a uniform 0.35 discount, yielding 35 N / 35 N / 21 N. Those are deliberately conservative benchmark thresholds derived from literature, not independently measured clinical limits, and the paper is explicit that its dimensions describe physical contact rather than serving as validated care scores. safety gate (force gate)
The divergences are the finding. A structured LLM-augmented state machine — vision at initialization, then scripted Cartesian primitives selected by language — achieved the highest completion, 102 of 140, yet dropped to 79 of 140 once the force gate was applied, with 36 exceedances across the suite (including trials beyond the subset that were both originally successful and region-correct). VoxPoser shows the opposite shape: it established forearm target-region contact on 95.0% of runs, matching the state machine, while original-evaluator success on that same forearm set was only 33.3%. Its dominant failure was insufficient progress, 68 of 101 region-failed trials, alongside high motion activity — persistent movement that never converted into task progress. A zero-shot π0.5 DROID checkpoint recorded 1 of 140 original successes and zero correct-region or gated successes, with failures almost entirely no-contact (106) or off-target (30).
Read the mechanism, not just the table. A value map that optimizes a safety-shaped objective can settle into a regime that is safe and ineffective; the authors suggest that once contact is established, a later phase driven by an explicit progress signal — a target sliding distance for scrub, a target indentation for push — may be more appropriate than continuing to shape the trajectory toward low cost. That is their proposed direction, not a tested result. Their caveats matter too: unload recovery and energy dissipation are excluded from the real-to-simulation calibration claim, and the upper-arm simulation reuses forearm calibration parameters without independent region-specific validation. Within those bounds, the lesson stands: reaching the site, finishing the action, making progress, and staying below force are empirically separate properties, and if your domain has physical validity criteria, they need their own column.
The judge is an instrument, and it has an error rate
Both papers above rely on an evaluator to decide what counted. The third paper audits that layer directly. FailBench assembles fourteen sources — twelve real, two simulated — screened from roughly thirty candidates. Each sample is one robot attempt carrying the instruction, the visual record, and one binary success label. Crucially, the labels come from the original recorders; the benchmark does not relabel videos itself. Some sources were built for failure detection; others were policy evaluations, reward-model scoring sets, or ordinary teleoperation collections, grouped by how the failure arose: organic during a rollout, planned by a human deliberately making the mistake, or synthetic edits applied after recording. macro balanced accuracy
Thirteen detectors were tested: six general-purpose VLMs, two embodied-work models, five purpose-built failure detectors. The best of them, Gemini 3 Flash, reached 0.77 macro balanced accuracy, against a chance floor of 0.50. The hardest real subset, contact-rich assembly on a NIST task board, averaged 0.52 across detectors, with its best scoring only 0.60. That is near-chance on precisely the kind of precise, small-object manipulation where you would most want an automated verdict. Every purpose-built detector scored below every general-purpose model except the smallest, and two specialists answered “failure” almost regardless of input, which earns a perfect score on a failure-only slice and chance performance wherever classes are balanced.
The simulation comparison sharpens the deployment consequence. Real and simulated halves produced very similar detector rankings (Spearman ρ = 0.95), but the simulated slices were about 1.3 balanced-accuracy points easier on average, with visibly less dispersion across detectors. Choosing a judge on simulated slices can therefore hide gaps that appear on real footage.
Keep the scope tight. FailBench scores one retrospective binary outcome per completed attempt, from vision plus the instruction. It does not test whether a detector can spot a bad plan, anticipate a failure, name the failure type, or estimate progress. Four sources record force-torque or audio that the benchmark does not use. Every embodiment is a parallel-jaw tabletop arm; humanoids, mobile manipulators and dexterous hands are untested. Within that scope, the interpretation I would carry forward is that an automated success label is a measurement with a known-nonzero error rate, and before its labels summarize policy performance, it should be validated on real attempts and media conditions resembling yours — same camera count, same task family, same failure sources.
What to do with three separate questions
Each study suggests a separate question for a deployment review. None tests the complete evaluation process.
First, was recovery evaluated at all, or only uninterrupted rollouts? If mistakes in your setting are recoverable rather than terminal, the LIBERO-Recover ladder gives a shape for that test — and its results warn that ranking on clean episodes may not survive the transfer.
Second, does your domain have physical criteria that completion cannot express? In contact-rich care, force, contact site and sustained progress each caught something the others missed. The specific gates are conservative and setting-specific; the practice of separating them is transferable, the thresholds are not.
Third, who assigns the label? If it is a VLM, its error rate belongs in your reporting alongside the policy’s score, measured on data that looks like yours. That error rate is not a ready-made threshold for what counts as a real gain — how finely a judge can separate two policies also depends on how its mistakes are structured and how many attempts you compare — but reporting it is what lets a reader see how much of the score rests on the instrument.
None of this makes task success useless. It makes it one column instead of the whole table — and each of these three papers tells you which other column it cannot substitute for.