Learning from reward signals through trial and interaction with an environment. Spans classic RL, RLHF, and modern post-training methods for language and agent models.
Reinforcement learning after pretraining: rewards, credit, and reliable feedbackModern reinforcement learning for language and agent models often starts with a simple signal: did the completed attempt succeed? That works when an answer, program, or action sequence can be checked, but it gives little guidance about which intermediate decisions helped or hurt. Credit assignment turns that trajectory-level result into more local training signals.The right mechanism depends on the available feedback. Verifiable reasoning can use the first mistake, changes between policy checkpoints, or even a weak teacher’s policy direction. Long-horizon agents can redistribute rubric judgments or distill hindsight skills from their own trajectories. Visual systems need rewards aligned to local actions and temporal quality. In every setting, judge reliability matters: scalable feedback that accepts failures can optimize the wrong behavior.
Cliff: Learning Process Rewards from the First MistakeStart here for an implementation-oriented account of first-mistake credit assignment in RLVR, including teacher filtering, prefix/suffix advantages, isolating ablations, and length-hacking behavior.Tested on math and code with automatic verifiers; open-ended and multi-turn agent tasks were not evaluated.
What are you trying to solve?
Extract better updates from sparse verifier rewardsA pass/fail verifier treats a nearly correct derivation like an unproductive one. Cliff localizes the first invalid step; RISE extrapolates movement between policy checkpoints into dense token targets; OPRD uses a teacher’s policy shift as a direction for verifier-supported updates rather than as an output target. These are different ways to enrich a sparse outcome signal without treating one teacher as an unquestioned destination.
Cliff: Learning Process Rewards from the First MistakeStart here for an implementation-oriented account of first-mistake credit assignment in RLVR, including teacher filtering, prefix/suffix advantages, isolating ablations, and length-hacking behavior.Tested on math and code with automatic verifiers; open-ended and multi-turn agent tasks were not evaluated.
RISE: Recursive Improvement via Self-Extrapolating Policy DistillationRead for a contrasting route to dense supervision: RISE builds a refreshed synthetic teacher from the model’s own RLVR checkpoint trajectory instead of using an external teacher.The abstract reports results across reasoning, STEM, code, and multi-turn agent tasks but omits implementation details.
Eliciting Weak-to-Strong Generalization with On-Policy Reverse DistillationRead to understand how a weak teacher can guide rather than cap a student: OPRD amplifies verifier-supported updates that align with the teacher’s policy shift.The abstract covers language-model transfer across successive models, multiple teachers, and strong-to-weak distillation.
How can final-answer rewards guide intermediate reasoning steps?Cliff splits each failed rollout at its first mistake. In the tested λ=0 setup, the valid prefix receives no positive advantage but avoids the suffix’s stronger negative signal. RISE instead uses the displacement between a current checkpoint and a trailing anchor to create a refreshed token-level teacher, then combines that dense target with verifier-grounded RL.Cliff: Learning Process Rewards from the First MistakeRISE: Recursive Improvement via Self-Extrapolating Policy Distillation
When should I use an external judge instead of a self-extrapolated teacher?Choose first-mistake judging when outputs are automatically verifiable and a competent LLM can localize errors without matching the student’s tokenizer or style. Consider self-extrapolation when successive RLVR checkpoints are available and teacher mismatch is the larger concern. RISE reports results in reasoning, STEM, code, and multi-turn agent tasks.Cliff: Learning Process Rewards from the First MistakeRISE: Recursive Improvement via Self-Extrapolating Policy Distillation
Can a weaker teacher accelerate a stronger model without becoming its performance ceiling?OPRD measures the teacher’s policy shift relative to its reference policy on student rollouts. It amplifies only the component of the student’s verifier-driven gradient that aligns with that direction, rather than training the student to reproduce teacher outputs. The paper reports faster learning in successive-model, multi-teacher, and strong-to-weak transfer settings.Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Assign credit across long agent trajectoriesLong tasks contain useful, redundant, and harmful actions, yet one terminal score pushes on all of them together. DRACO redistributes dynamic rubric judgments to responsible steps. OPID extracts episode- and step-level hindsight skills from on-policy trajectories and converts their effect into token-level supervision. OSReward asks whether a vision-language trajectory judge is reliable enough to supply feedback in the first place.
OPID: On-Policy Skill Distillation for Agentic Reinforcement LearningRead for distribution-matched hindsight supervision: OPID extracts hierarchical skills from completed on-policy trajectories and combines a token-level distillation advantage with outcome RL.Reported experiments cover ALFWorld, WebShop, and search-based QA; implementation details here are limited to the abstract.
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward ModelsRead before treating a vision-language trajectory judge as ground truth. It contributes computer-use judge benchmarks, identifies leniency bias, and presents lower-cost trained reward models.Evidence concerns cross-platform computer-use trajectories and the OSReward benchmark suites.
When are hindsight skills useful for agent training?OPID fits tasks where completed trajectories contain reusable workflow or failure-avoidance knowledge and supervision should follow the current policy’s state distribution. It routes critical decisions to step-level skills and otherwise uses episode-level guidance, avoiding dependence on a separately maintained external skill memory.OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning
Why evaluate a trajectory judge separately from the policy it trains?OSReward finds that vision-language judges for computer-use trajectories can be systematically lenient, labeling failed runs as successful. Those errors can corrupt evaluation, data selection, and RL rewards. Its ordinary, hard, and fine-grained benchmark suites make judge reliability a measurable component rather than an assumption.OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Align video rewards with local actions and visual qualityInteractive-video rewards must capture whether a commanded camera action occurred and whether appearance and motion remain coherent. Whole-video judgments can bury brief action evidence in long context. WorldReward compares action-aligned chunks, forms separate action and visual-quality preferences, and aggregates local decisions into video-level feedback.
WorldReward: Reward Modeling for Camera-Conditioned World ModelsRead for domain-specific reward construction in camera-conditioned world models: local action alignment, separate quality criteria, and aggregation of chunk-level preferences.Claims concern the reported interactive-video preference benchmark and RL post-training of HY-WorldPlay 1.5.
Why score action-aligned video chunks instead of the full sequence at once?A short-lived camera response may be diluted when a judge receives an entire video and action sequence. Chunking aligns visual evidence with the relevant command, while separate preferences avoid collapsing action execution, appearance, and motion into one opaque judgment. The resulting reward can support preference evaluation and RL post-training.WorldReward: Reward Modeling for Camera-Conditioned World Models
Track what becomes autonomous as supervision recedesScaling RL is not only a policy-optimization problem. It also changes who creates rewards, tasks, curricula, and environments. The proposed L0–L4 framework separates a reward axis from an experience axis and pairs increasing autonomy with risks such as reward hacking, feedback drift, curriculum collapse, and environment errors.
What should be measured when models increasingly generate their own feedback and experience?Evaluate three objects separately: policy capability, feedback fidelity, and experience quality. A stronger task score alone cannot reveal whether rewards have drifted or a self-generated curriculum has narrowed. The framework helps locate remaining human control and identify which failure mode an autonomous training loop must monitor.Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence