CoRT redistributes Group Relative Policy Optimization (GRPO)'s single response-level advantage across tokens by rescoring the same response with the rubric criteria removed. Tokens whose likelihood drops more get more credit, lifting instruction-following accuracy by ~4.4 pp on average over matched response-level GRPO.
You’re training an instruction-following model with RL against a rubric verifier that checks things like “response must be in English, exactly three bullets, ends with a specific phrase, wrapped in quotes.” The verifier returns one score per response. Today, Group Relative Policy Optimization (GRPO) takes that score, turns it into one advantage number, and applies it identically to every token, the filler prose and the load-bearing P.S. marker alike. That’s wasteful: the rubric already tells you which aspects of the response matter, but the training signal throws that structure away.
The obvious fix is to train a second model that predicts per-token relevance, which is what Rubrics-to-Tokens (RTT) does. That works but adds a whole data-generation and training pipeline for the relevance discriminator. CoRT asks whether you can get token-level credit for free, using only signals the policy already produces.
The intuition: if you rescore the same generated response but with the rubric criteria stripped out of the prompt, tokens that were “about” the criteria (the quotation mark that satisfies the wrapping constraint, the * that starts a required bullet) will see their log-probability drop, while generic content tokens barely move. That per-token drop is a free proxy for “how much did this token depend on the rubric?”
Concretely, for each sampled response, CoRT does one extra forward pass under a criteria-free version of the prompt, computes the per-token log-prob difference (call it Δ), and squashes Δ through a sigmoid to get a bounded score. Those scores are turned into token weights that average to 1 across the response. So the response-level advantage from GRPO is preserved on average; it’s just reshuffled toward rubric-dependent tokens. Two stability tricks matter: response-mean normalization (keeps the average multiplier at 1, so updates stay on GRPO’s scale) and a SmoothStep schedule ramp that starts at 0 and ramps to full strength over 100 steps, so the early training looks like plain GRPO while reward statistics stabilize.
for group in batches:
responses, rewards = rollout(policy, prompt_with_rubric)
A = (rewards - rewards.mean()) / (rewards.std() + eps) # GRPO advantage
logp_full = policy.score(responses, prompt_with_rubric)
logp_bare = policy.score(responses, prompt_without_rubric) # replay
delta = logp_full - logp_bare
s = sigmoid(tau * (delta - b)) - 0.5
w_tilde = 1 + eta * lambda_k * s # lambda_k from SmoothStep
w = w_tilde / w_tilde.mean(axis=response) # normalize per response
A_token = stop_gradient(w) * A # per-token shaped advantage
loss = clipped_ppo_surrogate(policy, responses, A_token)
The replay weights are wrapped in stop_gradient, so they act purely as coefficients. The sign of the update still comes from the original GRPO advantage: good responses reinforce their rubric-dependent tokens, bad responses suppress theirs more strongly.
The common instinct for finer-grained credit is to train a second model, like Rubrics-to-Tokens (RTT)'s learned token relevance discriminator, that scores each token’s importance. This paper shows the opposite. The policy already knows which of its own tokens depend on the rubric; you just need to ask it by rescoring the same response with the rubric removed. The strongest evidence isn’t the headline benchmark bump but the case-study controls showing that replay contrast localizes to the exact tokens realizing each criterion (the * for bullets, P.S. for the postscript), and disappears when the removed criterion has no local trace in the response.
The load-bearing evidence is the replay case study, not the leaderboard. When they remove one rubric criterion at a time and re-score the same fixed response, the log-prob drop concentrates on exactly the tokens that realize that criterion: removing the bullet requirement lights up * markers, removing the highlight requirement lights up emphasis markers. Removing global criteria like “respond in English” produces diffuse changes with no concentrated cue tokens. That’s what makes the replay contrast a credible per-token relevance signal rather than a length-of-prompt artifact.
On top of that mechanism, the aggregate result: CoRT beats matched response-level Group Relative Policy Optimization (GRPO) in the vast majority of the head-to-head cells across two Qwen instruction-tuned models, two reward modes (Constraint Satisfaction Rate (CSR) and All-or-Nothing (AON)), and four instruction-following benchmarks (IFEval, IFBench, MultiDimIF, AdvancedIF), with an average gain of 4.4 pp. It matches or beats Rubrics-to-Tokens (RTT) without needing a separate relevance model. It scales to Qwen3-14B (with one exception where it trails GRPO on IFEval under AON). And it composes with other GRPO variants: adding CoRT on top of DAPO improves all metrics, and on top of GSPO improves four of five.
The stability ablations matter for practitioners: dropping response normalization causes the mean token weight to drift above 1, which then leaks into length clipping and gradient-norm spikes. Dropping the SmoothStep ramp causes late-training gradient and entropy spikes. Both controls are necessary, and they fix different failure modes.
Reach for this when you’re training an instruction-follower with rubric-style rewards (a verifier that returns a checklist score or all-or-nothing pass) and you’re on GRPO or a close relative. The change is small: one extra forward pass per rollout under a version of the prompt with the rubric criteria deleted, then a bounded reweighting of the existing advantage. You keep the same verifier, the same reward, the same clipped surrogate, and the same rollout distribution. No auxiliary model, no new labeling stage. It also plugs into DAPO and GSPO without changes.
The paper doesn’t mention a public code release, so you’d be reimplementing from the algorithm box and hyperparameters. The training data is HIR-16k, which is convenient because each example already ships with a prompt-minus-instruction-list version, giving you the criteria-free prompt for free. If your data doesn’t have that split, you’d need to construct it, either by templating rubrics separately from prompts or by having a rubric-generation step you can toggle off.
When your reward is structured but your update is uniform, ask the policy itself where the structure lives. A single counterfactual forward pass over the same tokens, with the rubric stripped from the prompt, surfaces per-token relevance that would otherwise require a whole second model. The signal is only as good as the criteria’s local footprint in the response, so this helps most when rubrics dictate visible spans, formatting, or keywords, and less when they’re diffuse constraints like tone or global length.
•
The replay signal degrades when criteria have no local trace in the response (global constraints like “respond in English” or minimum sentence counts show diffuse, weak contrast in the case studies). If your rubric is mostly such global constraints, CoRT may reduce to near-GRPO.
•
Gains average 4.4 pp but aren’t uniform: at 14B under All-or-Nothing (AON), CoRT trails GRPO on IFEval. Sparse all-or-nothing rewards seem to interact unpredictably with token reweighting, and the paper doesn’t fully diagnose why.
•
Every benchmark is instruction-following on Qwen models with an HIR-16k-style rubric structure. Whether the counterfactual-replay signal transfers to reasoning tasks, code, or multi-turn agents, where the “remove the criteria” intervention is less clean, is untested. The paper flags criterion-specific allocation and multi-turn credit as future work rather than demonstrated capabilities.