DRACO trains long-horizon tool-using agents without a task verifier by generating a fresh rubric per rollout, scoring it once, and splitting that trajectory-level judgment across the steps the judge cites, so step-level credit lands on the responsible turns without any learned attribution module.
You’ve shipped a customer-support agent that makes 20 API calls to resolve a refund. You want to train it with reinforcement learning, but there’s no unit test for “handled the refund correctly.” The usual escape is RL with verifiable rewards, which needs a programmatic checker you don’t have. The next fallback, a rubric scored once by an LLM judge, gives you one number for the whole 20-step episode and pushes every token by the same amount. That’s the setting this paper attacks: process-only reward, long trajectories, no gold answer at any point during training.
The method has two moving parts sitting on top of Group Relative Policy Optimization (GRPO).
First, rubrics are written per task and per rollout instead of once for the whole benchmark. For each task, a frozen judge (GPT-5.4) proposes evaluation criteria from the instruction, then extends them by looking at each sampled rollout to catch the failure modes that rollout actually exhibits. Criteria from all rollouts in the group are merged and deduplicated. A discriminative dropout step then throws away any criterion that every rollout passed, because a criterion nobody fails carries no learning signal after group normalization. The judge scores each surviving criterion pass/fail/not-applicable, and the trajectory reward is (passes − fails) / (passes + fails).
Second, that single scalar is redistributed across steps. When the judge scores each criterion it also cites which agent turns are responsible. For step j, count the passing citations p_j and failing citations f_j, and define step quality Q_j = p_j / (p_j + f_j). On a winning trajectory (positive advantage), the step weight is Q_j, so good steps get reinforced more. On a losing trajectory, the weight is 1 − Q_j, so bad steps get suppressed more. The per-step advantage is then scaled so that the sum over all tokens exactly equals what baseline GRPO would have applied. Nothing is trained. It’s a closed-form reweighting.
for task in batch:
rollouts = policy.sample(task, G=6)
criteria = judge.propose(task)
for r in rollouts: criteria += judge.extend(task, r)
criteria = drop_if_all_pass(dedupe(criteria))
for r in rollouts:
verdicts, citations = judge.score(r, criteria)
R[r] = (passes - fails) / (passes + fails)
A = (R - R.mean()) / R.std() # GRPO
for r, step in rollouts_and_steps:
Q = passes_citing(step) / total_citing(step)
w = Q if A[r] >= 0 else 1 - Q
a[step] = A[r] * N * w / (n_step * sum_w)
The default assumption when you have no verifier is that you must choose between a cheap trajectory-level rubric (one scalar for 20 steps) or an expensive per-step judge call. This paper shows the middle path works. Score the rubric once, but make the judge cite which steps it’s grading, and redistribute the single scalar in closed form using those citations. The load-bearing evidence is the interaction: step credit is worth almost nothing on a static rubric (+0.8 TGC) but jumps to +3.2 when paired with per-trajectory rubrics, because the attribution has nothing to attribute unless the criteria are specific enough to implicate specific steps.
The ablation that proves the mechanism: on AppWorld, adding step credit to a fixed task-distribution rubric buys almost nothing, but the combination of per-trajectory rubrics and step credit is worth +4.2 TGC and +10.7 SGC over the fixed-rubric baseline, growing to +8.1 and +14.3 at the strictest consistency level. On the harder test-challenge split, step credit on a fixed rubric actively hurts by 3.7 points, while on per-trajectory rubrics it helps by 1.4. The two components are complements, not additives.
Secondary results as evidence:
•
On Qwen3.6-27B, DRACO lifts AppWorld test-normal from 69.4 → 85.3 TGC and 41.1 → 70.6 SGC, beating the same GRPO run trained on ground-truth unit tests by +5.3 TGC and +11.3 SGC despite using no verifier.
•
Zero-shot transfer to \u03c4-bench banking: 15.8 → 20.4 success rate.
•
Replacing the frontier GPT-5.4 judge with the policy model judging itself (with a 3-of-3 unanimity rule) still exceeds the verifier-trained baseline, at 5.1× lower judge cost.
•
Trained rollouts are shorter (18.7 → 14.7 turns on test-normal), so accuracy isn’t bought with verbosity.
Reach for this when you’re training an agent whose success you can describe in natural-language criteria but can’t check programmatically: customer-support flows, research assistants, multi-step form-filling. The recipe: use a strong frozen judge to generate a fresh rubric per rollout, drop the criteria nobody in the group fails, ask the judge to cite responsible steps when scoring, and plug the closed-form step-advantage formula into your existing GRPO trainer. The self-judge variant matters if judge cost dominates: it uses the policy itself with a 3-of-3 pass rule (to catch the lenient-false-pass failure mode the paper measured directly) and still works.
The code is released. Training uses LoRA adapters on 8 H100s; the paper uses Qwen3.6-27B and Qwen2.5-32B-Instruct as base policies. AppWorld and τ-bench are both released benchmarks the authors use under their existing licenses.
When you can’t verify outcomes, make the judge point at which steps it’s grading, and let the citations do the credit assignment for free. Trained attribution modules and per-position judge calls are both avoidable if your rubric is trajectory-specific enough to name names.
•
The paper cannot show credit lands on the right steps, only that end-task performance improves. A wrong attribution that still moves the policy in a useful direction would look identical in these results.
•
Discriminative dropout makes the rubric set a function of which rollouts you happened to sample, and the authors don’t measure the resulting training-time variance. Reported error bars are inference-time repeats of fixed checkpoints.
•
The self-judge saving depends on the base model being strong enough to grade itself. On the weaker Qwen2.5-32B-Instruct the self-judge isn’t reported, and the frontier-judge run there closes only most of the gap to a verifier-trained baseline, not all of it.
•
No human validation of whether the auto-generated criteria actually describe task success; a systematically-biased judge would still produce internally-consistent training and the experiments couldn’t distinguish that from a good one.