TL;DR
Interlock generates agent training tasks by encoding each task as a graph of evidence, decisions, and required downstream effects, then fine-tunes agents only on rollouts that satisfy every effect. The result transfers to three independent benchmarks, adding roughly 3–7 points across them.
Why It Matters
Suppose you build an agent that handles payroll close. It correctly computes an employee’s net refund and holds the cash, but forgets to post the matching gross-cost accounting row. Every individual tool call looked right. The task is still unfinished, and a downstream auditor will catch it.
The authors call this the completion gap: being locally correct on each step while leaving dependent artifacts, schedules, or records out of sync. For long-horizon agents working across many files and turns, this is the dominant failure mode, not reasoning errors on any single step.
Prior synthesis work (surveyed in the paper, including ToolLLM, TaskCraft, and ClawGym) generates tasks by composing tool calls, extending workflows, or scaling environments. Those approaches organize tasks around tool connectivity. They do not explicitly track which decisions must flow through to which downstream artifacts, so training trajectories can themselves exhibit the completion gap.
How It Works
Each synthesized task is defined by a typed completion graph with three edge types:
•
Grounding: which pieces of public evidence (records, policies) justify a given decision.
•
Coupling: joint constraints that must hold across several decisions (a budget cap, a shared schedule window). These are hyperedges because one condition can tie together more than two components.
•
Realization: for each decision, every downstream record or artifact that must reflect it.
Three specialized generator agents then compile the same graph into three artifacts, keeping each other honest:
1.
A reset-able initial workspace (files, records, policies) plus a private evidence index.
2.
A public task request and a private deliverable contract listing required outputs.
3.
A private rubric of binary criteria tied back to graph nodes.
Two independent reviews gate the task. A source-coverage review checks that every original graph relation survives into the three artifacts. A public-input critic sees only what the solver sees and checks that the rubric is actually supported by public materials. Then a candidate rollout runs; an Agent-as-judge inspects workspace diffs, writes verification scripts, and returns per-criterion pass/fail. Tasks are admitted only if all reviews and the execution judging are clean.
Training is RFT: each backbone generates rollouts, only trajectories where every mandatory criterion passes are kept, and the backbone is fine-tuned on its own accepted trajectories.
for blueprint in blueprints:
G = build_completion_graph(blueprint)
pack = compile_task_pack(blueprint, G) # workspace, task, rubric
task = review_and_admit(pack) # source + public reviews + exec
if task: admitted.append(task)
for backbone in [qwen27b, qwen35b_a3b]:
rollouts = run(backbone, admitted, attempts=5)
accepted = [r for r in rollouts if judge(r).all_criteria_pass]
sft(backbone, accepted) # only your own accepted trajectories
What They Found
They fine-tuned two backbones, Qwen3.6-27B and Qwen3.6-35B-A3B, each on its own ~8,957 accepted trajectories (~154M supervised tokens). Evaluation held out three externally authored benchmarks: AutomationBench, JobBench, and Harvey LAB.
Against the original checkpoints, Interlock-RFT gained 4.00–4.50 pp on AutomationBench task pass rate, 3.20–6.64 points on JobBench weighted score, and 3.89–4.16 pp on Harvey LAB criterion pass rate. It also beat an SFT baseline trained on released ClawGym demonstrations across all columns for both backbones.
Two ablations are worth noting:
•
Domain breadth matters at fixed data size. Holding trajectory count at 8,957 and varying training-domain count over 16/32/64/128, every metric rose monotonically with coverage. Broader domains vary the completion requirements the model sees, not just the surface topics.
•
Agentic verification beats text-only judging. Swapping the agent-as-judge for a text-only LLM judge over the same rollout pool, with matched SFT settings, hurt all four downstream metrics. The agent judge used about 11% more output tokens (~1.25k extra) to inspect files and run checks, and that extra verification work translated into better training selection.
Interlock trajectories are also substantially longer than ClawGym’s: ~27.6 vs ~11.9 assistant turns on average, and ~42 vs ~15 tool calls, which the authors use to argue the synthesized tasks genuinely elicit sustained multi-step work.
What’s Useful
•
If you train agents on self-generated trajectories, the acceptance filter is the lever worth tuning. The paper’s evidence is that an executable, file-inspecting judge selects better trajectories than a text-only judge from the same pool. Worth testing in your own loop before adding more rollouts.
•
If you synthesize agent training tasks, treating “which downstream artifacts must reflect this decision” as a first-class relation (rather than hoping it falls out of tool chaining) is the specific design the paper validates. The dual review (source coverage plus public support) is how they prevent the generators from quietly dropping obligations.
•
For evaluation, the case studies are a reminder that aggregate pass rates hide the completion gap. If your eval only checks that the primary deliverable looks right, you can miss the missing budget row. Rubrics with explicit coupling and realization criteria surface these.
•
The paper does not release code or data in the supplied text, and the training recipe is specific to two Qwen3.6 backbones with thinking mode. Transfer to API-only models is not evaluated here.
Caveats
•
All three transfer benchmarks post-date typical pretraining cutoffs and are themselves fairly new; the paper reports they are held out from synthesis and RFT, but there’s no analysis of overlap between Interlock’s 128 synthesized domains and the benchmark domains beyond the embedding scatter plot.
•
Gains are 3–7 points, meaningful but not transformative. The ClawGym SFT baseline uses released demonstrations rather than a matched self-generated pool, so part of the delta reflects data provenance (own-model rollouts) in addition to the completion-graph design.
•
The whole pipeline depends on strong judge models (GPT-5.5 for two benchmarks; an agent-as-judge for trajectory filtering). The quality of selected trajectories is bounded by what those judges can actually verify through scripts and file inspection.
•
“Task admission” and “trajectory acceptance” are different gates in the paper. A task can be admitted even if the validation solver fails it; only the per-rollout acceptance rule governs what enters SFT. Keep that distinction when adapting the recipe.