ActionOPD trains a smaller agent to copy a bigger agent’s decisions rather than its exact tokens, by rewarding the student when its sampled tool call or message semantically matches one sampled from a black-box teacher at the same state.
Say you want a cheap 4B- or 8B-parameter model to behave like a frontier model inside an agent loop: pick the right tool, pass the right arguments, say the right thing to the user. Two standard shrink-the-model recipes exist, and both have known problems.
•
Offline distillation (also called sequence-level Knowledge Distillation): record the teacher’s full trajectories, then fine-tune the student to reproduce those tokens. Problem: the student at inference time wanders into states the teacher never visited, so the training data doesn’t cover what actually happens in deployment. This is the classic State-distribution mismatch issue.
•
Token-level On-Policy Distillation: let the student roll out, then at every token the student emits, pull the teacher’s probability distribution over the next token and push the student toward it. This fixes the state-mismatch problem but requires two uncomfortable things: access to the teacher’s per-token probabilities, and a shared tokenizer. If your teacher is a hosted frontier model from another vendor, you have neither.
ActionOPD’s observation: in an agent setting, what you actually care about is the decision, that is, which tool gets called with which arguments, or what the message to the user communicates. A tool call get_weather(city="Paris") is the same decision whether the model wrote three paragraphs of reasoning first or none, and whether the JSON keys came out in one order or another. So match at the decision level, not the token level.
The setup: at each step of an agent trajectory, the student is at some state (task, tools, history). ActionOPD samples N=4 candidate responses from the student and K=3 responses from the teacher at that same state. A semantic matcher then checks each student candidate against each teacher sample: do they call the same tool with equivalent arguments, or convey the same message intent? Matches earn reward; invalid outputs (malformed tool calls) get a penalty of -1; valid-but-no-match gets zero. A small frequency bonus (controlled by \u03bb=0.5) gives slightly more credit for matching a teacher action that showed up multiple times, so if the teacher sampled [A, A, B], matching A beats matching B by a bit.
Those rewards feed a REINFORCE-style policy gradient, implemented with Group Relative Policy Optimization (GRPO)-flavored group-relative advantages and PPO clipping. Crucially, the reward is one scalar per response, but it weights every token the student generated, including the reasoning tokens. So the student learns its own reasoning style, not the teacher’s prose.
One twist worth knowing: the student actually commits (executes in the environment) the highest-reward candidate of its four, not a random one. This is called state-group rollout. It biases the trajectory toward teacher-approved states, giving the training distribution a nudge toward places where the teacher’s guidance is informative. All four candidates still contribute to the gradient update.
for step in range(num_steps):
for task in sampled_tasks:
state = env.reset()
while not done:
teacher_samples = sample(teacher, state, K=3)
student_cands = sample(student, state, N=4)
rewards = [reward_lambda(c, teacher_samples) for c in student_cands]
commit = student_cands[argmax(rewards)]
state, done = env.step(parse_action(commit))
buffer.add(state, student_cands, rewards)
update_student(buffer) # group-relative advantages + PPO clip
Because the teacher is only queried by generation (sample a response), the teacher can be any black-box API, with any tokenizer, from any model family.
The main cross-benchmark result: training Qwen3-4B and Qwen3-8B students against two different teachers (Qwen3.7-Flash and DeepSeek-V4-Flash) on curated tool-use tasks from Agent World Model and EnvScaler, then evaluating on four held-out agent benchmarks (Berkeley Function Calling Leaderboard, ToolSandbox, AppWorld, tau2-bench), all four student+teacher configurations improved the four-benchmark average by 3.3 to 6.9 percentage points over the base student. The bigger student and the stronger teacher each independently helped. Notably, the stronger teacher helped even when it came from a different model family than the student, which is exactly the regime token-level OPD can’t handle.
On \u03c4\u00b2-bench specifically, with a Qwen3-32B teacher where token-level OPD is possible, ActionOPD was competitive. Offline KD got the best pass^4 (four-trial all-success reliability on a narrow task set), while ActionOPD and OPD got higher pass^1 (single-trial success) on test tasks. The authors read this as on-policy methods generalizing more broadly, offline KD overfitting to a narrow set of demonstrated trajectories.
The most informative ablation is Validity Reward: replace the semantic-match reward with a reward that just checks “is this a well-formed action?” Performance drops the most of any ablation. That’s the key evidence the semantic matching is doing real work rather than merely filtering out malformed outputs. A Binary Reward ablation (drop the frequency bonus) and a Random Commit ablation (don’t pick the max-reward candidate) also underperform the full method, though by less.
Separately, a Self-ActionOPD variant where teacher and student share weights but the teacher gets extra privileged context (private user goals, reference actions) also beats the base model on \u03c4\u00b2-bench after 50 steps, suggesting an avenue for self-improvement without an external teacher.
•
If you’re distilling an agent from a hosted frontier model you can only call via API, the usual token-level OPD recipe doesn’t apply. ActionOPD is a plausible fit because it needs only generation access. The pattern worth testing is: at each student state, sample a few teacher completions, match them semantically against student candidates, and run a REINFORCE-style update. Code and checkpoints are promised at thu-nics/ActionOPD.
•
The semantic matcher is doing a lot of load-bearing work and is itself an LLM for the fuzzy cases (natural-language messages, non-canonical argument differences). If your action space has a clean programmatic equality check (tool name plus canonicalized args), you can lean more on that and reduce matcher cost and noise. If your action space is mostly open-ended natural language, matcher errors become a bigger source of mislabeled reward.
•
Don’t read “ActionOPD beats OPD” as universal. On the \u03c4\u00b2 same-family comparison, ActionOPD and token-level OPD trade wins depending on the metric. The real argument for ActionOPD is interface flexibility (black-box teacher, cross-tokenizer, cross-family), not uniform superiority.
•
The gains are measured as benchmark averages on out-of-distribution agent tasks after 100 training steps. That is evidence of transfer, not of reliability in deployment. For a given application, re-measure the metric you actually care about (e.g., pass^4 if you need repeat reliability, not single-trial success).
•
Semantic agreement is not correctness. If the teacher makes a mistake, ActionOPD will reward the student for making the same mistake. There is no environment reward or verifier in the loop.
•
The matcher is an LLM for ambiguous cases and can produce false matches or missed matches, especially on natural-language replies where “same intent” is a judgment call. This directly mislabels training reward.
•
With only K=3 teacher samples and N=4 student candidates, in states where plausible good actions are highly dispersed (long-horizon planning, open-ended dialog), many groups will produce no matches at all and get dropped from training. The authors flag that scaling K and N could help but didn’t test it, so the method’s behavior in that regime is unknown.
•
The reported scaling trend (bigger students help, stronger teachers help) is based on two student sizes and two teachers. It is suggestive, not a scaling law.