Agent-Reflex teaches tool-using LLM agents to recover from noisy tool failures and user corrections by mining reusable environment-level reflection skills from cross-model success/failure pairs, then distilling them into the policy through supervised fine-tuning and a token-level On-Policy Distillation signal.
If you’ve built an agent that calls real tool APIs, you know the ugly middle of the trace: a 503 from a flaky endpoint, a 200 that returns the wrong record, a user who interrupts at turn 7 to say “no, I meant the other account.” Most agent-training pipelines pretend none of this happens. They train in clean, deterministic environments where tools always succeed and users never intervene, so the resulting policy has no idea how to detect a well-formed-but-wrong response, let alone repair the plan.
The paper’s framing: the authors replayed agents against nine public tool-use benchmarks and logged 24,510 real error returns out of 177,700 tool calls, then grouped them into eight failure families (timeouts, auth failures, 5xx, bad arguments, rate limits, etc.). This is the noise distribution a deployed agent actually meets.
Prior reflection work splits into two camps the authors argue both miss the point. Inference-time methods like Reflexion let the failing policy critique itself, which rarely contradicts a plausible-looking wrong answer. Training-time methods learn per-trajectory repairs, but nothing accumulates into reusable knowledge about an environment that a later, differently-failing episode can draw on.
The contribution is to treat reflection as a learnable meta-skill tied to an environment, not a per-trajectory fix. The pipeline has three stages, all built on top of qwen3-8b.
Stage 1, Reflection Skill Discovery. The authors anchor 557 real MCP (Model Context Protocol) server specifications from the Smithery registry, auto-build a database and executable tool set for each, and inject the eight mined failure families into tool returns with 20% probability. A pool of nine LLMs from 4B to 284B rolls out trajectories across these noisy environments. For each task, they pair a successful trajectory with a failed one, then an analyzer model summarizes the recurring failure pattern and corrective action into a structured skill record: a trigger, diagnosis, recovery template (retry, switch endpoint, ask user, abort), and reversibility flags. The library has 1,170 skills across 557 environments, indexed by environment rather than by task.
Stage 2, Reflective Supervised Fine-Tuning (ReSFT). For each failure, they restore the environment to the snapshot right before the first wrong turn, prepend the matched skill as a privileged hint, let Qwen3-8B continue, and keep the continuation only if the verifier says it succeeded. Three variants: Self-Reflection (resume mid-trajectory), Skill Recovery (restart the whole task), and User-Centric Reflection (a simulator rewrites the skill as a natural user correction message). Critically, the skill text is stripped from the stored input before training. The model learns the recovery behavior without needing a skill prompt at inference.
Stage 3, On-Policy Reflective Policy Optimization (ORPO). Standard agent RL only gives one scalar reward per episode. ORPO adds a dense per-token signal using on-policy distillation adapted from Seed. The idea in plain terms: roll out trajectories without the skill, then re-score the exact same action tokens with the skill prepended as a teacher context. If the skill-augmented score is higher, upweight that token in training.
for task_q in rl_pool: # EnvScaler tasks, env-aligned to skills
group = policy.rollout(q, G=8, noisy_frac=0.25)
advantages = group_normalize(verifier_rewards(group))
for tok in agent_tokens(group):
logp_plain = policy.logprob(tok, history)
logp_skill = policy.logprob(tok, skill_card + history) # teacher
delta = stop_grad(logp_skill - logp_plain)
weight = sigmoid(beta * delta) # soft, not a hard filter
loss_opd += weight * (stop_grad(logp_skill) - logp_plain)
loss = loss_rl + lambda_opd * loss_opd
The teacher is the same policy with a privileged skill card, so no second model is needed; gradients only flow through the student branch.
Trained from Qwen3-8B, Agent-Reflex is compared against reflection baselines and environment-scaling baselines at the same 7B–8B scale on five tool-use suites plus AgentNoiseBench.
•
Overall 58.6 across the five tool-use benchmarks, vs 53.6 for the strongest comparable baseline (Agent-World-8B) and 40.2 for the untrained Qwen3-8B backbone. On tau2-bench specifically, 62.1 vs 54.6.
•
On AgentNoiseBench, which injects unseen tool and user noise into two of the eval environments, Agent-Reflex reaches 29.1 overall vs 25.9 for the next-best 8B method. The authors emphasize AgentNoiseBench is out of distribution: their failure families came from a different taxonomy and injection process.
•
The ablation is where the mechanism claim lives. Removing Self-Reflection sampling causes the largest drop, then Skill Recovery, then User-Centric. Removing the OPD token-level signal while keeping the skill library still hurts clearly, and removing just the confidence weight also hurts. So the gains come from the token-level reflection signal, not from skill exposure during SFT alone.
•
A diagnostic split on AgentNoiseBench’s native axes shows the biggest wins on misleading-lure returns (well-formed but wrong, where no status code reveals the fault) and on cognitive overload on the user side. The authors read this as evidence the policy is actually repairing, not just getting lucky, since the scoring metric gates on trajectory validity.
Two honest caveats in the numbers: the RL stage reuses the EnvScaler task pool, so part of the margin reflects that pool’s coverage. And the eight hard-failure families (401, 429, 500s) show more uniform gains than the subtle semantic failures, suggesting the gains compound where the fault needs interpretation, not just a retry.
•
If you’re training an agent on executable environments, the mining recipe is the transferable idea: pair same-task successes and failures across a diverse model pool, then summarize them into environment-indexed records with explicit recovery templates. This decouples “what breaks here” from “which model happened to fail.” Worth testing whether it holds at scales other than 8B.
•
If you’re using RL on long-horizon tool tasks and your rewards are sparse, the ORPO estimator (re-score the same tokens under a privileged teacher context, gate by a confidence sigmoid) is a drop-in augmentation to Group Relative Policy Optimization (GRPO)-style training. It needs a privileged artifact per environment, but the artifact can be anything structured, not only reflection rules. The paper reuses the Seed estimator wholesale.
•
If you’re evaluating agents, the paper’s argument that whole-trajectory validity (Stability-Gated Success) is a better metric than final-answer correctness under noise is worth adopting, especially for user-interrupt scenarios where a lucky correct answer can mask a derailed plan.
•
For deployment, the stripped-skill design matters: inference needs no skill library, no retrieval, no second model call. The policy carries the recovery behavior in its weights. That makes this cheaper to serve than inference-time reflection like Reflexion.
•
Code, data, skill library, and the 8B checkpoint are at GitHub and HuggingFace.
The skill library is mined offline from a fixed snapshot. Tool APIs change, services deprecate endpoints, and the authors acknowledge they have no online update mechanism. All improvements are at 8B on a Qwen3 backbone; the mining pool spans 4B–284B but no result shows the library transfers to a different student model. The RL stage uses EnvScaler’s task pool, which overlaps at the environment level with the skill library’s environments, so part of the ORPO margin may reflect that alignment rather than reflection learning in isolation. Finally, the AgentNoiseBench “out-of-distribution” claim rests on the injectors being different, but the underlying environments (tau2-bench and VitaBench) are the same families the authors trained adjacent to.