Agentic Abstention asks whether tool-using agents stop acting when a task is infeasible, and shows they mostly can’t stop in time. A lightweight playbook method, convolve, lifts timely abstention on WebShop from 26.7% to 57.4% on Llama-3.3-70B with only 20 training trajectories.
You’ve shipped a shopping-assistant agent that searches a catalog, clicks through results, and buys on the user’s behalf. A user asks for “a red shirt that looks elegant.” There are no red shirts in stock, and “elegant” isn’t something your product filter encodes. Today your agent burns 10 tool calls, then either buys something wrong or times out. What you actually want is: recognize infeasibility, stop, tell the user.
Prior work on LLM abstention studied this as a single-turn answer-or-refuse decision in QA. That framing doesn’t fit agents, where infeasibility often only surfaces after the agent has poked at the environment. This paper is the first to define and measure abstention as a sequential decision inside a tool-use loop.
The authors formalize abstention as a three-way choice at every turn: ANSWER (commit), ABSTAIN (stop, optionally ask user to clarify), or ACT (call another tool). They frame it as a POMDP where the ground truth about feasibility is hidden and only revealed through interaction.
They build a 28k-task benchmark across three settings: web shopping (adapted from WebShop), terminal tasks (adapted from Terminal-Bench), and QA (adapted from AbstentionBench). Two flavors of unsolvable task matter: request-based, where the instruction itself is broken (contradictory, subjective, underspecified), and environment-based, where the instruction looks fine but the environment can’t satisfy it (the target item was removed from the catalog, a required file is missing).
The key evaluation move is separating timely recall ([[absrec-at-k|AbsRec@1]] for request-based, AbsRec@2 for environment-based) from overall recall (AbsRec@10). Overall recall asks “did the agent eventually give up?” Timely recall asks “did the agent give up as soon as it had enough evidence?” The gap between the two is the real story.
Their intervention, convolve, is context engineering, not fine-tuning. After each training rollout, a reflection model reads the full trajectory and a curator model writes short “stopping rules” into a structured playbook. The playbook gets prepended to the system prompt on future episodes.
playbook = ""
for episode in training_tasks[:20]:
trajectory = run_agent(task=episode, context=playbook)
reflection = reflect_model(trajectory) # what signals were missed?
playbook = curator_model(playbook, reflection) # structured add-ops
# at inference:
run_agent(task=new_task, context=playbook) # no weight updates
The prevailing assumption when agents misbehave is that they need a stronger base model or more reasoning tokens. This paper shows the opposite. Scale and reasoning improve eventual abstention but barely move timely abstention; the bottleneck is that agents don’t recognize the earliest turn at which evidence is sufficient to stop. The evidence is that larger Qwen models raise AbsRec@10 but leave AbsRec@1 nearly flat, and that a 20-example playbook beats scaling.
The load-bearing finding is the timing gap. Across 13 models and 2 scaffolds, every system scored under 40% average timely recall, even when overall recall was much higher. Agents eventually notice infeasibility, but only after wasting turns.
Secondary results that sharpen the picture:
•
Scale doesn’t fix timing. Larger Qwen-3 models (8B → 235B) improve AbsRec@10 but AbsRec@1 barely moves.
•
Reasoning trades off. Qwen-3-235B-Thinking improves timely recall on WebShop but lowers overall recall. More thinking helps you stop at the right moment or not at all.
•
Scaffold matters as much as model. On terminal tasks with GPT-5.4-mini fixed, Codex CLI reaches ~38% AbsRec@10 while Terminus-2 reaches ~18%.
•
Environment-based abstention is the hardest. “Missing Target” in WebShop and “Underspecified Intent” in terminal are where timely recall collapses.
•
Over-abstention is real. On solvable WebShop tasks, Qwen3-235B-Instruct wrongly abstains 34% of the time by turn 10. Longer horizons make agents jumpy.
•
convolve with 20 trajectories raises Llama-3.3-70B on WebShop from 26.7% → 57.4% timely recall and 83.2% → 100% overall recall, without touching weights. Lessons learned by an 8B model also transfer up to 70B and recover most of the gain.
Reach for this framing when you’re shipping any multi-turn agent behind a tool budget: code assistants that run tests, shopping bots, browser agents, terminal agents. Instead of only measuring task success, log the earliest turn at which the environment gave enough signal to stop, and measure how many turns your agent burned past that point. That single metric will tell you whether your agent is thoughtful or just stubborn. convolve’s recipe is directly reusable: run 20 rollouts, have a reflection model write stopping rules keyed to the failure patterns you saw, and paste those rules into the system prompt.
Code, data, and the constructed WebShop and Terminal-Bench abstention variants are at the project page. The QA slice is a filtered adaptation of AbstentionBench with a Wikipedia retrieval setup capped at 10 search calls.
Stronger models learn to eventually give up; they don’t learn to give up on time. Timing is a separate skill from capability, and a 20-example prompt playbook closes more of the gap than scaling parameters does. If your agent is expensive per tool call, measure the turn it should have stopped, not just whether it stopped.
•
The benchmark’s environment-based abstention only covers “missing target” and “missing prerequisite.” Real deployments fail in messier ways: stale APIs, permission boundaries, conflicting tool outputs. Playbooks tuned on these clean failure modes may not transfer.
•
The convolve headline number (26.7 → 57.4) is on a 101-example held-out slice of WebShop, with the playbook trained on 20 examples from the same distribution. In-distribution playbooks are the easy case. The AbstentionBench and TerminalBench transfer numbers exist in the appendix but are more modest.
•
“Abstain” is scored as correct whether the agent refuses outright or asks the user to clarify. If your product needs to distinguish those (clarify is often better UX than refuse), you’ll need finer-grained labels than this benchmark provides.