TraceDance turns natural-language complaints about agent misbehavior into targeted benchmarks by cutting real Deployment traces just before a bad decision and grading what a fresh LLM does next, no environment replay needed. Nine frontier models pass only 26.7% on average.
You run an LLM agent in production. It closes tickets, but sometimes it commits a .env file with secrets, or reruns a failing job three times without reading the error log, or claims tests passed when it never ran them. Task-completion benchmarks like Terminal-Bench check whether the final container looks right, so this class of misbehavior slips through.
Today your options are: (a) wait for a new safety suite that happens to cover your exact failure, or (b) hand-audit logs. Neither scales when the misbehavior you care about is specific to your deployment (“in scheduled jobs, when a tool error names the fix, does the agent apply it?”). The paper’s premise: your production traces already contain concrete examples of the problem. Turn those into a benchmark you can run against any candidate model.
Prior auditing tools find bad behavior in traces but stop there. OpenAI has described regenerating candidate-model responses on deployment conversations before release; TraceDance is framed as the first system to make this an on-demand benchmark builder for user-specified behaviors.
The key move is called decision-point continuation. Instead of replaying the whole trace in a live environment, TraceDance finds the exact turn where the source agent misbehaved, cuts the trace right before that turn, and asks the evaluated LLM to produce the next assistant turn from the same context. A behavior-specific rubric grades that single turn. No tools execute, no environment is needed, so traces from private repos and internal MCP (Model Context Protocol) servers become usable.
Cuts come in three frames: action (what to do next), failure (how to respond to an observed error), and claim (what to say about completed work). Each frame preserves different context and asks a different evaluation question.
Building benchmarks at 250k-session scale is the second problem. Reviewing every session with an LLM is too expensive. The answer is Anchor-and-Confirm: a cheap executable “anchor” (regex-ish rules over structured trace events, running on CPUs) scans everything and flags candidates; a fast LLM (DeepSeek-V4-Flash in this paper) then confirms only the candidates. When no predefined behavior specification matches the user’s query, an Anchor Synthesis Loop has a strong model (GPT-5.6-Sol) generate anchor code, a reviewer model audit it, and revision continue until confirmation rates hit thresholds.
Grading uses a three-LLM judge panel (LLM-as-a-Judge) scoring 0-5; a response passes if the mean is at least 4.
def build_benchmark(query, traces, k_min=5, k_max=50):
spec = match_catalog(query) or synthesize_anchor_loop(query, traces)
if spec is None: return Reject(reason)
candidates = spec.anchor.scan(traces) # CPU, no LLM
instances = []
for c in candidates:
if fast_model.confirms(c, spec.criterion): # cheap LLM check
x = cut_context(c.trace, spec.frame) # hide source turn
instances.append((x, spec.rubric))
if len(instances) == k_max: break
return instances if len(instances) >= k_min else Reject(...)
def evaluate(model, x, rubric):
y_hat = model.next_turn(x) # one turn only
return judge_panel.score(x, y_hat, rubric) # pass if mean >= 4
Run on 252,557 sessions from Claude Code and OpenClaw (an internal general-tool-use harness), TraceDance built benchmarks for 95.3% of build-target queries and correctly rejected all 32 adversarial or underspecified ones, producing 107 benchmarks with 4,125 instances.
Validity checks: two human annotators independently confirmed the requested behavior actually occurred in 84% of sampled instances, and rated rubric quality at least 4/5 on 90% of them. The judge panel’s pass/fail agreement with humans (81.0%, Cohen’s kappa 0.40) was comparable to inter-annotator agreement (kappa 0.31). The panel does grade more leniently than humans on average.
The headline model result: across nine frontier LLMs, mean pass rate is only 26.7%, ranging from 22.9% (Qwen3.7-Max) to 33.5% (Claude Opus 4.8). But the informative finding is where they fail. The authors group behaviors by what the rubric requires:
•
Valid call (well-formed tool invocation): 67.9% mean pass rate
•
Check first (read the log, verify a prerequisite, confirm before destructive action): 8.1%
Worst individual families: Failure-log inspection 0.6%, Commit hygiene 0.9%, Secret protection 6.9%. These are cases where the agent might still complete the task, so Terminal-Bench-style scoring wouldn’t catch them.
Two secondary observations. First, overall rank hides behavior-specific strengths: Kimi-K3 ranks last overall but scores 57.8% on Error-guided correction versus 27.8% for Opus. Second, responses that begin with planning tools like TodoWrite pass at 4.6% versus 30.8% for other first actions on the same instances, a pattern that holds across all nine models. The authors flag this as association, not causation.
If you operate an agent in production and log traces, this is a template for turning your incident reports into regression tests. The pipeline needs structured event logs (tool calls, tool results, message roles) so anchors can scan them, and a scale where the desired behavior actually recurs. The code is at GitHub. If you’re picking a model for an agent product, the overall leaderboard is misleading for your use case. The paper shows GPT-5.6-Sol trailing DeepSeek-V4-Pro and GLM-5.2 on behavior benchmarks while leading them on task-completion benchmarks like SWE-bench Pro. Worth testing candidates on the specific failure modes that matter to you rather than trusting a headline number.
If you’re doing safety or reliability evaluation research, decision-point continuation is a cheap alternative when environment reconstruction is infeasible. The trade-off is honest and stated: you cannot observe what the model would do after the graded turn, so a response that starts with planning may recover later, and a passing turn may still fail if actually executed.
Worth testing, not established: whether these benchmarks predict deployment behavior, and whether they work as a training signal in a Recursive Self-Improvement (RSI) loop. The authors explicitly leave the loop evaluation to future work.
Pass rates describe behavior at decision points where the source agent already misbehaved, so they don’t estimate how often the bad behavior would occur in fresh deployment. Numbers depend on the passing threshold (mean judge score ≥ 4); rankings are stable across thresholds but absolute rates shift substantially.
Judges include two models that are also evaluated. The paper checks for self-preference and finds a small effect for GPT-5.6-Sol (+0.22 points on its own responses) and none for Claude Opus 4.8; removing each model’s own judge doesn’t change its rank.
The deployment traces come mostly from Doubao Seed agents and cannot be released, so exact benchmarks are not reproducible from public data. The pipeline can be applied to other trace collections, but you need your own logs. Behaviors that require reading every session (no observable event signature) are out of scope: the anchor needs something programmatic to grep for.