LoopArena benchmarks LLMs as runtime Controllers that instruct a fixed coding Worker (agent) round-by-round, isolating loop-control skill from coding skill. Even the strongest Controller finishes only 24.69% of full tasks, and simply re-stating the goal each round doesn’t beat running the Worker autonomously.
You’ve shipped a coding agent that grinds through multi-step feature work. Around it sits an outer loop: a manager process that reads the last round’s output, decides whether the agent should keep coding, run tests, or stop, and drafts the next prompt. When the run fails, you don’t know whether the coding model was too weak or the manager gave a bad instruction. LoopArena calls this outer role Loop Engineering and turns it into its own evaluation target. Prior coding benchmarks like SWE-bench score the whole agent stack end-to-end. This paper instead freezes the coding Worker (same model, tools, budget, evaluator) and swaps only the Controller, so the number you get back is about instruction quality, not raw coding ability.
The harness runs a two-agent loop. The Worker is a fixed coding agent using a ReAct loop with real tools; it’s the only component that touches the repo. When it hands control back, a temporary Reporter (same model as the Worker, but read-only) summarizes progress into four fields: task context, work-so-far, verification evidence, and open issues, with citations to specific Worker turns. The harness deterministically packages this into an Evidence Packet, a structured read-only view for the Controller. The Controller, which has no tools, returns a Loop Contract: a JSON decision to advance, verify, or stop, plus a bounded next instruction with goal, required outcomes, prohibited actions, and completion condition. The harness renders that Contract as the Worker’s next user message.
while not stopped and within_budget():
worker_segment = worker.run(next_instruction) # ReAct, tools
report = Reporter(model=worker_model).summarize(worker_history)
packet = format_packet(report, cited_turns, budget_left)
contract = controller(packet, controller_history) # no tools
if contract.action == "stop":
break
next_instruction = render(contract) # advance or verify
submit_workspace_to_evaluator()
Evaluation comes in three scopes. Type III runs the full repo task from its original state (sourced from SlopCodeBench (SCBench) and BeyondSWE). Type II starts from a prepared mid-task workspace and evaluates one stage. Type I is a four-way multiple choice at one control point: candidate Contracts were pre-executed twice from the same restored state during construction, and only questions where both replays agree on a unique winner are kept. Type I costs one API call per question, no Worker execution.
The usual way to compare coding agents is to score the whole stack: one model, one harness, one benchmark number. This paper argues the opposite. Pin the coding Worker and the execution setup, and treat the outer control loop as the thing being measured, because the manager’s ability to read evidence and pick the next assignment is a distinct skill from writing code. The load-bearing evidence is that a deterministic policy that just restates the original goal every round (fixed control) does not beat running the Worker with no manager at all on full tasks: state-blind persistence is not control.
•
The load-bearing result: fixed control, which mechanically restates the task goal at every handoff, scores 18.52% on Type III, exactly matching the no-control baseline of 18.52%. It helps on the shorter Type II slice (39.51% → 46.91%) but adds nothing on full tasks. Adaptive, evidence-aware guidance is what separates a real Controller from a persistent nag.
•
Full-task control is still hard. Type III Strict Success Rate across five Controllers (Qwen3.7-Plus, DeepSeek-V4-Flash-0731, GLM-5.2, GPT-5.5, Claude Opus 4.8) ranges 16.05% to 24.69%, with GPT-5.5 on top. Every Controller beats no-control, but none clears one-in-four.
•
Type II tracks Type III cheaply. Spearman’s rank correlation between the two settings under the main scoring rule is ρ = 0.9747, and Type II costs on average 64.4% less in estimated inference dollars. Under alternative SCBench scoring rules the correlation collapses (0.15 and −0.30), so the agreement is criterion-specific.
•
Type I isolates single decisions. Contract Accuracy ranges 72.22% to 87.78% with 0% invalid responses. Deterministic shortcuts (position bias, candidate length, lexical overlap) top out at 31.11%, so the questions aren’t gameable by surface cues.
•
Cost gaps are large: a Type III run with Claude Opus 4.8 or GPT-5.5 as Controller costs roughly $17–19 in estimated inference, versus $4.86 for GLM 5.2 and $2.01 for the no-control baseline.
Reach for this when you’re building or picking the outer model in a manager-worker coding setup, for example a nightly agent that iterates on a repo until CI is green. LoopArena lets you A/B different Controller models against a fixed Worker and fixed tools, so a regression in end-to-end success rate can be attributed to the manager rather than confounded with a Worker or scaffold change. Type I is the cheap smoke test for a new Controller (90 four-way questions, no code execution); Type II is the closed-loop check that still runs the Worker but on task slices; Type III is the full run.
The benchmark data and evaluation code are released at GitHub under CC BY 4.0, including 90 Type I questions, 27 paired Type II/III tasks, harness code for Worker/Reporter/Controller/reference policies, prompts, and the frozen price schedule used for cost accounting. Evaluator container assets that can’t be redistributed are referenced by source identifiers with preparation instructions.
A manager loop that just re-states the goal is not a manager. Control has to react to what the run actually did.
•
Small executable panel: 27 paired tasks across two source benchmarks, three repeats each. Rankings among close Controllers (e.g., Qwen3.7-Plus vs Claude Opus 4.8 at 23.46% vs 20.99% on Type III) are within noise you should not over-read.
•
The Type II ↔ Type III agreement (ρ = 0.9747) holds only under SCBench’s Core scoring; the all-checks and all-non-error variants give ρ = 0.15 and −0.30. Whether Type II substitutes for Type III depends on which success criterion you care about.
•
Scope is repository-level coding with a single fixed Worker model (Qwen3.7-Plus) and one Controller-Worker organization. Results may not carry to multi-Worker setups, other Worker families, or non-coding domains, and GLM 5.2 in particular hit the Controller output-token limit on ~20% of its calls, which biases its numbers downward.