TUA-Bench is a 120-task benchmark that forces agents to do everyday computer work (email, spreadsheets, media, expert science workflows) through a Linux terminal instead of a GUI, and the best frontier setup still tops out at 65.8% success.
You’re building an agent product and picking between a browser-automation stack that reads screenshots and clicks pixels, and a terminal-based stack that shells out commands. The terminal path is cheaper and more reliable in principle, but you have no idea how it does on the messy non-coding work your users actually want. Existing terminal evaluations like Terminal-Bench mostly grade shell and software-engineering tasks, and GUI benchmarks like OSWorld mix in visual grounding skill that has nothing to do with your agent’s planning ability. TUA-Bench is trying to be the missing evaluation for “can a CLI agent do general knowledge-worker computer use.”
The authors take 369 real desktop tasks from OSWorld (editing a spreadsheet, replying to email, fixing a slide deck) and rewrite the instructions so the agent must complete them purely via shell commands, with no fixed choice of application. They add 20 new expert workflows co-designed with PhD-level biologists, medical physicists, and engineers, built around specialized domain software. Each task ships as a Dockerfile with input files, a natural-language prompt, and an in-container verifier that checks the final environment state, all orchestrated by Harbor. To keep the benchmark from being trivially solved, they run three frontier models on every candidate task five times and keep the 100 least-solvable everyday tasks, on the theory that easy tasks won’t discriminate between models a year from now. Evaluation reports mean success across 5 trials plus Pass@k and All-5 (solved every single time), so you can tell flaky solvers from reliable ones.
for task in tua_bench: # 120 curated tasks
container = harbor.launch(task.dockerfile)
for trial in range(5):
rollout = agent.run(task.instruction, container.reset())
reward = task.verifier(container.final_state())
record(task, trial, reward)
report(success_rate, pass_at_1, pass_at_5, all_5)
The prevailing move for general computer-use agents is to have them watch a screen and click, because that mirrors how humans work. This paper argues the opposite. For LLM-based agents, a text-native command line is a more honest test of planning and tool use, because it strips away the visual-grounding tax that inflates GUI benchmark scores with skills unrelated to reasoning. The evidence is that even with visual perception removed and every task run in a clean container with a deterministic verifier, the best agent still fails a third of the time, so what’s left is genuinely about planning, execution monitoring, and recovery.
The load-bearing finding is that the ceiling is low and the failures are structural. Claude Code driving Claude Opus 4.8 at max reasoning tops the leaderboard at 65.8% success, with Codex CLI on GPT-5.5 right behind at 64.7%, and five different agent scaffolds all cluster within about 6 points of each other. Supporting evidence:
•
Reliability separates from headline accuracy. GPT-5.5 and Claude Opus 4.8 look tied on mean success (60.1 vs 59.7 on Terminus-2), but Opus solves all 5 trials on 42.5% of tasks vs 31.7% for GPT-5.5.
•
Time budget matters as much as the model. Raising the per-task wallclock from 150s to 2400s lifts success from 33.0% to 60.1% with no other change, because most short-budget failures are truncations mid-plan, not reasoning errors.
•
Reasoning effort has sharp diminishing returns. GPT-5.5 gains ~15 points going from no thinking to medium, but only 2.3 points from high to xhigh while roughly doubling output tokens.
•
Scaffold flips the model ranking. GPT-5.5 beats Opus 4.8 by 5 points under Mini-SWE-Agent, but Opus wins by 2 points under OpenHands. Which model is “better” depends on the harness.
•
Category gap. System & Software tasks are tractable across the board; Office productivity and Multimedia stay under 45% for most models, with specific slide-layout and chart tasks that no configuration solves.
Reach for this when you’re deciding whether to build your next agent product on a headless terminal stack vs a browser/desktop automation stack, and you need a realistic read on the terminal ceiling for non-coding work. The cost-vs-success Pareto chart is directly actionable: Terminus-2 with open-weight MiniMax-M3 hits ~47% at about $12 per task run, while pushing to Claude Code + Opus max costs $173 per run for 65.8%, so the frontier is roughly 15x more expensive for 18 extra points.
The benchmark, task specs, verifiers, and a live leaderboard are hosted at tuabench.ai, built on the open-source Harbor orchestrator that also underpins Terminal-Bench, so tasks are portable between the two. The paper doesn’t specify a license for the tasks themselves.
•
The everyday-task portion is a translation of OSWorld into shell form, so scores partly reflect how cleanly a GUI task can be re-expressed as file manipulation, not just terminal-agent skill. Applications without mature CLI or headless support are underrepresented by construction.
•
The expert track is only 20 tasks across four scientific domains, so per-domain conclusions are underpowered even though they anchor the “general-purpose” claim.
•
Public release means these exact tasks will likely leak into future training data. The paper flags this and promises refreshes, but any 2027+ score should be read with contamination risk in mind.