FDE-Gym measures whether an AI agent can deliver a working system that actually improves a customer’s business metric, scoring 0 for no change and 1 for an expert solution. The best model averages 0.56 and fully resolves only 14 of 95 tasks.
When a company hires a vendor to deploy AI, the sponsor’s written request is often not the real problem. A Forward-Deployed Engineer (FDE) has to talk to stakeholders, read the staff directory, dig through legacy code, and figure out which change will actually move a business number. Most agent benchmarks skip this entirely: SWE-bench gives you an issue plus hidden tests, \u03c4-bench gives client requirements, and clarification benchmarks hide a few fields of an otherwise correct spec. In all of those, the underlying problem is already defined.
The illustrative scenario in the paper: a VP at a drilling company asks for a daily summary email and a chat box, explicitly saying the downtime-coding system is out of scope. But that downtime-coding system is exactly what’s costing the customer money each month through disputed contractor credits. An agent that builds what the sponsor asked for ships on time, passes every check, and moves the business metric by zero.
Each of the 95 tasks drops the agent into a simulated customer: a workspace with the sponsor’s instruction, a copy of the production system, documents, historical data, and a staff directory with contact details. The agent runs Python and SQLite, cannot install packages or hit the open internet, and talks to stakeholders through a workspace command. When the agent submits, the delivered code is deployed to production unchanged and run on held-out data.
Scoring is grounded in the customer’s own KPI. The do-nothing baseline is 0, the expert reference solution is 1, and the Normalized Business Uplift measures where the agent lands between them. Three things then get subtracted or capped. If a predefined incident fires (for example, an unauthorized action), positive uplift is zeroed out but negative uplift stays. Every stakeholder message, trial run, and day of waiting incurs a Stakeholder Contact Cost in the same units as uplift. A task counts as resolved only at score ≥ 0.8 with no incident.
To isolate where agents fail, each scenario is served at four Task Instruction Levels (L0–L3): L0 states the problem plus all rules, L1 drops the rules, L2 also makes the problem vague, and L3 is the sponsor’s raw (possibly wrong) request. Environment, stakeholders, and grader stay fixed, so performance deltas isolate the information withheld. Stakeholders are scripted, not LLM-played, because prior work showed swapping the user-simulation model shifts success rates by several points.
Each task must pass four Admission Anchors before entering the benchmark: reference solution ≥ 0.95, do-nothing = 0, a literal-compliance solution ≤ 0.2, and any shortcut strategy ≤ 0.8. Graders were stress-tested against 270+ shortcut and bypass attempts.
for task in fde_gym: # 95 tasks, 20 scenarios, 18 sectors
agent = Agent(tools=[exec, read, write, edit, list])
while not agent.submitted and within_limits(): # 200 turns, 150 min, $8
agent.step(workspace, scripted_stakeholders)
deploy(agent.system, production_entry_points)
U = (kpi_delivered - kpi_baseline) / (kpi_reference - kpi_baseline)
U_adj = U if no_incident else min(U, 0)
C = sum(contact_charges) # messages, trial runs, waiting days
score = U_adj - C # resolved if score >= 0.8 and no incident
Thirteen model configurations from seven providers were run, each task three times, in one minimal agent framework.
•
Headline gap. At L3, Claude Opus 5.5 leads with mean score 0.56 and resolves 14 of 95 tasks. Five of thirteen configurations resolve zero. Even picking the best model per task only reaches 0.70 with 30 resolved.
•
The biggest drop is at L0→L1. Giving the correct problem but hiding the rules drops the three leading models from 0.73–0.82 down to 0.39–0.62. Going further to L2 and L3 barely moves scores for leading models (Claude Opus 5.5: 0.62, 0.54, 0.55). The authors’ prior hypothesis, that problem definition would be the bottleneck, is not supported for leading models. Weak models keep bleeding across every level.
•
Having the facts is not enough. At L3 the leading three obtain 72–85% of key facts from stakeholders but deliver only 0.60–0.70 of the reference uplift. In the 73 cases where a leading model got every key fact, uplift hit 0.8 in just 37%.
•
Contact cost eats the rest. For gpt-6-astra, 36 of 95 delivered systems reach uplift 0.8, but only 9 stay resolved after contact cost. Agents are not told questions cost anything and behave accordingly.
•
Weak agents cause harm. Before any contact-cost deduction, systems from Mistral Small 4 score below the do-nothing baseline on 51 of 95 tasks; on 23 of those the harm exceeds the reference solution’s benefit. Extended thinking on Claude Haiku 4.5 does not help (-0.33 vs -0.35).
•
Spend and effort don’t predict score. GPT-6.1 Sol spends $0.24 and 25 turns to score 0.37; Gemini 3.8 Flash spends $1.54 and 121 turns to score 0.19. Qwen 3.8 Flash submits within limits on only 11 of 152 runs.
•
Component benchmarks mislead. Models that solve up to 88%, 90%, and 61% of tasks on Terminal-Bench 2.1, Vibe Code Bench, and Finance Agent score 0.19–0.56 on FDE-Gym. The authors place these side by side on their own scales rather than treating the gap as directly comparable.
•
If you are evaluating agents for customer-facing delivery work, FDE-Gym gives you a scale where zero isn’t the floor and literal compliance is capped near 0.2. That design choice is worth borrowing even if you build your own eval: do-nothing baseline, expert reference, and a literal-compliance anchor keep an agent from gaming the metric.
•
The headline practical signal for buyers is that strong performance on terminal, coding, or finance-research benchmarks is a weak guide to delivery outcomes. If you plan to deploy agents into live customer environments, test them against your own held-out business metric rather than a success-rate leaderboard.
•
The L0→L1 collapse is the actionable diagnostic. If your in-house evaluation only varies implementation difficulty, you are missing the capability that matters most for leading models: discovering unstated constraints. Worth adding a condition that forces the agent to find rules through stakeholder interaction.
•
For agent builders: tell the agent that stakeholder contact has a cost. The paper’s agents weren’t told, and that alone drains 0.14–0.22 off the leading models’ uplift. Also worth testing whether agents can recognize when their work is actually complete, since the leading three submit on every run while weaker models hit timeouts.
•
The harness, runner, and three example tasks are open at GitHub and Hugging Face. The 20 original tasks and graders are available on request; the 75 variants and reference solutions are held out to protect the leaderboard.
•
Stakeholders are scripted for reproducibility, so real-customer effects like changing their minds or holding grudges are not represented. Contact costs are expert-set models of time cost, not measured values.
•
L0–L2 are reported only on the 20 original tasks (and three scenarios reuse the L3 instruction as L2). The near-flat L1→L3 result for leading models could still hide up to a 0.14 further drop at the aggregate level.
•
No human baseline is reported yet. The scale is anchored by do-nothing and the expert reference; how far human FDEs get from only the sponsor’s request is still being collected.
•
All models run in one minimal agent framework with five tools. A different scaffold could change both the scores and the ranking, so these numbers describe models in this harness, not the products built around them.
•
Scenarios skew toward regulated operations in energy, finance, and similar sectors. Scenarios are adapted from real deliveries since 2024, and the authors cannot rule out that related material appears in training data, though variants change the key facts.