MerchantBench runs LLM agents as the sole operator of a simulated online store for a full year of hourly ticks, exposing whether they keep coherent policy as delayed order feedback accumulates. The best LLM configuration reaches only 27.3% of human final net assets.
Imagine you’ve deployed an agent to run a support queue, a marketing pipeline, or a small operations workflow autonomously for months. It handles day one fine. Six weeks in, is it still pursuing the original objective, or has it quietly narrowed its behavior to reacting to whatever pings it most loudly? Most agent benchmarks can’t tell you. They score bounded tasks with immediate pass/fail signals: file a ticket, book a flight, finish a workflow. Prior long-horizon business benchmarks like Vending-Bench pushed on sustained operation, but with narrow catalogs and simpler feedback. MerchantBench targets the specific failure mode of long-horizon deployment: the agent has to associate consequences with decisions made many days earlier and revise its own policy accordingly.
The agent operates a dropshipping store on a simulated version of 1688 for 365 days, activated every 12 simulated hours. It picks products from a catalog of 98,843 real product records, sets prices, manages a starting cash balance of RMB 2,000 plus a RMB 1,000 security deposit, and can hold at most 50 active listings. The store dies if the deposit hits zero.
The load-bearing design choice is the mixed-latency feedback loop. Upstream events (a supplier raises price, delists, or ships slow) are visible almost immediately through tool calls. Downstream order outcomes (cancellations, returns, bad reviews, stockouts, late shipments) are sampled at order creation but only surface days later through the order lifecycle. That means a bad sourcing decision on Day 40 might not show up as accumulated fines and rating damage until Day 60, and only if the agent bothers to look. Demand itself is drawn from real 365-day product-level histories, so seasonal peaks like 618 and 11.11 shift what’s worth stocking over the year.
The formal wrapper is a partially observable Markov decision process with 8,760 hourly steps. Plain reading: the agent never sees the true hidden risk parameters, only realized events. Reward is paid once at the end, as terminal net assets (cash + deposit + funds in transit + receivables).
for hour in range(8760):
advance_demand_supply_orders(state) # hidden dynamics tick
if hour % 12 == 0:
obs = render_visible_state(state) # store, recent orders, supplier changes
while not agent.end_of_step():
action = agent.decide(obs, history) # one of 26 tools
state = apply_tool(state, action)
reward = terminal_net_assets(state) # settled after horizon
Agents run under two scaffolds: ReAct (26 store tools only) and Hermes (same tools plus code execution, persistent memory, and a skill-authoring system). Eight LLMs are evaluated, three runs each, giving 48 total 365-day runs. Three humans with no e-commerce experience serve as the upper reference.
The prevailing way to benchmark agents is to score them on bounded tasks where success is checked at the end of the episode. MerchantBench argues that this misses the actual failure mode of long-running deployments. What breaks is not any single decision but the agent’s ability to keep operating on-objective for months while delayed evidence trickles in. The evidence that makes this thesis stick is not the headline net-asset gap. It’s the trace-level pattern of Operational Coherence and Strategic Coherence failures the authors surface: agents that quietly stop acting, or keep acting but drift off-goal.
The load-bearing finding is behavioral, not scoreboard. Human operators hold a Sustained Window Rate of 100%, meaning every rolling 30-day period contains real store actions. LLM configurations range from 10.6% to 99.4% under ReAct and 17.8% to 66.1% under Hermes. Several models exhibit Activity Decay: Qwen3.7-Max’s quarterly effective window rate drops from 68% to 23% under ReAct. One Kimi K2.6 run concludes on Day 104 that the store cannot recover and takes no environment action in 355 of the remaining 523 windows, despite feasible moves. A Claude Opus 4.8 run contracts its shelf from 47 listings to 3 based on a false inference that removing weak listings concentrates traffic. These are coherence failures, not capability failures.
Secondary numbers back the same story. The best LLM configuration (Qwen3.7-Max under Hermes) reaches roughly RMB 59.5k in final net assets against the human mean of RMB 217.6k, i.e., 27.3% of human performance. Framework matters: averaged across models, Hermes lifts final net assets 53.3% over ReAct, but Kimi K2.6 actually does worse under Hermes, so richer scaffolding is not uniformly a win. Humans also reallocate their portfolio seasonally, with monthly demand alignment percentile climbing from 56 in June to above 80 in December; most LLMs plateau or drift down.
Reach for MerchantBench when you’re about to ship an agent that has to run for weeks against a system with delayed side effects, and you want a stress test that surfaces slow-drift failures your unit tests won’t catch. The concrete transferable idea is the diagnostic split: separate whether the agent is still acting (Sustained Window Rate over rolling 30-day windows) from whether its actions still serve the goal (portfolio alignment against a no-reallocation counterfactual). Both are cheap to instrument on your own system and catch different failure modes.
The paper introduces the benchmark and reports evaluation results. It does not link a public code or data release in the text provided, and the underlying 1688 product records are proprietary marketplace data, so direct reuse likely requires the authors’ cooperation. If you want the scaffolding piece independently, the Hermes agent framework used as one of the two conditions is released by Nous Research.
Long-horizon agent quality is a coherence problem, not a capability problem. The frontier models tested here can each execute any single merchant action competently. What they lack is the discipline to keep pursuing the same objective for a year while delayed evidence rewrites the picture. If your product depends on an agent staying on-task for months, measure whether it is still acting and still on-goal separately, because the same score can hide either failure.
•
Three human participants with no prior e-commerce experience is a thin upper bound; the 27.3% gap is real but the exact multiplier should not be over-interpreted.
•
The simulator’s demand, risk, and rating dynamics are calibrated to one marketplace (1688) and one year of data. Coherence failures that show up here may partly reflect that specific reward shape, not agent behavior in general.
•
Only two agent scaffolds are tested and context is compressed by the evaluated model itself. A stronger memory or planning layer might close much of the gap, so the results bound what current off-the-shelf setups do, not what is achievable.