EmbodiedCLBench tests whether LLM-driven household agents can reuse experience from simple tasks to solve harder compositional ones, finding that one prior task roughly triples success on unseen advanced tasks, but adding more examples barely helps.
Suppose you deploy a household robot agent powered by an LLM. You’ve shown it how to “put the cup on the table” and “close the cabinet.” Tonight you ask it to “tidy up the room,” which requires stitching those skills together with a few others it has never performed as a unit. Does the prior experience actually transfer, or does the agent start from scratch each time?
That question. called continual learning or self-evolution. has been studied mostly in text-only or web-automation settings. Prior work from the authors’ own survey literature covers knowledge QA, web agents, and persistent sandboxes, but not physically-grounded household tasks. The paper argues embodied settings are different because they involve spatial state, physical constraints, and long action sequences, and no existing benchmark targets this gap.
The core idea: operationalize continual learning as Compositional generalization. If an agent genuinely learned a skill, it should recombine learned skills to tackle novel combinations without being retrained on them. The benchmark builds three task tiers on top of the TongSim household simulator: atomic actions (e.g., walk-to, pick-up), basic tasks that chain 2 atomic actions (6 steps), and advanced tasks that chain 4 atomic actions (12 steps) drawn from 2 to 4 different basic-task categories.
Because the closed-source simulator can’t be run live at scale, the authors pre-render each task as a state-transition graph. Each node is a world state with a first-person image and a menu of candidate actions. Edges are the actions themselves. At every state the agent picks from three action types: optimal (on the shortest path), suboptimal (executable but off-path), or invalid (not executable). This gives offline, deterministic, parallelizable evaluation while preserving the multi-turn feel of embodied control.
Evaluation has two phases. In the learning phase, the agent completes 2 related basic tasks. In the testing phase, it attempts an unseen advanced task. Two learning methods are compared:
# ICL: keep raw trajectories in context
context = [trajectory(basic_1), trajectory(basic_2)]
for step in advanced_task:
action = agent(obs=image, candidates=opts, memory=context)
# Skill Learning: compress trajectories into a markdown summary first
skill_note = agent.summarize([trajectory(basic_1), trajectory(basic_2)])
for step in advanced_task:
action = agent(obs=image, candidates=opts, memory=skill_note)
The benchmark holds 268 tasks (168 basic, 100 advanced) across 6 household categories, run through 4 agent harnesses (OpenClaw, Claude Code, Codex agent harness, Hermes) paired with two backbones (Qwen3.8-Flash and GPT5.6-Terra). Primary metric is success rate within a step budget of shortest-path length plus 3 extra steps.
•
Zero-shot advanced tasks are hard. Agents hit around 50% on basic tasks but only around 10% on advanced ones without any prior experience. The composition itself, not the primitives, is what trips them up.
•
Experience helps a lot. Both In-Context Learning and Skill Learning lift advanced-task success from ~10% to above 30% consistently across all four harnesses. Neither method dominates the other.
•
One sample does most of the work. Going from 0 to 1 basic task raises success from 8\u201310% to 28\u201340%. Going from 1 to 6 basic tasks produces noisy fluctuation with no upward trend. Current agents can’t extract additional signal from additional examples.
•
Benefit is stable across complexity. As advanced tasks scale from 3 to 6 atomic actions, zero-shot collapses (28% to 4%), while learned-from-experience settings keep roughly a 30 pp gain over zero-shot. At the hardest level, ICL still reaches 32% versus zero-shot’s 4%.
•
What changes in behavior: experience cuts invalid action selections by 13\u201320% and redistributes those choices toward optimal (and some suboptimal) actions. The authors interpret the gain primarily as the agent avoiding non-executable moves rather than discovering shorter paths. Skill Learning produces smaller behavioral shifts than ICL but similar success rates, hinting that compressed summaries convey more focused guidance per token.
•
If you’re building a self-evolving agent for physical or household settings, this benchmark is now the obvious yardstick. Code and data are at the project page.
•
The 0-to-1 jump versus the 1-to-6 plateau is the most actionable finding. If your continual-learning pipeline shows gains at k=1 but flat scaling after, don’t assume a bug. The paper’s evidence suggests this is a general limitation of current ICL and summary-based memory, not something you’ve broken.
•
Skill Learning (compressing experience into a markdown summary) matches ICL’s success rate while using much less context. Worth testing in your own stack when context length or latency matters.
•
When diagnosing why an agent fails long-horizon tasks, measure the invalid-action rate separately from the overall success rate. The paper shows that “stops picking impossible moves” and “finds the shortest path” are different wins, and experience helps more with the former.
•
For benchmark designers: the state-transition graph trick is a reusable pattern when you have a rich but non-scalable simulator. You get reproducibility and parallelism at the cost of restricting the agent to pre-enumerated action choices.
•
The agent chooses from a fixed menu of pre-generated candidate actions rather than producing free-form actions. This simplifies evaluation but makes the task easier and less representative of open-ended control.
•
All results depend on action-option quality from Qwen3.7-Plus plus human verification. Shortcut leakage in option phrasing could inflate success rates.
•
Only two backbones were tested, both from the same era, and the paper reports results on 100 advanced tasks. Treat specific percentages as directional rather than precise.
•
The sample-scaling result (no gain beyond k=1) uses Hermes with Qwen3.8-Flash for the main analysis. Whether frontier reasoning models would show better scaling is untested here.
•
Success rate is the only metric. Efficiency, safety, and partial-credit measures are not reported.