FACET builds terminal-agent training tasks by realizing the Docker environment first, then generating the instruction, solution, and test verifier against that same observed container state, lifting end-to-end task validity from ~16-28% to 70% on shared inputs.
You want to fine-tune a smaller model to be a competent CLI agent, one that can pip install, edit configs, run scripts, and recover when a command fails. To do that you need thousands of tasks that each ship four things in lockstep: a natural-language instruction, an initialized environment, a reference solution, and a pytest-style verifier. When these are generated independently by an LLM, they drift. The instruction mentions a file that was never created, the verifier checks a port the environment doesn’t expose, the reference solution assumes a schema the fixture doesn’t have. The task looks fine on paper and is unsolvable in practice.
Existing pipelines like Nemotron-Terminal, Terminal-Lego, and TerminalWorld scale synthesis by mining recordings, skill catalogs, or procedural templates, but they treat the four artifacts as separately-generated text. FACET reorders the pipeline so the environment is a physical fact all other artifacts observe.
The key move is refusing to generate the instruction, solution, or verifier until a container has actually been built and inspected. The paper calls this executable-state grounding.
Start with a pool of ~71K reusable Agent Skills scraped from OpenClaw, ClawHub, and GitHub. An LLM judge clusters related skills into scenario-skill pairs, e.g., “parse a CSV, validate against a schema, publish a report.” Then a five-module reconstruction pass expands each pair into a rich workflow spec covering goal, context, per-skill capability, state transitions, and required inputs/outputs/tools. This spec becomes shared references R_S (solution) and R_I (instruction) that a consistency check aligns.
Stage 3 is where grounding happens. The environment agent writes a manifest, materializes fixtures (possibly downloading and localizing public data, then perturbing it to avoid template-y tests), and builds the Docker image. Up to three repair rounds fix build failures. Only then does the pipeline read the realized container state e_0 and generate artifacts sequentially, each seeing the actual filesystem:
build_and_repair(env, spec) # up to 3 tries
e0 = observe(container) # realized state
I = gen_instruction(R_I, e0)
S = gen_solution(I, R_S, e0)
eT = execute(S, e0) # actual final state
V = gen_verifier(I, R_S, e0, eT)
for _ in range(5): # targeted repair
fail = validate(I, env, S, V)
if not fail: break
repair_only(fail.artifact)
Validation demands four things: environment builds, verifier rejects the initial state, reference solution runs clean, verifier accepts the final state. If it fails, a router blames one specific artifact and rewrites only that one, avoiding regenerating the whole bundle.
The prevailing approach to synthetic agent tasks is to generate all the artifacts from a shared textual spec and hope they cohere. This paper shows the opposite. Build the environment first, treat the realized container state as the source of truth, and generate instruction, solution, and verifier as observations of that state rather than descriptions of a plan. The cleanest evidence is the Forward vs Reverse vs Joint ablation isolating generation order, not the Terminal-Bench 2.1 fine-tuning numbers.
On 100 shared scenario-skill pairs, three generation orders were compared. Forward (environment, then instruction, solution, verifier in sequence) hit 46.5% initial validity. Reverse (verifier before solution) hit 24.2%. Joint (all three artifacts in one LLM call) hit 37.5%. In a paired test on the 88 pairs all three schemes reached, Forward beat Reverse on 29 pairs versus 9 the other way (p=0.0017). That’s the load-bearing finding: solution-before-verifier alignment matters, and grounding artifacts in an observed environment matters more than clever prompt structure.
•
End-to-end yield: on 500 shared skill-pair inputs, FACET validates 70.0% of tasks versus 27.8% for a reproduced TerminalWorld-style pipeline and 15.6% for a no-reconstruction baseline. Validated FACET tasks are harder, not easier: P@1 of 25.1% with 21.5 avg commands per rollout, versus 80.8% and 12.8 commands for baseline.
•
Downstream fine-tuning: only 1.2K successful trajectories, distilled from DeepSeek-V4-Pro on ~6K FACET tasks, lift Qwen3.5 on Terminal-Bench 2.1: 4B goes 17.60 → 24.72, 9B goes 27.34 → 35.58, 27B goes 40.82 → 47.57. The fine-tuned 27B lands within 1.5 points of a 397B model on the same benchmark.
•
Task density: FACET tasks carry an average of 22.77 executable checks each, roughly 4-7x other datasets, which explains the lower P@1: 54% of failed rollouts miss only one or two checks.
Reach for this pattern when you’re building an evaluation or SFT corpus for any tool-using agent that operates on real state (terminal, browser, IDE, database). The transferable recipe: never let a generator write a test for state it hasn’t observed. Materialize the target environment first, expose it read-only to whatever writes the instruction and verifier, run your reference solution and let the resulting state ground the verifier. If validation fails, route the blame to one artifact and rewrite only that one, don’t regenerate the bundle.
The authors release the code, the models and datasets (fine-tuned Qwen3.5 checkpoints at 4B/9B/27B, the ~6K validated task bundles in Harbor format, and 1.2K SFT trajectories), and a project page. Tasks are directly runnable under Terminal-Bench’s evaluation harness.
Generate the environment first and let it write the test, don’t ask an LLM to imagine what the test should check. Every synthetic-task pipeline for agents that touch real state should treat the built container, not the spec doc, as the shared source of truth across instruction, solution, and verifier.
•
The reproduced TerminalWorld comparison is a best-effort reimplementation, not the authors’ own system evaluated at parity, so the 70% vs 28% yield gap partly reflects adapter and prompt differences the paper acknowledges.
•
Forward, Reverse, and Joint ran under different repair budgets (5 vs 3 vs 3 rounds), so the final-yield numbers mix generation-order effects with repair-budget effects; only the initial-validity and paired sign-test numbers cleanly isolate order.
•
Downstream gains are shown only on Terminal-Bench 2.1 with the Terminus-2 scaffold and a single teacher (DeepSeek-V4-Pro). Whether the 1.2K trajectories transfer to different scaffolds, longer-horizon tasks, or RL rather than SFT is not tested.