EnvHarness wraps a static agent benchmark with three plug-in transforms (change the starting state, filter actions/observations, or chain two tasks together) so a diagnosis agent can automatically reshape the environment to hit a policy’s specific weaknesses, lifting held-out success by up to 9.0 points while keeping the benchmark’s original verifier untouched.
You’ve shipped an agent that resolves GitHub issues by running through a fixed set of training repos. Once it can pass those, you’re stuck: rebuilding fresh training environments from scratch is expensive, and the ones you have don’t target where your current agent actually fails. The dominant alternative is to generate new environments with an LLM, as in GenEnv, but those pipelines are benchmark-specific and their generated verifiers can hallucinate success. This paper’s move is to leave the benchmark and its verifier alone, and instead layer behavior changes on top of the existing reset/step interface.
The framing is an analogy: an agent harness wraps a frozen LLM with tools and memory to make it an agent; EnvHarness wraps a frozen environment with plug-in components to make it customizable. Three component types cover three axes. A Stage mutates the initial state by replaying actions through the environment’s own step function, so you get a reachable start state without touching internals (e.g., hide the mug in a drawer before the episode begins). A Contract intercepts the step loop with three hooks that can block actions, rewrite observations, or fake outcomes (e.g., reject a code submission until tests have run). A Chain concatenates two environments into one longer episode whose reward is the AND of both verifiers.
Because every component honors the same reset/step contract, they stack as decorators and stay domain-agnostic. Crucially, since the original transition function and scorer are never modified, the human-built verifier still grades the reshaped task.
The automation piece is EnvRigger, a four-stage loop that plays the target policy as a black box:
for task in training_tasks:
trajs = rollout(policy, env, k=5) # Observe
diagnosis = llm_diagnose(trajs) # find failure modes or
# note task is too easy
for _ in range(revision_budget): # Write + Validate
candidate = llm_write_components(diagnosis)
wrapped = apply(candidate, env)
fresh = rollout(policy, wrapped, k=5)
if accept(fresh): commit(candidate); break
diagnosis = revise(diagnosis, fresh)
Accepted components either scaffold missing steps (for a struggling policy) or inject harder scenarios (for a saturated one). Skills are then distilled from trajectories in the wrapped environments using ReasoningBank, and the skill-equipped policy is evaluated on held-out original tasks.
The prevailing move when a static benchmark stops teaching is to generate more environments, either by simulating them with an LLM or by synthesizing new task instances. This paper argues the opposite. You don’t need new environments; you need a programmable layer over the ones you already trust, conditioned on where the current policy actually fails. The evidence that makes this thesis load-bearing is the environment-scaling curve on SWE-bench Verified, where reshaping keeps improving while both real and LLM-generated environments flatten out under the same budget.
The load-bearing finding is the scaling curve, not the headline table. On SWE-bench Verified, as the environment budget grows to 300, EnvHarness climbs from 47.67 to 54.79 while the same budget of original environments plateaus at 52.13 and SWE-Smith-generated environments at 50.37. The gap widens with scale, because each EnvHarness batch is written against the policy after it has absorbed the previous skills, so agent and environment co-evolve.
•
On ALFWorld, skills from reshaped environments beat skills from original environments by +9.0 points on the out-of-distribution split and +5.9 on average.
•
On SWE-bench Verified, success rate rises +2.7 points while average episode length drops from 55.0 to 49.6 steps, because Contracts explicitly disrupt repetitive action loops.
•
Under online RL with Group Relative Policy Optimization (GRPO) on Qwen3-8B, training on reshaped environments beats training on originals on 3 of 4 metrics across ALFWorld and WebShop, with up to +6.5 points in-distribution SR.
•
Across four backbones from Gemini 3.1 Flash-Lite to Claude Sonnet 4.6, the gain over original-environment skills is a consistent +2.7 to +3.7 points, so the loop doesn’t depend on a specific model tier.
•
On five untouched original benchmarks, the same interface, prompts, and pipeline apply; specialized generators like GenEnv, VeriEnv, and SWE-Smith each only cover one benchmark.
One caveat visible in the tables: skills from unmodified environments can actively hurt (below the no-skill baseline on SpreadsheetBench, longer episodes on SWE-bench), so the write-and-validate gate is doing real filtering work.
Reach for this when you’re operating a code or web agent against a fixed set of training tasks with a deterministic verifier (a test suite, a DOM assertion, a spreadsheet diff) and you’ve noticed the agent has stopped improving. Instead of commissioning new tasks, wrap the existing ones: run 5 rollouts, have an LLM read the trajectories, propose a Contract that blocks the shortcut the agent is exploiting or a Stage that starts the episode past the easy prefix, and only keep the wrapper if fresh rollouts show a useful learning signal. The Contract pattern of “reject submission until tests have run” is a directly portable trick for any code-agent stack.
Code and prompts are released at github.com/google-research/envharness with a project site at envharness.com. The paper notes the released classes are named Setups, Rules, and Link rather than Stage/Contract/Chain. Each benchmark needs a one-time bridge to the shared ActionableEnv interface; after that, the loop is domain-agnostic.
Reshape the environment you already trust; don’t hallucinate new ones. The verifier is the expensive artifact, not the task distribution, so wrapping the reset/step interface with policy-conditioned components buys you targeted training signal without giving up ground-truth grading, and it keeps paying out as long as the diagnosis loop can still find weaknesses.
•
Requires a resettable environment. Anything backed by a live account, a physical robot, or a non-idempotent service breaks the Stage replay assumption and the Chain handoff.
•
The design loop is not free: the ALFWorld run spent ~1.46M designer tokens plus 226M rollout tokens, roughly 3.5× GenEnv’s total (though GenEnv’s rollouts are LLM-simulated, not executed).
•
Chain only supports serial concatenation with an AND-of-verifiers reward. Semantically related composite tasks (branching workflows, shared intermediate state) fall outside what a trusted composite verifier can express here.