EvoPolicyGym measures whether a coding agent can iteratively rewrite an executable Reinforcement Learning policy under a fixed 128-episode feedback budget, and finds that the leaderboard gap is driven by Structural Synthesis on pixel and symbolic tasks, not by parameter tuning.
Suppose you’ve shipped an agent that writes a Python controller for some closed-loop system: a trading rule, a retry policy, a game bot. It runs the code, sees rewards, edits, tries again. Today you report one number: did the final version work? That hides everything interesting. Did the agent discover the right shape of solution, or did it just fiddle with constants on a broken skeleton? Prior evaluations of self-improving agents like Reflexion score the final answer, and coding-agent benchmarks like SWE-bench score a one-shot patch against unit tests. Neither isolates the loop where an agent has to convert a bounded stream of rollout feedback into a policy that generalizes.
The paper defines Autonomous Policy Evolution as a controlled version of that loop. One run pins the agent to one environment, one starting policy file, and a hard budget of 128 rollout episodes. The agent edits a Python file policy.py (plus any helper modules it writes), calls a /submit endpoint to spend some of its remaining episodes on a train evaluation, reads back returns and per-step trajectories, and revises. Validation and held-out splits stay server-side. Scoring is the held-out return of whichever checkpoint wins on the hidden validation split, so the agent can’t just overfit to what it sees.
The benchmark instantiation is Core16: 16 tasks pulled from four families (Gymnasium classic control and Box2D, MuJoCo locomotion, MiniGrid symbolic navigation, and robotics/driving including Fetch and CarRacing). Because raw reward scales aren’t comparable across environments, the leaderboard aggregates using per-environment ranks against the other agents plus a uniform-random baseline, not raw returns.
The diagnostic contribution is a mechanical classifier for what kind of edit each submit represents. The paper parses the submitted policy bundle into an Abstract Syntax Tree, strips numeric constants, and asks: did the topology change? If yes, this is a synthesis edit (new module, new branch, new planner). If no, it’s a parametric edit (same skeleton, different constants). An edit is counted as a hit only if it beats the previous validation best.
for submit in run.score_bearing_submits():
prev, curr = strip_numeric_constants(submit.prev_ast), \
strip_numeric_constants(submit.curr_ast)
if curr == prev:
kind = "parametric" # same shape, tuned constants
else:
kind = "synthesis" # new topology
hit = submit.val_score > run.best_val_so_far
Environments are also pre-split into synthesis-dominant (pixels, symbolic planning, memory) versus tuning-dominant (low-dim control where a plausible controller family exists). That split is what lets the analysis attribute the leaderboard gap to a specific failure mode.
The usual read on “which frontier model is the best coding agent” is a single leaderboard number. This paper shows the opposite. The leaderboard gap between strong and weak agents on iterative policy tasks is almost entirely a gap in structural synthesis: inventing the right kind of program, not tuning constants inside one. The evidence isn’t the aggregate rank, it’s the per-edit-type success rate on synthesis-dominant tasks, where the top agents convert new-topology edits into validation gains 4 to 16 times as often as the weakest ones.
The load-bearing finding is the edit-type table. On synthesis-dominant tasks, GPT-5.5 and Claude Opus 4.7 turn synthesis edits into new validation bests at 41% and 48%. MiniMax-M3 and DeepSeek-V4-Pro sit at 10% and 3%: they change the code’s shape constantly but rarely land on a shape that works. On tuning-dominant tasks the four agents cluster much closer (normalized scores 0.67 to 0.99), so the suite-wide ordering is basically decided by whether an agent can invent the right controller when the task demands it.
Secondary results back this up:
•
On the aggregate rank score, GPT-5.5 leads at 0.891 with top-two placement on all 16 environments; Claude Opus 4.7 is second at 0.750, with the best score on the MiniGrid family.
•
MiniMax-M3 (0.531) and DeepSeek-V4-Pro (0.359) each win exactly one environment but collapse on the rest. The uniform random baseline scores 0.109, mostly from MiniGrid ties where every agent scores zero.
•
On the normalized synthesis axis, GPT-5.5 and Claude Opus 4.7 hit 0.98 and 1.00; MiniMax-M3 and DeepSeek-V4-Pro sit at 0.19 and 0.03, and neither solves any of the three locked-door MiniGrid rooms.
•
Case studies illustrate the mechanism: on CarRacing, the winning runs build a road-mask perception module first, then tune. On BipedalWalker, only GPT-5.5 discovers a gait topology that produces positive return; the other three churn structures without ever crossing zero.
The paper flags that AST topology is a coarse proxy: two topologies can behave alike, one topology can mix good and bad ideas. The signal survives anyway because it lines up with scores, edit outcomes, and hand-audited traces.
Reach for this if you’re building or evaluating a code-writing agent that improves under a verifier and a budget. The concrete move is the AST-topology diff: it’s a cheap, deterministic way to separate “the agent found a new approach” from “the agent nudged a constant,” which lets you diagnose whether a weak run is stuck exploring wrong shapes or stuck failing to tune a right one. That distinction changes what you fix: prompting for more architectural variety versus giving the agent tools to sweep parameters.
The code and experiment traces are released, including full submission histories, feedback logs, and validation curves for all four agents on all 16 environments. The adapter layer already wraps a much larger surface (Atari, MetaWorld, BrowserGym, MiniWoB++, HighwayEnv), so extending to new tasks is mostly writing new observation schemas rather than a new protocol.
When an agent has to write code that runs against a verifier, measure whether it can find the right shape of program, not just tune constants inside a wrong one. The frontier gap between models on iterative policy tasks lives in that first step; if your eval collapses to a final score, you can’t see it.
•
Only four agent configurations are tested, and each model is locked to one harness (GPT-5.5 through Codex, the rest through Claude Code). Harness effects and model effects aren’t separated, so “GPT-5.5 wins” is really “GPT-5.5-plus-Codex wins.”
•
The 128-episode budget rules out conventional RL training entirely, so the comparison is coding-agents versus coding-agents. Claims about relative strength don’t transfer to settings where you’d actually train a policy network.
•
AST-topology-after-stripping-constants is a proxy for “different mechanism.” A cosmetic refactor can look like synthesis; a genuinely new idea implemented with the same control flow can look parametric. The authors acknowledge this and treat the diagnostic as converging evidence, not a measurement.