TL;DR
Self-HPO trains a coding agent by alternating two steps: the current policy proposes fixes to its own Agent harness based on recent failures, then reinforcement learning updates the policy under that new harness, with hard tasks from training seeding the next harness search.
Why It Matters
An LLM coding agent that resolves GitHub issues is really two things glued together: a policy (the model weights that pick actions) and a harness (the scaffold that decides what the model sees, how its bash commands run, what counts as a submission, and when the episode ends). A good policy can still fail if the harness accepts an empty patch as a final answer. A good harness can still fail if the policy doesn’t know how to use it.
Most prior work picks a side. RL with verifiable rewards holds the harness fixed and trains the model. Self-Harness holds the model fixed and rewrites the harness. Both plateau: a harness tuned for one checkpoint stops matching the failures of a smarter checkpoint, and policy training only ever sees trajectories the fixed harness permits. A few recent systems try to co-evolve both, but they bring in a separate frozen proposer model to redesign the harness, which makes the whole loop depend on external supervision.
The paper’s bet is that the policy you’re training is already the best-informed judge of its own failure modes, so it should also be the one proposing harness changes.
How It Works
Think of two interleaved loops. In the H-step, the current policy is frozen and used as a code-writing agent whose job is to inspect its own failed trajectories on a small search set and propose a revised harness (a Python subclass of the seed agent, plus prompt tweaks). The authors generate K=6 candidate harnesses per round, score each on the search set, and keep the best. In the P-step, that harness is frozen and the policy is updated with reinforcement learning on a 2,484-task training set, using a binary pass/fail reward from an executable verifier.
Two design choices tie the loops together. First, the proposer in round t+1 is the policy that just finished training in round t, so harness proposals track the policy’s current weaknesses rather than the base model’s. Second, the search set for the next H-step is not fixed. It’s rebuilt from hard tasks: training tasks that the policy failed on every rollout during the preceding P-step. So harness search chases a moving target of real current failures.
For the RL optimizer, the paper uses Group Relative Policy Optimization (GRPO) combined with Balanced Adaptive Rollout, which oversamples rollouts and keeps groups that have a mix of successes and failures. This matters because binary rewards produce many all-zero groups that give GRPO no learning signal; BAR cuts that waste.
H, P = seed_harness, base_policy
S = random_sample(train_data, 300)
while not converged:
# H-step: policy proposes K harness edits, keep best on S
candidates = [P.propose_harness(H, failures_on(S)) for _ in range(K)]
H = argmax(candidates + [H], key=lambda h: eval(h, P, S))
# P-step: RL-train policy under new harness
P, rollouts = grpo_bar_train(H, P, train_data)
S = [task for task in rollouts if never_resolved(task)]
The seed harness is mini-SWE-agent, a basic ReAct-style loop. The base policy is Qwen3.5-35B-A3B, a mixture-of-experts coding model. One concrete example of what the harness learned to do: the second-round optimized harness adds a one-shot guard that rejects empty patches, nudges the model to run git add -N for new files, and falls back to the sandbox’s own git diff if the final patch doesn’t apply cleanly. These are all failure modes of the specific policy, not generic improvements.
What They Found
On three benchmarks (SWE-bench Verified, SWE-bench Multilingual, and SWE-bench Pro), Self-HPO lifts the base agent’s average resolve rate from 57.8% to 64.8%, beating both the policy-only baseline (RLVR, 61.9%) and the harness-only baseline (Self-Harness, 60.1%). On SWE-Bench Verified specifically, it goes from 61.0% to 69.5%. Gains transfer to the two out-of-distribution benchmarks even though training only used Python tasks.
A cross-evaluation ablation is the most informative result: swapping the final harness H2 onto the base policy gives +3.5 points, and swapping the final policy P2 under the seed harness gives +3.4 points, but combining them gives +8.5 points, which exceeds the sum of their individual contributions. So the two components are not just independently better, they’re co-adapted.
Two other findings worth flagging. Replacing the evolving hard-task search set with a fixed one drops SWE-Verified from 71.6% to 70.1%, so chasing current failures helps modestly but measurably. And in the external-proposer setup, using Opus 4.8 to propose harnesses pushes SWE-Verified further to 71.6%, beating Supervised Fine-Tuning distillation from Opus 4.8 trajectories by a wide margin, especially out of distribution where SFT actually hurts.
The authors also report pass@k gains: pass@1 improves across all task difficulty bands except “already resolved” tasks (where it drops 2.2 points), and pass@8 rises from 80.0% to 84.8%.
What’s Useful
If you’re training a coding agent and currently only tune one side (either prompt-engineering a harness around a fixed model, or RL-tuning a model under a fixed harness), the cross-evaluation result is the actionable finding: the two components leave gains on the table individually that only appear when co-adapted. Alternating rounds is worth testing before scaling either side further.
If you can’t run RL on the policy, the Self-Harness-style approach (harness-only) still gives a meaningful chunk of the improvement, but it clearly plateaus faster than co-training in the paper’s trajectories.
The hard-task search set idea is cheap to retrofit onto any harness-search pipeline: instead of validating candidate harnesses on a static dev set, validate them on tasks that your current policy actually fails. The paper shows ~1.5 points from this alone.
For practitioners picking between SFT on a strong teacher model’s trajectories versus harness optimization with that same teacher, the paper’s evidence favors harness optimization, particularly for cross-language or out-of-distribution generalization. SFT in their setup underperformed the base agent on both OOD benchmarks.
Caveats
The method requires full training access to the policy, including RL rollouts on executable sandboxed environments, so it’s not something you apply through a hosted API. The RL stack is specific: GRPO plus BAR on eight B200 GPUs, with the Orchard framework, which the paper doesn’t claim to have ablated against simpler RL recipes.
All evaluation is on SWE-bench-family benchmarks with a single base policy and a single seed harness. The headline numbers don’t tell you how much of the gain comes from Qwen3.5-35B-A3B’s particular failure modes being well-matched to this harness search space, versus a general property of the method. The optimized harness changes are quite specific (empty-patch guards, git-add-N nudges), which suggests the search is finding model-and-domain-specific fixes, not universal ones.
The self-proposer underperforms the external Opus 4.8 proposer by ~2 points on SWE-Verified. So “the agent improves itself with no external help” is true but costs some performance versus letting a stronger model do the harness design.
Finally, the paper only runs two full H/P rounds before the hard-task set stabilizes. Whether further rounds would keep improving the agent, or whether this is close to the ceiling of what the search space allows, is not established.