ProgramGym builds 1,648 verifiable training environments for coding agents that rebuild a program’s behavior from scratch, by auto-generating reference executables, sandboxes, and black-box test suites from GitHub repos, then pairs them with a role-isolated analyze/implement/review loop.
Most coding-agent benchmarks hand the model a repo and ask for a patch (SWE-bench) or hand it a spec and ask for an implementation. Neither captures what a real engineer does: start from a working program plus docs, poke at it, and rebuild its behavior from nothing. That setting is called program reproduction from scratch, and the main benchmark for it, ProgramBench, is brutal. No flagship model fully solved any of its 200 tasks at release.
If you want to train agents for this task rather than just evaluate them, you need thousands of environments where (a) a reference binary exists, (b) a sandbox can run candidate code, and © a test suite can judge whether the candidate’s observable behavior matches. Existing issue-resolution gyms like SWE-Gym don’t give you that. The one prior effort for from-scratch reproduction, MindForge, was limited to command-line programs and didn’t verify that trajectories actually matched reference behavior, so bad code could be labeled as success.
ProgramGym’s contribution is a pipeline that turns ordinary GitHub repos into these verifiable environments at scale, plus an agent harness designed for the long-horizon workflow this task demands.
Two pieces: Program-Env (makes the training environments) and Program-Agent (solves tasks in them).
Program-Env starts with ~30k candidate repos in C, C++, Go, and Rust, filtered by stars and build-config presence. For each repo it runs an agent (built on Claude Code) with a feedback loop: generate a download.sh for online dependency setup and compile.sh for offline compilation, then validate. If the build fails, structured diagnostics (which lint rule tripped, which library is missing, suggested fix) get fed back for up to four attempts. Repos that build produce a reference executable and a frozen sandbox image.
Then a dispatcher splits each repo into functional modules and assigns each to a parallel subagent that writes black-box tests. The clever bit is the four validation checks that filter generated tests each round: pass (test must pass on the reference), double (same result across two runs), dummy (test must fail against an empty stub, otherwise it isn’t actually checking anything), and weak-assert (37 lint rules rejecting tautological or exit-code-only assertions). Rejected tests plus coverage gaps become structured feedback for the next round. Joern and CodeQL trace uncovered code back to public entry points so the agent knows which inputs could reach them.
Program-Agent runs three stages in separate sessions with no shared conversation history: Analysis (read docs and probe reference, write frozen notes), Implement (write code without access to the reference, based only on handoff artifacts), Review (compare built candidate against reference, log discrepancies). Rounds alternate Implement and Review for a fixed budget K. Independent compilation of a clean source snapshot gates each round; failed rounds roll back to the last validated revision.
A = analysis(docs, reference_binary) # frozen, read-only
for k in range(K):
source_k = implement(docs, A, reviews[:k]) # no ref access
exe_k = independent_build(source_k) # gate
if k < K - 1:
reviews.append(review(docs, A, source_k, exe_k, reference_binary))
return source_k
On ProgramBench, the authors evaluate the base Qwen3.8-27B and an SFT version fine-tuned on trajectories collected from Program-Env, under two scaffolds: Claude Code (single-session baseline) and Program-Agent.
•
Training on Program-Env helps. Holding the scaffold fixed at Claude Code, SFT raises Qwen3.8-27B from 34.87% to 42.84% Tests Passed, closing part of the gap to DeepSeek-V4-Flash-0731 (57.68% under the same scaffold).
•
Program-Agent helps more than SFT does. Holding the model fixed, swapping Claude Code for Program-Agent lifts the base model from 34.87% to 55.71% and the SFT model to 56.25%, both above Claude Opus 4.7 (50.90%) under Mini-SWE-Agent.
•
Extra iteration rounds keep paying. Tests Passed goes from 43.49% at K=1 to 54.13% at K=3 to 56.25% at K=5, with most gain by K=3.
•
Full reproduction is still unsolved. Even at 50%+ Tests Passed, the stricter “Almost Resolved” (>=95% tests pass) and “Resolved” (all tests pass) numbers stay near zero. Flagship models under mini-SWE-agent report 0.00% Resolved. The authors are clear: partial behavior matching is not whole-program reproduction.
•
Environment stats. 1,648 environments across four languages, 665k validated test cases, mean per-repo coverage 72.99% (median 81.63%), built at a cost of ~137B tokens.
The Program-Agent gain is a scaffold effect, not just an SFT effect. Agent-behavior analysis shows Program-Agent shifts tool-call distribution toward read and git actions (exploration and revision) versus Claude Code’s execute and probe lean.
•
If you’re training a code agent and your current environment pool is issue-resolution style (patches on a given repo), ProgramGym is a different and complementary signal source. The pipeline is automated, so you can in principle run it on the ~29k repos the authors filtered out but didn’t process due to budget. Worth testing whether the SFT gain generalizes to your base model.
•
If you’re building an agent harness for long-horizon coding, the role-isolation idea (separate sessions for analysis, implementation, review, with frozen handoff docs instead of shared history) is cheap to try and the paper’s ablation suggests it is where most of the gain comes from, not the fine-tuning. The specific trick of forbidding the implementer from seeing the reference program is counterintuitive but motivated: it stops the agent from drifting into open-ended edge-case matching.
•
If you’re designing test-generation pipelines, the four-check filter (pass, double, dummy, weak-assert) is a transferable recipe. The dummy check in particular catches a failure mode (tests that pass regardless of implementation) that pure coverage metrics miss.
•
Treat the headline numbers as Tests Passed, not Resolved. A 56% Tests Passed score with ~0% Resolved means the agents are producing partial reimplementations, useful as training signal but not as deployable programs.
The authors say they will release ProgramGym; the paper does not include a public link in the supplied text.
•
Difficulty skews easy: 1,492 of 1,648 tasks are rated easy under the paper’s complexity score, 5 are hard. The authors acknowledge this and argue harder tasks yield too few successful trajectories to be useful training signal yet.
•
Coverage is 73% on average, not complete. Passing the test suite proves behavioral agreement on tested inputs, not semantic equivalence. The benchmark’s near-zero Resolved rate reflects this gap.
•
Both the environment construction and the solving scaffold lean on Claude Code and DeepSeek-V4-Flash-0731. How much the pipeline’s quality depends on those specific agent harnesses versus the method itself isn’t isolated.
•
The paper does not compare head-to-head with MindForge because that work’s environments weren’t released. Prior-art positioning is therefore partially argumentative rather than empirical.
•
Model and benchmark names in the paper (Qwen3.8-27B, GPT 5.4, Claude Opus 4.7, Gemini 3.5 Flash, DeepSeek-V4) reflect the paper’s stated versions; verify against current releases before drawing cross-paper comparisons.