CodeMidas builds coding RL training tasks straight from source code (not issues, commits, or existing tests) by having agents delete a function, write a spec for it, and generate execution-grounded tests, yielding 5,545 tasks across 23 languages that improve an LLM coding agent on every benchmark tried.
If you want to train a coding agent with RL, you need thousands of tasks where a verifier can reliably say “this attempt worked.” The dominant recipe, popularized by SWE-bench, scrapes GitHub issues and pull requests: the issue becomes the task, the PR’s tests become the verifier. That works, but you’re bottlenecked by whatever people happened to file issues about, in whatever language their project used (mostly Python). Follow-ups like R2E-Gym and SWE-Smith loosened this by generating tasks from commits or by mutating existing tests, but they still assume the repo comes with a usable test suite or change history.
The practical cost: your training distribution mirrors “bugs people reported in popular Python repos,” which is a narrow slice of what coding agents actually do (writing new programs, terminal work, translating code across languages). CodeMidas asks whether the implemented code itself, with no issues, no PRs, no docs, no pre-existing tests, is enough raw material to manufacture training tasks.
The core trick: take a working piece of functionality in an open-source repo, rip it out, and reconstruct a task around the hole. Concretely, an agent walks the codebase to find functionality with a clear public interface (a CLI command, a library function, a stateful API). It then does four things per candidate task:
1.
Task design. The agent removes the target implementation, patches up the surrounding code so the repo still builds, and writes a behavioral spec describing inputs, outputs, and required public interfaces. The original code is kept aside as a reference solution.
2.
Test construction, grounded in execution. Rather than asking an LLM to guess expected outputs, the agent runs the original implementation on chosen inputs and records what happens. Those recorded outcomes become the assertions. A review pass strips assertions that overfit to incidental details (exact error wording, internal ordering) so alternative correct implementations still pass.
3.
Execution consistency check. Each task is run six times in fresh containers: twice with the code removed (must fail) and four times with the reference solution restored (must pass). Flaky tasks are dropped.
4.
Post-rollout filtering. Before the task enters training, three more agent-driven checks run: an adversarial agent hunts for leakage (compiled artifacts, caches, installed copies of the project) that would let a solver cheat; a reviewing agent cross-checks verifier verdicts against actual submitted solutions to catch bad tests; and tasks where a frontier model gets all-pass or all-fail on repeated attempts are dropped as uninformative.
Rough pseudo-code for one task’s construction:
for repo in codebases:
target = agent.pick_functionality(repo) # public interface, cross-file reasoning
starter, reference = agent.remove_and_patch(repo, target)
spec = agent.write_behavioral_spec(target)
tests = agent.build_tests(spec, run=reference) # execute reference to get outputs
if not (fails_on(starter, tests) and passes_on(reference, tests)):
continue
if adversarial_agent.finds_leak(starter): continue
if reviewer.disagrees_with(tests, sample_solutions): continue
if frontier_model.all_pass_or_all_fail(tests): continue
yield Task(spec, starter, tests)
Training then uses Group Relative Policy Optimization (GRPO) with a binary pass/fail reward from the hidden verifier. No reward model, no learned verifier.
The initial policy is MiMo-V2.5 (Xiaomi’s model), and RL on the 5,545 CodeMidas tasks improved every one of five held-out benchmarks:
•
DeepSWE pass rate: 10.0% → 21.7% (issue repair)
•
ProgramBench Almost Solved: 4.5 → 21.5 (whole-program construction, scored as % of tasks passing ≥95% of tests)
•
Terminal-Bench v2.1 pass rate: 63.7% → 72.2% (shell/terminal tasks)
•
Gains also reported on SWE-bench Pro and RepoZero C2Rust (a C-to-Rust translation benchmark)
The most informative ablation is on task quality vs. quantity. They compare three high-quality pools (1k, 3k, 5k tasks, all filtered) against a vanilla 8k pool that skipped the environment cleaning, execution consistency, and post-rollout filtering. The full 5k pool beats the vanilla 8k by ~4.5 pp on both DeepSWE and CodeMidas Val, and the 3k filtered pool also beats vanilla 8k across all three evaluations. Scaling within the filtered pool helps monotonically (1k → 3k → 5k gives progressively higher scores), but filtering matters more than raw count.
Behaviorally, RL-trained agents change how they act: pre-edit read/search calls rise from 27.2 to 40.1, the “drafting ratio” (fraction of written code fragments that appeared in prior reasoning) rises from 0.36 to 0.63, and distinct post-edit verification commands rise from 2.03 to 2.53. Rollouts where the agent wrote and ran its own checks had a 4.2 pp higher pass rate than those without (95% CI 1.8–6.6). These behavioral shifts show up on held-out benchmarks too, which is the authors’ evidence that the model learned generalizable habits rather than benchmark-specific tricks. Note: the pass-rate association with self-verification is correlational within the same task and checkpoint, not a causal claim.
•
If you’re building a coding RL dataset and hitting the Python-issue ceiling, this is a concrete recipe for going wider: 23 languages, 15 domains, no dependence on issue trackers. The core insight worth borrowing is that executing the original code is a cheap and reliable way to generate test oracles, which sidesteps the “who writes the ground truth?” problem that plagues LLM-generated test suites.
•
The filtering pipeline is where the value concentrates. Post-rollout filtering (leakage checks, verifier-vs-solution agreement, keeping only tasks with mixed pass/fail outcomes) is what makes 3k filtered tasks beat 8k unfiltered. Worth testing the individual filters in isolation on your own pipeline; the paper reports them as a bundle rather than ablating each.
•
If you’re evaluating coding agents, the behavioral metrics in Appendix B (pre-edit exploration calls, drafting ratio, distinct post-edit verification commands) are a lightweight instrumentation kit for tracking whether RL is teaching habits or just memorizing. They don’t need model internals, just trajectory logs.
•
Not established by this paper: whether the method transfers to models other than MiMo-V2.5, whether the constructed tasks are competitive with human-curated ones at equal count, or how much of the gain comes from GRPO specifically vs. the data. The paper does not release the dataset or pipeline code within the supplied text.
•
Single-model result. All numbers come from RL on MiMo-V2.5; there’s no comparison against training the same base model on a competing dataset like SWE-Smith or R2E-Gym under matched compute, so “CodeMidas data is better” isn’t directly shown; only “CodeMidas data helps this model.”
•
The pipeline itself relies heavily on capable agents at construction time (task design, test writing, adversarial leakage hunting, verifier review). The paper doesn’t specify which model drives these construction agents, so reproducing the pipeline requires guessing at that budget.
•
Task selection favors functionality with clean public interfaces and observable outcomes. Work that’s hard to spec behaviorally (UI, ML training code, anything with heavy nondeterminism) is likely underrepresented, though the paper doesn’t quantify this.
•
The behavioral generalization story is suggestive, not causal: agents explore more and verify more on held-out benchmarks after RL, but the paper doesn’t isolate which training-time signal produced which habit.