A function call and one step of a coding agent share the same four-part shape: context, call, externally-returned value, continuation. The authors add a dedicated Fill-in-the-Middle mid-training stage that masks whole functions chosen by dependency-graph analysis, lifting SWE-bench Verified by +2.8 to +3.2 points across three base models.
You ship a coding agent. It reads an issue, edits a file, runs the tests, sees a stack trace, and has to keep going. The behavior you need is: use the tool return to decide what to do next. Standard code pretraining doesn’t practice that. It reads code left to right and predicts the next token, so the model rarely learns to condition on a value that was computed elsewhere and then flowed back in.
The dominant fix right now is trajectory-based post-training. Pipelines like R2E-Gym or SWE-Smith synthesize agent trajectories and fine-tune on them. This paper argues there’s a missing stage before that post-training, and it can be filled from ordinary GitHub code without any agent trajectories at all.
The key observation: when function foo calls bar(x), the code around that call has the same structure as one agent step. There’s setup context, a call, a returned value produced somewhere else, and downstream code that uses it. So if you train a model to reconstruct bar's body given the file around it, you’re teaching the same bidirectional conditioning an agent needs.
The recipe: for each Python file, parse the AST and build a Program Dependency Graph with call edges and same-class sibling edges. Score every function on two axes. Complexity (lines, McCabe Cyclomatic Complexity, nesting depth) says the function is worth learning. Inferability (caller argument specificity, in-file callees, type signature, docstring, class siblings) says the surrounding file gives enough signal to reconstruct it. A harmonic-mean-like combination forces both to be high, so you don’t mask trivial getters or unrecoverable black boxes.
For each selected function, Gemini-3-Flash writes a chain-of-thought rationale plus a candidate body from the masked file (never seeing ground truth), and a second judge call filters low-quality pairs. Rationale and body both get placed inside the Fill-in-the-Middle middle span, so the model is trained to think before it writes code. A multi-function variant masks groups of 2 or 3 coupled functions to cover patches that span helpers.
for file in corpus:
pdg = build_pdg(parse_ast(file))
for fn in pdg.functions:
H = complexity(fn) # LoC, cyclomatic, depth
I = inferability(fn, pdg) # caller/callee/sig/doc/class
score = (H*I)/(H+I) * difficulty_penalty(H, I)
if score >= 0.08:
rationale, body = gemini_generate(mask(file, fn))
if judge(rationale, body, ground_truth) >= keep_threshold:
emit_fim_sample(prefix, suffix, rationale + body)
The corpus is 968 GitHub repos, decontaminated against SWE-bench, yielding ~400K samples and ~2.6B tokens. Mid-training runs one epoch on Qwen2.5-Coder-Instruct (7B, 14B) and Qwen3-8B, then standard agent post-training (R2E-Gym, SWE-Smith, or SWE-Lego) runs unchanged on top.
The prevailing move to make a base model into a better agent is more agent trajectories: scale synthetic issue-resolution data in post-training. This paper shows a different lever exists one stage earlier. The conditioning structure agents need (act, observe an externally-produced value, continue) is already sitting in every function call site on GitHub, and a function-granularity fill-in-the-middle objective concentrates that signal right before agent post-training. The load-bearing evidence isn’t the SWE-bench Verified lift, it’s that the same Python-only corpus also rescues non-coding tool-use benchmarks that agent post-training otherwise degrades.
The finding that makes the thesis credible: agent post-training silently degrades capabilities the base model already had, and mid-training largely undoes that damage even on tasks the corpus never saw. On the 14B model, R2E-Gym post-training alone drops LiveCodeBench by 13.1 points, Berkeley Function Calling Leaderboard by 7.4, and \u03c4-bench by 2.3 relative to the instruct base. Adding FIM mid-training before the same post-training restores +11.1 on LiveCodeBench, +2.4 on BFCL, and +3.9 on \u03c4-bench. The mid-training corpus contains zero tool-use data and only Python code, so the only way this transfer can happen is if the function-call structure it installs really is the same structure tool-use needs.
The headline in-domain numbers: +2.8 / +3.0 on SWE-Bench-Verified at 7B and 14B with R2E-Gym, +3.2 on Qwen3-8B with SWE-Lego, and +5.3 when swapping in SWE-Smith at 7B. On tasks whose gold patch spans 2 or more functions in one file (n=88), the gain jumps to +9.1 pp, more than 4\u00d7 the single-function bucket (+2.1 pp) — exactly the slice the isomorphism argument predicts should benefit most.
One ablation matters. Removing the Gemini rationale entirely still delivers roughly half the gain, and using the trained model’s own rationales recovers most of the rest. The recipe is not a hidden teacher-distillation pipeline; the function-aware masking is doing the real work.
Reach for this when you’re building a coding agent on an open-weight base and you’re about to run trajectory-based post-training on it. Instead of only scaling trajectories, spend a small budget mid-training the base on function-granularity FIM from decontaminated repos in your target language. You keep the LiveCodeBench-style single-shot coding ability the base had, you get a few points on SWE-Bench-style agent tasks, and the behavioral shift is toward iterate-and-verify: more edits per solved task, near-elimination of empty-patch failures (baseline emits <finish> with no diff; mid-trained agent keeps editing).
Artifacts: the authors promise the 968-repo corpus (~400K FIM samples), the selection pipeline, and mid-training checkpoints at GitHub. Repos are filtered to permissive licenses (>80% MIT/Apache/BSD) and decontaminated against SWE-Bench sources. Directly reusable if you’re on Python; the pipeline is language-agnostic in principle but only Python parsers are implemented.
The structure your agent needs to learn is often already in your pretraining data, just in a shape the pretraining objective hides. Before you scale more trajectories, ask whether a targeted self-supervised objective can surface the conditioning pattern you actually want.
•
Python only. Corpus, selection heuristics, and every in-domain benchmark are Python; transfer to Java, Rust, or C++ is untested. The pipeline also presupposes modular code with real function boundaries, so notebooks and monolithic scripts yield few eligible targets.
•
Cross-base evidence is thin. The one non-Qwen2.5-Coder result (Qwen3-8B) also swaps the post-training pipeline to SWE-Lego, so “transfers across model families” is a hope, not a demonstration.
•
Multi-file patches don’t benefit (~11.3% for both checkpoints). The isomorphism only pays off within a file because that’s where the masking operates; cross-file coordination needs a different objective.