Models that write, edit, and reason about source code — from autocomplete to autonomous coding agents, plus evaluation suites and execution-based training.
Code Generation and Program Synthesis: From Plausible Output to Working SystemsCode generation now spans more than autocomplete: models repair repositories, construct persistent workflows, emit interactive applications, and build other agents. The hard part is preserving the constraints that make an artifact useful—executable behavior, repository intent, editable structure, deployment budgets, and alignment with the user’s actual requirements.Use this guide by matching the failure mode to the reading. Start with executable verification when an artifact can look correct while doing nothing; move to repository context and edit fidelity when the risk is an unsafe patch; and use end-to-end environments when local tests hide failures in requirements gathering, integration, or long-horizon behavior.
CogEvol: Towards Efficient and Reliable Learning Environment GenerationStart here for the clearest mechanism-level account of why visual grading fails for interactive code and how executable reward can catch dead controls and reward hacking.The verifier covers slides and self-contained interactive HTML; it excludes some gestures and typed-format controls.
What are you trying to solve?
Verify behavior, not just appearanceGenerated artifacts can pass superficial review while failing at the interaction or state-transition level. Executable probes expose this gap by driving controls and observing effects. For structured systems such as data pipelines, constrain generation to typed, incremental changes against live platform state so the result remains a native, editable artifact rather than a detached script.
CogEvol: Towards Efficient and Reliable Learning Environment GenerationStart here for the clearest mechanism-level account of why visual grading fails for interactive code and how executable reward can catch dead controls and reward hacking.The verifier covers slides and self-contained interactive HTML; it excludes some gestures and typed-format controls.
How can a generator detect pages that look correct but have dead controls?Exercise the artifact. CogEvol’s verifier loads the page, records baseline animation, drives supported controls, and checks for new DOM or Canvas behavior. A hard-fail gate rejects pages that render but never change state. This matters because screenshot-only reward produced visually convincing yet unplayable games.CogEvol: Towards Efficient and Reliable Learning Environment Generation
How can an agent produce an editable workflow instead of a one-off script?Make the platform’s artifact model the action space. DataFlow-Harness exposes the live operator registry and pipeline state, then has the agent apply typed mutations to a DAG—a directed acyclic graph—while keeping conversational and visual editing synchronized.DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines
Repair evolving repositories without losing intentRepository repair combines three problems: finding the right code as it changes, limiting the patch to what the bug requires, and applying domain knowledge without becoming anchored to a misleading explanation. Check context validity, functional correctness, and edit fidelity separately rather than collapsing them into one success score.
CodeNib: A Multi-View Data System for Serving Repository Context to Coding AgentsUseful for repository-context infrastructure that survives edits. It distinguishes lexical, dense, and structural views and shows why reuse must be validated by operation rather than inferred from update success.Static definitions matched live-server results on 86% of requests; reuse accuracy varied by view and operation.
When Models Edit Too Much: On the Fidelity of Minimal Code EditsUse this to treat minimality as a measurable repair objective rather than a stylistic preference. It separates passing tests from excess edits and added cognitive complexity.Known minimal patches come from AST-level corruptions of 400 BigCodeBench reference solutions.
SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?Read this for a failure taxonomy specific to scientific repository repair and for evidence that domain guidance can improve efficiency or induce anchoring depending on alignment.The benchmark contains 119 tasks from 98 repositories across 20 scientific domains.
How can an agent reuse repository context without trusting stale indexes?Maintain lexical, dense, and structural views behind a shared source-address contract, then validate reuse per operation. CodeNib shows why successful index updates are not enough: some reused views closely matched trusted routes, while symbol-level repair often diverged despite completing without errors.CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents
How can repair systems discourage unnecessary rewrites?Measure edit fidelity independently of test success. In controlled AST-corruption tasks, preservation instructions reduced excess edit distance and added cognitive complexity while improving Pass@1. The study also reports that reinforcement learning transferred edit fidelity better than supervised fine-tuning to unseen corruption patterns.When Models Edit Too Much: On the Fidelity of Minimal Code Edits
When can scientific guidance make a code repair worse?Guidance can narrow exploration when it matches the defect, but poorly aligned guidance can anchor the agent on the wrong mechanism. SWE-bench Science separates failures in knowledge, exploration, repair coverage, integration, and generalization instead of treating more domain context as automatically helpful.SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Test delivery loops and construct executable training tasksPatch-level tests can miss whether an agent discovers requirements, understands records, respects serving limits, or succeeds with users. Training has a parallel problem: instructions, workspaces, solutions, and verifiers must describe the same executable task. Reconstructed or synthesized environments make these dependencies reusable and testable.
$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent ConstructionUse this benchmark to think beyond patch completion toward full agent delivery under data, API, client, inherited-code, model, and serving-cost constraints.Evaluation covers 53 simulated customer-service engagements in four domains.
Terminal-Universe: Turning Agent Trajectories into Scalable Terminal EnvironmentsRead this when training is limited by scarce executable environments but terminal trajectories already exist. It reconstructs workspaces from traces and expands them into new and multi-round tasks.The environments were reconstructed from public terminal-agent trajectories.
Recursive Synthesis for Long-Horizon Terminal TasksUse this when you have verified terminal tasks and want to synthesize harder descendants while keeping the instruction, environment, solution, and verifier aligned through recursive validation.Reported evidence covers 15 synthesis rounds and downstream terminal-agent training.
FACET: Preserving Source Intent and Executable State in Terminal Task SynthesisRead this when source intent may be lost during task synthesis. FACET uses a shared executable container state to ground task artifacts and targeted repair to avoid regenerating valid components.The reported training evaluation centers on synthesized terminal tasks and Terminal-Bench 2.1.
What does end-to-end evaluation reveal that isolated coding tasks miss?It reveals failures in requirements discovery, data comprehension, architecture exploration, deployment constraints, and user-facing behavior. τ^τ-Bench has a coding agent deliver a customer-service agent from business records, client requirements, inherited code, a production API, and serving limits, then evaluates it with held-out simulated users.$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction
What should be checked before using a synthesized terminal task for training?Check that the initialized environment supports the instruction, the reference solution reaches the intended state, and the verifier recognizes that state. Fresh-sandbox validation in RST and shared executable-state grounding in FACET address different ways these artifacts drift apart; neither reduces validity to plausible task text.Recursive Synthesis for Long-Horizon Terminal TasksFACET: Preserving Source Intent and Executable State in Terminal Task Synthesis