ScienceIDE turns scientific code repositories into agent-runnable environments with expert-calibrated numerical checks, so the same environment yields graded tasks for evaluation, Supervised Fine-Tuning, and Reinforcement Learning. Even leading agents leave ~one-third of hard scientific tasks unsolved under a one-hour budget.
A scientific codebase (say, an Magnetohydrodynamics solver in Fortran) encodes decades of physics as executable numerical models. You’d like an agent to repair, extend, or accelerate it. The trouble: these repos have finicky build toolchains, physics conventions the code doesn’t spell out, and correctness that depends on numerical tolerances (does the simulated plasma conserve energy to within some bound?), not on assert result == expected. The authors call this the scientific experience bottleneck: papers and repos exist, but they don’t automatically become environments where an agent can act, get scored, and learn.
Existing scientific coding benchmarks like SciCode or SWE-bench Science evaluate agents on fixed task sets. They don’t give you infrastructure for producing new tasks or verifying new agents’ outputs against physics-grade criteria. ScienceIDE is trying to be that infrastructure, not just another benchmark.
The unit of construction is a module: a coherent scientific responsibility inside a pinned version of a real scientific codebase (Athena++, MITgcm, PLUTO, LAPS, PHANTOM, and 22 others). A domain expert defines the module’s boundary and what “scientifically equivalent output” means. An AI agent handles the grunt work of building the code, running the shipped tests, and drafting the wrapper.
The key primitive is a check: fixed inputs, graded outputs, and a pass policy. Two policy shapes are allowed. A pointwise policy compares graded values against a reference with an absolute-plus-relative tolerance band, written plainly as “the answer must be within tolerance a + ρ·|reference|.” An invariants policy is used when run-to-run variation (random draws, chaotic flows) would make pointwise comparison meaningless; it checks conserved quantities, moments, or distributions instead. Every check carries a plain-language warrant explaining what it distinguishes and why a valid implementation can pass it. Tolerances are calibrated by running nominal and perturbed initial conditions and, where possible, an alternate legitimate build of the same source.
Once checks are fixed, a factory generates tasks against that module: inject a defect (repair), delete a routine (implementation), speed up a hot path (acceleration), and so on. A candidate task is only admitted if a known-good “witness” solution passes the checks and the defective baseline fails them, with headroom in between. Repair tasks are graded on r_repair = max(0, (r − f)/(1 − f)), meaning: how much of the gap between the broken baseline f and a perfect score did you close.
for module in expert_approved_modules(repo):
checks = calibrate_checks(module) # pointwise or invariants + warrant
for candidate in factory.propose(module):
if witness_passes(candidate, checks) and baseline_fails(candidate, checks):
tasks.append(candidate)
for task in tasks:
episode = agent.run(task) # edit, execute, submit
reward = grade(episode, checks) # r_repair in [0, 1]
The same episode interface feeds three consumers: evaluation, Supervised Fine-Tuning on verified trajectories, and Reinforcement Learning with the verifier’s numerical reward.
Evaluation on ScienceIDE-Hard (85 tasks, 18 environments). Fifteen agents from eight providers ran through Codex, Claude Code, or Gemini CLI with a one-hour budget per task. Fable 5.1 led at 67.1% strict success, Opus 5 at 64.6%, Astra at 63.1%; eleven of the fifteen scored under 40%. The authors caution against reading this as a stable ranking: Fable has only single-attempt measurements, and Opus/Astra’s repeat intervals overlap.
Cost and time don’t track accuracy. Across the fifteen agents, the descriptive Spearman correlations of success with runtime and with output token volume are −0.22 and 0.01. DeepSeek V4.1 Flash burned 172.4k output tokens per task for 36.0% success; Astra hit 63.1% at 13.9k tokens and $3.56 per task.
Trajectory review of Fable and Astra failures. Reference-convention mismatches (code runs, but uses the wrong precision, state-retention rule, or unit convention the surrounding code silently expects) account for 71.4% of Fable’s and 64.9% of Astra’s task-balanced failures. Nine failed attempts across three PLUTO cooling tasks all made the same natural-log to base-10 edit in the wrong routine. A local passing test often just showed the agent’s edit was self-consistent, not that it addressed the assigned defect.
Supervised Fine-Tuning on GPT-5.6-sol trajectories (4,567 segments from 564 tasks, LoRA on all linear layers, three epochs) improved repair reward on held-out ScienceIDE environments (e.g., LAPS from 0.3125 to 0.5000 for the 9B model) and lifted scores on 15 model-benchmark comparisons by ≥3 percentage points, including BIG-Bench Hard (BBH) Word Sorting (9B: +33.6 pts), HumanEvalFix JavaScript (4B: +10.98 pts), and GSM8K. Not uniform improvement, and one HumanEvalFix Python confirmation showed a decline.
Reinforcement Learning with the verifier as reward on Qwen3.5-4B, two environments (LAPS, MITgcm-biogeo). After 30 steps, held-out reward roughly doubled: LAPS 0.357 → 0.857, MITgcm-biogeo 0.286 → 0.571. The critical finding: budget-truncated trajectories must be masked from the loss but kept in the group-mean baseline (following Dr. GRPO and DAPO design choices). Without this, the policy learns to shorten trajectories rather than solve tasks, and reward collapses below its starting point.
•
If you’re building agent evals for a scientific or engineering codebase, the check-calibration recipe is directly transferable: pair every numerical check with a warrant, use pointwise tolerance where sensitivity fits a physics-meaningful band, fall back to invariants when it doesn’t, and calibrate the band using a perturbed initial condition and an alternate build. Skipping the warrant is how you end up with checks that pass wrong answers.
•
If you’re running Reinforcement Learning on long, outcome-only reward with a chance of budget timeout, the truncation-mask lesson probably generalizes: keep truncated episodes in the advantage baseline, drop them from the token loss, and turn off length penalties. The paper shows the failure mode (reward collapse, trajectories shrinking 3×) concretely.
•
If you want to reproduce or extend the SFT/RL results, code, models, and the ScienceIDE-Hard task set are released via the project page, GitHub, and Hugging Face. The RL evidence is on one 4B model in two environments, so treat any inference beyond that as worth testing rather than established.
•
If you’re just picking an agent for scientific code work, the paper’s own framing is the right one: leaderboard order is unstable, and the union of tasks solved by all 15 agents (72 of 85) far exceeds any single agent’s solved set. Complementarity across models is real, though realizing it would need a routing or verification policy the paper does not build.
•
Tasks are overwhelmingly repair (2,515) and implementation (295); acceleration has only two tasks. Claims about “scientific agents” here mean “agents that fix or reconstruct code in computational physics and geoscience repos,” not open-ended discovery.
•
The authors document historical contamination: earlier runs allowed container-side fetches (455 upstream-directed commands, 21 confirmed successful fetches) and provider-side tool use (Gemini received substantive tool content on 71 of 85 tasks; one trial pulled an exact upstream file). Reruns with allowlists closed the applicable channels, but the paper explicitly does not claim pretrained models never saw public upstream code.
•
Supervised Fine-Tuning gains are on selected public benchmarks with unadjusted paired-bootstrap intervals and no multiple-comparison correction. Reinforcement Learning gains are within the training environments on tasks with L1 hints (file, line, edit class provided). Neither establishes transfer to unseen codebases.
•
Verification is a modeling choice. Passing the checks means agreeing with chosen observables under calibrated tolerances, not correctness in general; legitimate alternative implementations can disagree with the reference.