AIM treats automated-research agents as a Bayesian-optimization-style search over research ideas rather than code, clustering candidate ideas, ranking them, auditing whether solver code actually implements the chosen idea, and reaching the strongest baseline’s score up to 3.1× faster.
Suppose you build an agent that automates ML or systems-optimization research: it proposes approaches, writes code, runs a benchmark, reads the score, and decides what to try next. Each try costs real GPU-hours, so you cannot afford to evaluate every plausible approach. Two design choices exist. You can search directly over code (mutate the current implementation and keep the better one), which is what AIDE and AlphaEvolve do. Or you can search over natural-language research ideas, pick which idea to try, and delegate the coding to a solver. The authors call these solution-driven and idea-driven search.
Idea-driven search sounds cleaner but introduces three problems the paper calls out: how do you organize a growing pool of ideas, how do you pick which to try next, and how do you make sure the code the solver wrote actually reflects the idea you asked for. Existing idea-driven systems punt on these. ScientistOne uses a beam-search-style elite set, AI-Scientist-v2 uses tree search with MCTS, DeepScientist uses a flat list with UCB. None explicitly check that the executed code matches the proposed idea.
AIM borrows the structure of Bayesian optimization: a surrogate that estimates promise and an acquisition step that picks what to try. Both are implemented as LLM calls rather than mathematical models.
The Agentic Surrogate does two things each round. Organize re-clusters the full idea pool into 2–5 semantic groups (for example, on a FlashAttention task, clusters like “SIMD vectorization” or “fast exp approximation”). Clustering is redone from scratch every iteration, so an idea can move groups as evidence accumulates. Estimate then produces an ordinal ranking of clusters and of ideas inside each cluster. The authors use rankings rather than numeric score predictions because LLMs are more reliable at relative comparisons than at calibrated scores (an ablation confirms this).
The Agentic Acquisition step picks ideas for parallel solver branches. Each branch gets a two-level action: at the cluster level, EXPLORE or EXPLOIT, and at the idea level, EXPLORE or EXPLOIT. So one branch might exploit the top cluster but explore a low-ranked idea inside it. After solvers run, an Expand step pushes new candidate ideas into the pool using four modes: refine a high-scorer, cross-pollinate two ideas, repair a failed one, or propose something genuinely new.
The Solution Auditor is the component that addresses the idea-solution integrity gap. For each completed run it checks four failure modes: trivial solution, task mismatch, idea mismatch, and reward hacking. If the only problem is idea mismatch (the code works and scores well but implements a different idea than requested), the auditor rewrites the idea text to match what the code actually did, then attributes the score to that rewritten idea. This stops the surrogate from learning the wrong lesson about which approach worked.
Finally, the Resource Planner decides how many parallel solver branches to launch each round, trading off breadth (more parallel tries) against frequent feedback (narrower rounds with more adaptive updates).
pool = initial_ideas() # ~10 seeds
while budget_remaining and time_remaining:
B = resource_planner(state, remaining) # branches this round
clusters = organize(task, pool, history) # LLM reclusters
ranks = estimate(clusters, history) # ordinal rankings
picks = dispatch(pool, clusters, ranks, B) # 2-level explore/exploit
results = parallel_solve(picks) # each branch runs code
audited = auditor(results) # drop or realign
history, memory = update(audited)
pool += expand(pool, history, memory) # refine/mix/repair/new
return best_scoring_solution(history)
AIM is evaluated on 10 tasks from the AutoLab benchmark across System Optimization (FlashAttention, Radix Sort, FFT, AES128-CTR, Z-order Range Scan), Model Development (Moving MNIST world model, Data Selection for IFEval), and CUDA kernels (Huffman decode, NTT (Number-Theoretic Transform) butterfly, ICP (Iterative Closest Point) correspondence). The backbone LLM is Gemini 3.1-Pro-Preview throughout. Scores are normalized so that the task’s starter code gets 0 and a hand-written reference solution gets 50.
•
Average on System Optimization: AIM 67.0% vs ScientistOne 65.4% vs AdaEvolve 63.3%. The paper frames this as +1.6 points over the strongest baseline.
•
Average on the long-horizon Model Development & CUDA tasks: AIM 55.8% vs ScientistOne 50.9%, framed as +4.9 points.
•
Wall-clock on FlashAttention: AIM hits its final 90.5% at 3.3 hours; ScientistOne needs 3.5 hours to reach only 86.2%. The paper summarizes this as reaching the strongest baseline’s best score up to 3.1× faster.
•
One task bucks the pattern: on Data Selection IFEval, the solution-driven AdaEvolve scores 82.7 versus AIM’s 61.8. The authors conjecture this task rewards fine-tuning one heuristic rather than exploring distinct strategies, which plays to evolutionary code-mutation.
•
Ablations on FlashAttention: removing the Agentic Surrogate drops score from 90.5% to 85.9% (the biggest hit), removing the Solution Auditor drops to 87.6%, removing the Resource Planner drops to 88.1%, and dropping the explicit explore/exploit actions drops to 89.2%.
•
Replacing the default Gemini Deep Solver with Claude Code as the per-branch coder still has AIM matching or beating ScientistOne on all five system tasks, suggesting the gain is from the idea-management scaffold, not a specific solver.
The paper also offers a theoretical argument: idea-driven search makes semantic coverage (number of distinct research directions actually tried) directly controllable, which matters most when a task has many plausible approaches but only a few are competitive. Convex-hull areas of solution embeddings support this empirically. FlashAttention shows a large coverage gap between idea-driven and solution-driven methods, while Radix Sort (where the algorithm is basically fixed) shows a small gap.
•
If you are building a research or coding agent that spends real compute per attempt, the single most portable idea here is the Solution Auditor. Before you attribute a score back to the hypothesis you asked for, verify the code actually implemented that hypothesis, and if not, rewrite the hypothesis to match the code. Without this, your meta-learner learns wrong lessons from your best-scoring runs. This can be built on top of any existing agent loop.
•
The ordinal-ranking finding is a reusable prior: when you ask an LLM to score candidates for selection, prefer “rank these” over “give each a 0–100 score”. AIM’s ablation shows direct score estimation underperformed ordinal ranking by about 1.2 points even inside their pipeline.
•
The paper’s framing of when breadth helps is a useful decision tool. If your task has a mostly-fixed algorithm and the gains come from low-level tuning (think kernel-level optimization of a known approach), a solution-driven evolutionary loop like AdaEvolve may be competitive or better. If your task has many qualitatively different approaches and you do not know which family wins, bias toward idea-level search with explicit cluster coverage.
•
Worth testing: swapping the solver inside AIM for your own coding agent. The Claude Code substitution result suggests the scaffold transfers, though the paper only shows this on five tasks.
•
The authors released a project page and provide pseudocode plus prompt templates in the appendix. Code release is not confirmed in the supplied text. •
The LLM-call overhead is significant. AIM uses roughly 90–630 million input tokens per task-run, often 2–3× the solution-driven baselines. The wall-clock win comes despite that, because fewer bad experiments get executed, but your dollar cost depends on token pricing versus compute pricing.
•
All experiments use one backbone (Gemini 3.1 Pro Preview). Whether weaker models can produce reliable ordinal rankings and audit judgments is not tested.
•
The theoretical analysis assumes faithful realization: a chosen idea can be implemented as code that reflects it. The Solution Auditor is the practical patch, but on hard tasks a solver may still fail to realize the idea at all, and the paper acknowledges this.
•
Idea-driven search lost clearly on Data Selection IFEval. The lesson is not “idea-driven always wins” but “idea-driven wins when the task’s gain comes from exploring different approaches rather than tuning one”.
•
Only three runs per configuration, which is standard for this cost regime but means small per-task differences (fractions of a point) should not be over-interpreted.