Φ-Bench measures whether frontier LLMs can engineer the software stack that runs LLMs themselves, spanning single-kernel completion up to whole-repo optimization, and finds the best model clears only about one-third of the maximum score.
Building and speeding up LLM training and serving infrastructure (think CUDA kernel work, KV cache managers, distributed schedulers) is where a lot of real GPU cost goes. If an LLM agent could do this work, a small team could keep up with a stack that today needs specialized systems engineers.
Existing evaluations don’t tell you whether that’s realistic. Benchmarks like KernelBench and TritonBench hand the model one function with a fixed signature and ask it to fill in the body. Repo-scale efforts like ISO-Bench still tell the model which component to optimize and where the bottleneck is. None of them test the workflow a real infrastructure engineer does: read an unfamiliar codebase, guess where the slow part is, patch several files, run tests, profile, and try again. Φ-Bench is built to test exactly that loop.
The authors collected 2,260 systems papers and 1,852 GitHub repos related to LLM infra, then used an LLM to sort them into a three-level taxonomy: 9 top-level areas (training, inference & serving, kernels, I/O, hardware & edge, etc.), 62 mid-level topics, and 410 fine-grained tags. That taxonomy tells them which topics the benchmark still needs to cover.
Tasks come in three shapes with widening scope:
•
KFC (Kernel Function Completion): one file, one function signature fixed, submit once. Tests whether the model can write a correct and fast primitive.
•
LHI (Long-Horizon Implementation): a feature request against a real repo, multiple files editable, up to 16 submissions. Tests navigating and modifying a codebase.
•
E2EO (End-to-End Optimization): only a workload and a system-level goal (e.g., minimize validation loss under a wall-clock and parameter budget on nanoGPT). The whole repo is editable. Tests bottleneck-finding and iterative optimization.
Tasks are built three ways: mining real pull requests and issues (repo state before the PR is the starting point, the merged code is the reference), an agent-driven pipeline that scans repos for good candidate sites and auto-generates test cases by tracking which code branches existing tests miss, and human experts curating problems from influential papers.
Scoring uses two metrics, both gated on the code actually building and passing correctness checks. For performance tasks, they run an AB-BA paired measurement against the unmodified baseline. Plain English: candidate must beat a reference solution’s speedup, and the reward grows logarithmically:
speedup s = median over pairs of (t_baseline / t_candidate)
s_ref = median speedup of the reference solution
if s <= s_ref: reward = 0
elif s >= s_ref ** 2: reward = 1
else: reward = ln(s / s_ref) / ln(s_ref) # in (0, 1)
For implementation tasks, reward is binary: all tests pass or zero. Two anti-cheat layers run on every trajectory, a rule-based scanner (blocks fetching the original PR, the upstream package, etc.) and a Proctor Agent that reads the trajectory for subtler tricks like hard-coding expected outputs.
The final benchmark is 85 tasks: 55 KFC, 20 LHI, 10 E2EO, spread across all nine top-level topics.
Eight frontier models were run with their strongest reasoning setting, on a single NVIDIA H20 with 8 CPU cores and 32 GiB RAM. Proprietary models used the vendor’s own agent scaffold (Codex CLI for the OpenAI model, Claude Code for everything else).
•
Overall ceiling is low. Claude Opus 5 tops the board at 36.53%, then Kimi K3 at 28.12% and Qwen3.8 Max at 27.73%. Even with the full repo and all public tests visible, no model gets close to solving the benchmark.
•
Coverage is uneven. Claude Opus 5 leads 5 of 9 categories, but the Hardware & Edge category tops out at 5.4% across every model, suggesting frontier LLMs have little grip on hardware-specific tuning.
•
Open-endedness hurts. Averaged over models, LHI scores are consistently below KFC scores. Interestingly, E2EO scores are often higher than LHI for the top models (Claude Opus 5 hits 62.94% on E2EO), likely because E2EO allows 16 submissions and a partial-credit performance metric, while LHI requires passing every test.
•
More reasoning budget is not monotonic. On 20 LHI tasks tested at multiple reasoning-effort levels, all three models tested were best at max effort, but intermediate levels were noisy. Kimi K3 lost roughly 45% of its score at the low setting, suggesting its performance leans hard on test-time compute.
•
Better models make more errors, not fewer. The stronger models log more runtime and CUDA errors because they push through more attempts. Weaker models bail out earlier or submit something simpler. Claude Opus 5’s error mix is unusual: fewer Python runtime errors, more CUDA execution errors, which the authors read as evidence it clears the boilerplate on the first try and spends its budget on the actual low-level work.
•
Case study on an E2EO nanoGPT task. Opus 5 runs cheap local experiments before spending a submission, controls for measurement noise and cache effects (for example, noticing a code change invalidated the Torch Inductor compile cache and re-running the comparison with matched caches), and keeps a coherent multi-round plan. Weaker models change several variables at once, treat single-run differences as signal, and attribute outcomes to the most recent edit. This is an observation from a handful of trajectories, not a controlled ablation.
•
If you’re picking an agent for real infra work today, the honest takeaway is that none of the tested models is close to doing this autonomously. Treat them as assistants inside a human-supervised loop, not as replacements. The Hardware & Edge collapse is a specific warning: don’t hand any of these models a device-specific tuning job unattended.
•
If you’re evaluating an LLM agent on code work harder than function completion, Φ-Bench’s design is worth borrowing even if you don’t use the tasks. The graduated KFC → LHI → E2EO structure, the AB-BA paired timing, and the two-layer proctor (rule-based plus agent-based) address failure modes (data leakage, timing noise, reward hacking) that simpler harnesses miss. Artifacts are on GitHub and Hugging Face.
•
If you’re training or scaffolding an agent for long-horizon coding, the case-study behaviors (cheap local sanity checks before an expensive submission, isolating one variable at a time, checking for confounders like compile-cache invalidation before believing a regression) are concrete habits worth prompting or fine-tuning toward. The paper observes these correlate with success but does not prove causation.
•
Iterative refinement helps for some models on E2EO but not others: worth measuring on your own task before assuming that giving an agent more rounds will pay off.
•
Sample sizes are small. 20 LHI and 10 E2EO tasks means per-category numbers rest on very few problems, especially for the harder formats. Treat category-level rankings as suggestive.
•
Scaffold is confounded with model. GPT 5.6 Sol runs under Codex; every other model runs under Claude Code. Some of the gap between systems reflects tool-use scaffold quality, not just the underlying LLM.
•
Hardware is fixed to a single H20 GPU. Kernel and hardware results won’t necessarily transfer to H100, MI300, or consumer cards.
•
Model naming. The paper evaluates models labeled Claude Opus 5, GPT 5.6 Sol, Qwen3.8 Max, Kimi K3, GLM 5.2, DeepSeek V4Pro, etc. These names are what the paper uses; readers should check whether they map to models they actually have access to.
•
The case-study behavioral claims (Opus 5 plans better, controls variables, attributes cautiously) come from qualitative reading of a handful of trajectories on one task. Suggestive, not a controlled experiment.