How to evaluate LLMs when headline scores hide the failureLLM evaluation is a measurement-design problem before it is a leaderboard problem. A useful benchmark must expose the decisions that matter in the intended workflow: whether an agent notices missing requirements, whether a detector fits the available review budget, whether a model uses the evidence it was given, or whether a system recovers after its first plan fails. Aggregate success can conceal each mechanism.The readings below organize evaluation around four moves: reproduce the conditions of use, separate distinct failure modes, intervene on suspected shortcuts, and replace saturated scores with targeted challenges. Use them to choose tests that diagnose why a system succeeds or fails—not merely to produce another score.
Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive OptimizationStart here for a concrete account of evaluating pre-formulation clarification. It explains hidden-slot construction, stopping and silent-assumption metrics, the gap ledger, ablations, and the interaction-cost trade-off.OR-Clarify uses 100 optimization cases, bounded interaction, and a simulated user; the strongest InterOPT result is in the choice-based protocol.
What are you trying to solve?
Test the decision before the outputSome failures happen before an artifact is produced. An optimization assistant must decide whether the specification is complete; a coding agent must investigate records, communicate with a client, and explore designs before shipping. Evaluate these intermediate decisions under bounded resources, then connect them to end-to-end outcomes.
Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive OptimizationStart here for a concrete account of evaluating pre-formulation clarification. It explains hidden-slot construction, stopping and silent-assumption metrics, the gap ledger, ablations, and the interaction-cost trade-off.OR-Clarify uses 100 optimization cases, bounded interaction, and a simulated user; the strongest InterOPT result is in the choice-based protocol.
$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent ConstructionRead this to see how an agent-construction benchmark preserves client, data, API, codebase, model, and cost constraints through delivery. Its outcomes diagnose development failures within simulated customer-service engagements.The evaluation covers 53 customer-service construction tasks in four domains and scores delivered agents against held-out simulated users.
How can an agent be tested on whether it asks for missing requirements rather than silently choosing defaults?Hide formulation-critical facts, let a simulated user reveal only what is asked, and score exact recovery, premature stopping, silent assumptions, turns, and question count. OR-Clarify makes readiness measurable; its published overview also shows that a persistent gap ledger helped most in a choice-based protocol but increased interaction cost.Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
What should an end-to-end coding-agent benchmark include beyond unit tests?Include the business records, client-held requirements, production API, inherited code, and model or serving limits that constrain the delivered system. Then evaluate the result against held-out simulated users. This can reveal failures in investigation, client communication, architectural experimentation, and cost choices that isolated coding tasks miss.$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction
Match hallucination detection to review capacityA detector is useful only at an operating point the workflow can afford. Without trusted context, uncertainty signals can disagree; with evidence, the system must connect an unsupported claim to the exact text a reviewer should inspect. Treat false-positive budget, calibration, granularity, and inference cost as separate design choices.
From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMsUse this to choose black-box hallucination signals under a fixed false-alarm budget. It compares semantic, token-level, hybrid, and supervised approaches and emphasizes calibration rather than a universally best detector.Results span seven text and multimodal benchmarks, four models, and false-positive-rate budgets from 1% to 15%.
Enoki: Efficient Multi-Level Hallucination DetectionRead this when detection must support both claim verification and reviewer-facing span localization. Its shared extraction representation avoids a separate alignment pipeline.Evidence covers claim verification and span/entity localization using LLM-, encoder-, and rule-based extraction regimes.
How can a black-box API flag possible hallucinations when no trusted reference is available?Compare semantic disagreement across sampled meanings with uncertainty derived from token log-probabilities. Their errors differ: semantic entropy can collapse when samples share one meaning, while token signals can miss confidently repeated errors. Evaluate combinations at explicit false-positive-rate budgets and recalibrate for the model, dataset, and generation setup.From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs
How can a detector verify claims and also identify the unsupported span?Use text-anchored relational facts as a shared representation: extract facts, verify them against evidence, and project unsupported facts back to source spans. Enoki is useful when reviewer-facing localization and claim-level decisions are both required without a separate claim-to-span alignment stage.Enoki: Efficient Multi-Level Hallucination Detection
Intervene on suspected shortcutsCorrect answers do not show which evidence caused them. Remove or replace the intended input, construct counterfactuals that hold wording nearly fixed, and intervene on internal states when the model is open-weight. These tests distinguish correlation, representation, and causal use.
Tracing Audio Grounding and Answer Selection in Audio LLMsUse this as a compact example of grounding evaluation by intervention rather than accuracy alone. Audio replacement and layer tracing test whether acoustic evidence affects answer selection.The findings concern the studied Audio LLMs and answer-selection tasks, comparing pretrained and trained models.
Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language ModelRead this for a progression from decodability to confound checks and causal intervention. It shows why absolute hidden-state organization can mislead and why controlled state changes are stronger evidence.Tests use three open-weight Gemma 4 models, materials-science prompts, a 60-law counterfactual benchmark, and matched interventions.
How can an audio benchmark reveal reliance on language priors?Replace the audio with silence or unrelated audio and measure how predictions and answer-choice representations change. A model using acoustic evidence should be sensitive to this intervention. Layer analysis can then locate where acoustic information enters representations and where training increases its influence on the final prediction.Tracing Audio Grounding and Answer Selection in Audio LLMs
Why is hidden-state decodability insufficient evidence that a model uses a scientific mechanism?A readable state pattern may reflect an answer-correlated shortcut such as numerical comparison. Stronger evidence comes from matched counterfactual prompts and causal interventions: reverse only the physical input direction, test whether state movement follows the supplied law, and check whether state patches shift constrained answers in the predicted direction.Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Model
Turn saturated success scores into targeted stress testsWhen broad metrics approach saturation, preserve the failure conditions they average away. Collect concrete model-breaking examples, contrast nearly identical visual inputs, or begin from failed executions. Pair these cases with checks that distinguish the capability or recovery step being tested rather than collapsing everything into one success rate.
Last Translation BenchmarkUse this for benchmark design centered on model-breaking examples and example-specific verification rules. It offers a way to keep evaluation challenging and actionable as standard translation tests saturate.LTBv1 is a live, peer-reviewed multimodal translation benchmark with handcrafted checks for concrete failures.
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation ModelsRead this when ideal-start task success has become a weak proxy for embodied reliability. It organizes real execution failures into distinct recovery levels and reasoning demands.LIBERO-Recover contains more than 1,000 scenarios across action retry, action adaptation, object-state recovery, and environmental recovery.
VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference IdentificationRead this for a targeted diagnostic of fine-grained visual comparison. Its paired images, hard negatives, no-difference distractor, and category-level results expose failures that broad single-image evaluations can miss.VDiff-Bench has 1,756 four-choice questions across ten change categories and evaluates 11 open- and closed-source MLLMs.
What can a live benchmark provide when aggregate translation metrics stop revealing progress?Collect human-authored, peer-reviewed examples across the modalities a translation system must handle, then pair each example with handcrafted verification rules for its concrete failure cases. Accepting new examples can keep the test challenging, while specific rules make failures more reproducible and actionable than an undifferentiated quality score.Last Translation Benchmark
How can an embodied-agent benchmark test what happens after execution goes wrong?Start from recorded execution failures and separate recovery demands such as retrying an action, adapting it, restoring object state, and repairing the environment. LIBERO-Recover uses more than 1,000 scenarios to test these levels alongside spatial, object-structure, interaction, and topological reasoning.LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
How can a benchmark expose visual comparison failures hidden by general vision scores?Present paired images with one precise change, hard negative descriptions, and a no-difference option. Break results out by change type rather than relying on one score: VDiff-Bench found especially brittle behavior on subtle noise and texture changes, showing why semantic and low-level comparisons should be diagnosed separately.VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification