ElephantBench probes whether a closed-book LLM can recall all verified accounts of a long-tail fact, not just one. Even the strongest model returns both accounts on only 52.4% of questions, almost always omitting the less-exposed one.
You’re shipping an assistant that answers factual questions without retrieval, or you’re building a knowledge base where the model’s parametric memory is the fallback when Retrieval-Augmented Generation misses. Today you evaluate it with a benchmark that scores one canonical answer per question, so a model that confidently returns “Mother Teresa was born August 26, 1910” gets full marks even though authoritative sources also record August 27. That single-answer framing hides a systematic failure mode: on facts where reputable sources genuinely disagree, models tend to memorize the dominant account and drop the rest. Prior long-tail benchmarks like Long-tail QA benchmarks (PopQA-style) test whether rare facts are recalled at all; knowledge-conflict benchmarks like WikiContradict hand the model conflicting passages at inference time. Neither asks whether the model’s own weights preserve the disagreement.
The authors want long-tail facts that models can’t easily paper over from Wikipedia repetition. So they invert the usual pretraining pipeline: run the DCLM fastText quality classifier quality classifier on a web corpus and keep the documents it rejects, calling this partition $D_{low}$. Rare facts survive in there because standard filtering wasn’t inclined to upsample them.
Next, build a document graph over $D_{low}$ where edges mean “these two documents discuss the same subject-attribute pair” and are labeled either support (accounts agree) or conflict (accounts disagree). Comparing every document pair with an LLM would be quadratic, so they narrow candidates two ways: cluster documents by knowledge-point label from the SuperGPQA taxonomy taxonomy, and use Named-entity recognition to link documents across clusters that share an entity mention. Only those candidate pairs go to an LLM edge classifier. The paper reports this cuts pair comparisons by at least 15x.
for pair in candidate_pairs: # from KP clusters + NER index
label = llm_classify(doc_i, doc_j) # none|support|conflict
if label == "conflict":
subgraph = expand(pair, support_neighbors)
q, answers = llm_generate_qa(subgraph)
if web_agent_verifies(answers) and human_ok(q, answers):
keep(q, answers)
Each retained item has $k \geq 2$ verified answers, each traceable to a source document and independently corroborated by a web agent (using Wikipedia and similar authorities), then human-reviewed. At evaluation time, the model sees only the question. A judge LLM grades the free-form response as complete recall (all verified answers, no wrong ones), partial recall (some but not all), or failed recall (none or wrong). The headline metric $K = C/(C+P)$ isolates completeness among responses that at least got something right.
The prevailing assumption is that if a strong LLM knows a fact, evaluating on a single canonical answer is fine. This paper shows the opposite. Parametric memory is systematically lopsided: models remember the dominant account of a contested fact and quietly drop the minority account, and single-answer QA can’t see this. The load-bearing evidence is that on the top models, failed recall is tiny (~2%) but partial recall is nearly as large as complete recall, so the missing capability is completeness, not retrieval.
•
The dominant failure is partial recall, not failure to recall. For the three strongest models, failed recall sits at 2.19% to 2.65%, but complete recall stalls near 50%: Kimi-K3 at 52.38%, Gemini-3.1-Pro at 50.37%, GPT-5.5 at 50.18%. Partial recall runs 45-47% for these same models.
•
Scaling helps recall-anything more than recall-completely. Within the Qwen3.5 family, going from 2B to 397B raises complete recall from 1.65% to 32.27% and drops failed recall from 81.35% to 8.50%, but partial recall rises from 17.00% to 59.23%. Bigger models find the fact; they still miss the minority account.
•
Reasoning is inconsistent. Enabling reasoning gives GPT-5.6-Sol a +13.99 pp lift in complete recall, but on Qwen3.5 at 2B and 4B it actually reduces complete recall by 0.64 and 0.37 pp: smaller models can talk themselves into treating the dominant account as consensus.
•
Exposure asymmetry maps cleanly to the failure mode. A one-standard-deviation increase in majority-side supporting documents is associated with +14.18 pp partial recall and -10.17 pp failed recall (i.e., you at least get one answer). The same increase on the minority side gives +15.13 pp complete recall and -15.41 pp partial recall. Minority-side exposure is what buys completeness.
•
Oracle ceiling. Greedily unioning the best response across all 32 models lifts complete recall from 52.4% to 81.2%, but 18.8% of questions remain partial for every model. That residual is a shared blind spot, not a model-selection problem.
•
Domain gradient: complete recall is 38.7% for people/organizations/events but only 6.2% for consumer products and services, where prices and fine-grained numbers dominate.
Reach for this when you’re evaluating a closed-book factual assistant, especially in domains where sources legitimately disagree (prices, casualty counts, dates, jurisdictional scope). The standard eval will call your model correct as long as it returns one canonical answer. ElephantBench’s three-way grading (complete/partial/failed) plus the $K$ metric surfaces the completeness failure that a single-answer accuracy number hides. The exposure regression is also directly actionable for data curation: if you care about a model reporting minority accounts, you need to increase minority-side document exposure specifically, not just overall fact frequency.
The artifacts are released under Apache 2.0: 1,094 QA pairs with verified answer sets and source webpage texts on Hugging Face, pipeline code on GitHub, and a project page. Appendix J also shows that conditional perplexity over the verified answer spans tracks the graded outcomes closely, so you can approximate the eval cheaply on open-weight models without spinning up a judge.
Single-answer accuracy hides which account your model forgot. When facts have multiple verified versions, grade completeness separately from correctness, and remember that scaling and reasoning both surface the dominant account faster than they surface the minority one.
•
Exposure counts come from one public corpus (RePro), not the actual pretraining mixture of each evaluated model. The paper is explicit that this is an observational proxy, so the exposure-asymmetry claim is an association rather than a causal statement about any given model’s training data.
•
“Verified accounts” means supported by authoritative public web sources at review time. Some minority accounts are historical estimates or corrections that a well-calibrated model might reasonably down-weight; the benchmark treats them as equally required for complete recall.
•
The judge is itself an LLM (GPT-5.6-Sol). Human agreement is 90-93% with Cohen’s kappa 0.82-0.88, which is solid, but ranking claims among tightly-clustered top models rest on that judge being unbiased across model families.