Personalized LLMs fabricate roughly 42% of the claims they make about users when given only 3 facts, and the models that self-report the least over-inference are actually the worst offenders by external audit.
You’re building a chatbot with persistent memory. A user tells it three things: their job, a weekend hobby, and a story about their cat. By the next session your system “knows” they live in a minimalist apartment, are probably single, and like indie rock. None of that was ever said. Systems like Mem0 and MemGPT assume the model can tell what it actually knows from what it’s guessing. This paper says that assumption is broken across every frontier model tested, and worse, the natural fix (asking the model to audit itself before writing to memory) fails in a specific and misleading way when used to compare models.
The authors build MirageBench, a benchmark that deliberately starves models of evidence and then watches them personalize anyway. Each of 150 personas reveals exactly 3 first-person facts, but has 15 ground-truth attributes total, so any personalized response has to reach beyond what was said. Personas are balanced into stereotypical, counter-stereotypical, and neutral thirds to separate stereotype-driven guessing from pure invention. Models then perform 6 tasks along a groundability gradient, from concrete (“pick a $100 gift”) to imaginative (“describe my apartment”).
Every claim in every response gets classified into a four-way faithfulness taxonomy: Grounded (restates a fact), Reasonable (one common-sense step out), Stereotype (substitutes group priors for evidence), or Fabricated (no basis at all). The last two together define the over-inference rate. An independent judge model does the labeling, validated against a blind human annotator on 400 claims with Cohen’s kappa of 0.863 four-class and 0.900 binary.
for persona in personas: # 3 revealed facts, 15 hidden attributes
for task in six_tasks:
response = model.generate(task, revealed_facts)
self_audit = model.classify_own_claims(response) # what it admits
judge_labels = judge.classify(response, revealed_facts) # ground truth
oi_rate = (judge.stereotype + judge.fabricated) / judge.total
# Compare self_audit OI vs judge OI across 12 models
A separate Accum pilot runs 8 conversation rounds per persona and tracks whether inferred attributes are ever revised when later turns contradict them.
The prevailing intuition is that models with better calibration and higher self-reported honesty are the safer choice for memory systems. This paper shows the opposite. The models that most confidently report low over-inference are the ones that fabricate the most under external audit; self-reported safety is anti-correlated with actual safety across models, even though the same self-audit still works fine within a single model. The load-bearing evidence is the Spearman rank correlation of ρ = −0.60 between self-reported and judge-measured over-inference across 12 models, not the headline 42% fabrication rate.
The finding that makes the thesis true: at the model-selection level, self-audit inverts. Qwen3-8B reports the lowest over-inference (13.0%) but the judge flags 48.7% of its claims as over-inferred, the worst in the study. Kimi-K2.5 reports the highest (58.2%) but sits mid-pack under the judge at 43.1%. Spearman ρ = −0.60 (p = 0.044, though the paper flags the bootstrap CI as wide, [−0.90, +0.06], with n = 12). The authors read this as a metacognitive calibration asymmetry: “strict” self-auditors (Claude, GLM, Kimi) flag their own inferences readily and generate more carefully; “lenient” auditors (GPT-4o-mini, Qwen3-8B) label almost everything as reasonable and generate with less inhibition.
Secondary findings that support this:
•
Over-inference is universal. Every one of 12 models over-infers 35%–49% of claims, cross-model mean 41.6%. Only 24%–31% of personalized content is actually grounded in what the user said.
•
Fabrication dominates stereotyping. Mean 31.1% pure fabrication vs. 10.5% stereotype-based. Models mostly invent; they don’t just default to demographics.
•
Imagination amplifies fabrication. Task over-inference ranges from 27% (gift recommendation, can lean on stated hobbies) to 59% (apartment description, nothing to lean on). Recommendation letters hit 40.4% fabrication driven by genre pressure: the format demands specific achievements the evidence can’t supply.
•
Within a single model, self-audit still works. AUROC for ranking a given model’s own claims runs 0.58–0.83, with 9 of 12 models above 0.75. Useful for internal filtering, misleading for cross-model comparison.
•
Silent memory pollution. In the 8-round Accum pilot, 9 of 12 models add 5–15 new inferred attributes per round with R² > 0.90 linear growth. The fastest accumulators (GPT-5.5, GLM-5.1, Claude-Opus-4-6) revise prior inferences only 0.4–5% of the time. Even when later user turns contradict an earlier inference, it stays. The authors caveat this as a pilot on 2 personas with a retention-biased prompt.
•
Counter-stereotypical personas act as a real stress test. Stereotypical personas: 44.8% OI. Counter-stereotypical: 37.0%. The 7.8 pp gap holds in every one of the 12 models individually.
Reach for this when you’re shipping an assistant with persistent memory and considering “just ask the model if it’s confident before writing to memory” as your safety filter. The paper says: fine as a within-model ranking signal for internal filtering with a model-specific threshold, but do not use self-reported grounding to pick which model to deploy, because the ranking is reversed from what you want. The concrete alternative the authors argue for is provenance tagging on stored attributes (stated vs. inferred-with-link vs. generated-without-evidence) and external verification rather than trusting self-report.
The authors say they will release MirageBench in full, though no repository link appears in the text provided. The benchmark itself (150 personas, 6 task prompts, judge/probe/audit/accum prompt templates) is spelled out in the appendix and reproducible from the paper alone. The four-way taxonomy plus the judge protocol is directly usable if you want to audit your own personalization pipeline on your own personas.
A model’s confidence that it isn’t fabricating is not evidence that it isn’t fabricating, especially when you’re comparing models. Use self-audit as an internal ranking signal with a per-model threshold; use an external judge, or provenance tracking, when the question is which system to trust.
•
The self-monitoring inversion is exploratory: n = 12 models, correlated families, bootstrap CI crosses zero. Treat the direction as the finding, not the coefficient.
•
The over-inference rates depend on a single judge (Claude-Opus-4-7). Human validation on 400 claims is strong (κ = 0.863), but a second judge or additional annotators could shift absolute numbers. The paper’s relational findings (rankings, gradients, cross-model contrasts) should transfer; absolute rates may not.
•
The accumulation result is a 2-persona pilot with a memory prompt that explicitly tells the model to retain rather than prune. The cross-model contrast (near-zero vs. 70%+ revision) is informative; the absolute claim that memory grows is baked into the setup.
•
No mitigations are evaluated. The paper diagnoses the problem and argues for provenance and external audit, but doesn’t measure whether either actually reduces downstream harm or user dissatisfaction.