When a paper or a vendor says “hallucinations down 30%,” that sentence hides a fork. Either a detector got better at flagging suspect outputs, or a generation policy changed so the model produces different text. These are separate interventions with separate failure modes, and each needs its own scorecard. A black-box detector study shows that the leading flagging method and its performance at different thresholds shift with the generating model and dataset; its authors recommend considering review capacity when selecting an operating point. A vision-language mitigation study shows that methods which lower hallucination metrics frequently do so by saying less, and that the gains do not reliably carry over to general capability. Neither result certifies the other.
Are you detecting errors or changing the answer?
To interpret a lower hallucination score, first identify what changed: the detector or the generated answer. A detector does not change a token of what the model writes; it assigns a score or label. Any downstream review or release policy is a separate choice not evaluated here. A decoding intervention instead reshapes the distribution the model samples from. Both can be described with the phrase “reduces hallucinations,” and they are not substitutable.
Start with the alarm. In the black-box detection work from From Tokens to Semantics, the system samples N responses to the same query and reads two kinds of evidence off them. Token signals come from API-exposed log-probabilities: how confident the model was per token, how that confidence moved across positions, how much responses varied. Semantic signals come from clustering the sampled responses by meaning; if the model says four mutually incompatible things, that disagreement is itself information. The paper’s three proposed methods combine these differently. TopK just aggregates token uncertainty across samples. Gated routes on the semantic-cluster count K: when K ≥ 2 it uses a Semantic entropy score, and when K = 1 — where empirical semantic entropy is zero and therefore useless — it falls back to a logistic-regression classifier over aggregated token features. Stacked skips the routing and feeds both feature families through PCA into an L2-regularised classifier so both influence every prediction. Gated and Stacked need labelled hallucination data to train.
Notice what this machine produces: a score, and then a threshold on that score. It is a sensor, not a fix.
Why the alarm’s threshold is not a property of the method
Here is where a single headline AUROC misleads. Across 26 model–dataset comparisons, the leading family kept moving. Supervised classifiers led four of seven GPT-4.1-mini comparisons but only three of seven on GPT-5.1, while unsupervised methods led or tied on six of seven GPT-5.4 comparisons. Semantic entropy computed via Von Neumann entropy rose from 0.468 to 0.750 AUROC on Financial Summaries purely by changing the generating model. The signal that carries information about hallucination is not fixed; it depends on which model produced the text.
The False-positive budget matters just as much. At strict 1–3% FPR, token-based or supervised combinations led on the datasets with clearer separation; at looser budgets, CoCoA overtook on AmbigQA and the leading family changed on HotpotQA and Cheque Generation too. And relaxing the budget doesn’t uniformly buy you anything: moving from 5% to 15% FPR substantially improved hallucination coverage on AA Omni Finance, SQuAD, and Cheque Generation, but barely moved Financial Summaries or Long-Text QA. That asymmetry is the operationally important part. On some data, raising the FPR budget from 5% to 15% catches many more real errors; on other data, it triples the false-alarm rate among good outputs while adding little hallucination coverage. For a fixed number of good outputs, the false-positive count also triples. Total review volume would additionally depend on how many genuine hallucinations are flagged and which flags a downstream policy sends for review.
Even the API details bite. Raising retained token candidates from one to three lifted TopK from 0.621 to 0.721 AUROC on AA Omni Finance — a dataset full of precise dates and amounts where knowing the runner-up token is diagnostic — and lifted average Stacked there from 0.634 to 0.741. The same change left average Stacked essentially flat on SQuAD, 0.767 to 0.768. Temperature preferences ran in opposite directions across datasets: average Stacked peaked at 0.700 on AmbigQA at temperature 0.5 and collapsed to 0.461 at 1.2, while HotpotQA did best at 1.2. The authors’ conclusion follows directly: choose response count, token-candidate count, temperature, and threshold jointly with your dataset, model, label availability, false-positive cost, and review capacity.
Several caveats keep this honest. Zero-shot transfer for Gated and Stacked was never evaluated, so their strong showing assumes you have target-domain labels. Long-Text QA has only 30 questions. On every dataset, the leading method’s confidence interval overlapped at least one competitor, so “which method won” is a weaker statement than the numbers suggest. And token methods need exposed log-probabilities at all — GPT-5.4 returned only one candidate per token, cutting off the alternative-token information that helped elsewhere.
What happens when you change the answer instead
Now the other intervention. Does Playing it Safe Count as Faithfulness? evaluates six training-free inference-time mitigation methods — contrastive decoding (VCD, M3ID), attention calibration (AGLA, CAAC), and hidden-state modification (CEI, AFTER) — on three 7B vision-language models, across 54 configurations.
The design choice that makes it useful is paired metrics. On CHAIR, object-existence error rates are reported alongside Object Recall. On AMBER, hallucination rates sit next to Cover, the fraction of ground-truth visual entities the caption actually mentions. Why pair them? Because there is a trivial way to score zero on object hallucination: describe almost nothing. A caption reading “a photo of a room” cannot name a nonexistent lamp. The authors’ framing is that a genuine improvement moves toward the upper-left quadrant — fewer errors and more visual detail — whereas the methods they test tend to slide along a risk-hedging diagonal. They are careful to call this an observed evaluation pattern rather than a mechanistic account of why each method behaves as it does.
Informativeness proxies could still be dismissed as benchmark artefacts, so the third leg is capability. MMStar is built by purging examples solvable from textual cues or shallow visual shortcuts, then scoring six categories: coarse perception, fine-grained perception, instance reasoning, logical reasoning, science and technology, and mathematics. The additional question is whether fewer hallucinations come at a cost to other visual tasks. Instead, of 18 fine-grained-perception configurations, only two yielded isolated gains above 1%; the large majority degraded. No method consistently improved MMStar over vanilla decoding across models or categories, and every tested method failed to improve macro averages on more than one of three architectures. Category-level gains came with offsetting losses: AFTER lifted coarse perception 7.2% on InstructBLIP while dropping mathematical reasoning 4.4%; AGLA gained 6.4% on mathematics for LLaVA-1.5 while shedding 10.4 percentage points on science and technology.
The scope is real and worth stating: three 7B models, author-default hyperparameters rather than per-model tuning, inference-time distribution-shaping only — no training-based mitigation, preference optimization, or retrieval grounding. Larger models may respond differently, and most effects were modest and inconsistent rather than catastrophic.
Put the papers side by side and the fork in “hallucinations down” becomes concrete. For a team considering review routing, the detector results motivate selecting the score and threshold on representative data, with review capacity in view — a good AUROC on someone else’s dataset is not an operating point on yours. The study evaluates the detector; it does not test a downstream review policy. The mitigation work tells you that when you change generation, faithfulness metrics move in ways that a recall or coverage metric can partly explain away, and a capability benchmark can contradict.
These remain separate objects measured on separate populations. No supplied result evaluates a detector feeding a mitigation policy, and nothing here shows that a text-model detection finding transfers to 7B vision-language decoding or vice versa. Each paper exposes a different way a single score can mislead. The distinction helps you decide what else to measure.
For a team evaluating hallucination work — your own or a vendor’s — I would insist on knowing first which intervention is being claimed, then hold it to the matching report.
For an alarm. Ask for hallucination coverage at explicit false-positive budgets on data representative of your traffic, not aggregate ranking across public benchmarks. Ask which generating model produced the evaluated text, since the informative signal shifted with the model. Ask whether the method needs labels, and whether those labels came from your domain. If you plan to route flagged outputs to reviewers, use review capacity and false-positive cost to inform the operating-point choice. The honest limit: these are separation measurements, and total review volume includes both false-positive and true-positive flags. Whether that downstream routing policy improves outcomes for users needs a separate evaluation.
For an answer policy. Never accept a hallucination number alone. Require it paired with an informativeness endpoint — Object Recall on CHAIR, Cover on AMBER — so you can see whether errors fell because the model got more accurate or because it got quieter. Then require a capability check, overall and per category, because a method can look neutral on a macro average while systematically eroding fine-grained perception. These endpoints are benchmark-specific proxies, not universal measures of utility, but they are the ones with evidence behind them here.
What you gain from separating the two is diagnostic precision. When a hallucination rate drops, you now have a short list of explanations to rule out before believing the flattering one: the threshold moved, the evaluation population shifted, the model stopped committing to details, or capability was traded away in a category nobody plotted. A lower number stops being an answer and becomes a question about which of those happened.