September 5, 2026. Most evaluation records name a model, a prompt, a temperature, and maybe a seed. Two papers in this roundup take separate aim at the same blind spot in that record: the cache. One asks whether an identical request replayed against a live server produces the identical token sequence. The other asks whether evidence stored in a compressed cache is still used faithfully when it is read back. Each identifies something a standard model-and-prompt log will not capture.
Why identical requests can produce different tokens
Start with the observation that motivates the first paper. You send the same request twice to the same server, same weights, same greedy decoding, same seed, same machine — and you get different tokens. The obvious suspects are batching, sampling, or hardware. The paper (Same Request, Different Answer) removes them all: greedy decoding at temperature zero, seed fixed at 42, requests issued serially at batch size one so no two share a forward pass, KV cache precision pinned to 16-bit floating point in every arm so it never covaries with the weight format. Then it runs each configuration twice in full. With caching disabled, the repeats were bit-identical across 800 episodes, which bounds any other nondeterminism source at 0.5 percent under those conditions. So whatever remains is about the cache.
The mechanism turns out to be a second caching layer most evaluations never touch. Beyond the per-request Prefix caching under study, llama.cpp maintains a host-memory prompt cache that stores whole conversation states and selects among them by longest-common-prefix similarity rather than exact identity, evicting oldest-first. It is on by default at 8192 MiB. Your request arrives, the similarity search picks some stored state that partially matches it, the engine reuses that state instead of recomputing, and the arithmetic path — which additions happen in which order — now depends on what the server processed earlier. Nothing about your request changed. What it inherited did.
Changing the prompt-cache setting shows how large the effect was in this setup. Holding everything else fixed including a fresh server per arm, repeated cache-enabled runs on Qwen2.5-7B at Q4_K_M diverged on 1 of 80 episodes with the layer disabled (1.2%, 95% CI 0.2–6.8) and 31 of 80 with it at its default (38.8%, CI 28.8–49.7). Interposing a complete cache-disabled pass between the two cache-enabled passes pushed divergence to 77.5% — but only while the layer was active; with it disabled the same manipulation changed nothing. That asymmetry is the argument. Execution order does not matter on its own; it matters because it rewrites the state the next pass inherits. Divergence was also concentrated early in the run (16, 6, 6, 3 across quartiles, permutation p = 0.0002), which is what you would expect from a layer that fills and evicts as it goes.
The natural objection is that reusing state obviously changes arithmetic, so this is definitional. The paper answers with a reset control rather than an argument. Serving 40 items four times — twice recompute, twice cache-hit, with cold state re-established and verified from reported cached-token counts before each recompute pass — the recompute path reproduced on 40 of 40 and the cached path on 40 of 40, while the two paths differed from each other on 14 of 40. Both paths are individually deterministic. They are simply different functions, and which one you get depends on state that is not in the request, not in the response, and not reset between runs by default in any stack tested. A second engine points the same direction through a different lever: vLLM configurations that launched a fresh server per pass diverged on 0 of 80, while configurations serving both passes from one process diverged on 7 of 80, 22 of 80, and 8 of 80 across sessions. Those sessions also differed in launch script, GPU memory fraction, and driver, so the paper uses vLLM for direction and llama.cpp for magnitude.
Two boundaries matter for reading these numbers. Divergence here means different emitted token identifiers — two runs saying the same thing in different words count as divergent. That is the right endpoint for a reproducibility question and the wrong one for a quality question. And the reset control was run with the prompt-cache layer disabled, so it characterizes the controlled regime, not the default configuration that the flag and ordering experiments cover.
What follows for practice is a change in what a run record has to contain. If the effective input includes cache state, then versioning weights and sampling parameters is not enough: the record needs the engine build and launch flags, the cache configuration and whether cache exposure is even observable (llama.cpp reports cached prompt tokens per response; the vLLM endpoint used here left that field unpopulated, so the check moved to the engine log), the server lifetime or initial state, and the reset boundary used when comparing arms. For a stateful serving evaluation I would restore cold state at each comparison boundary and verify it from reported cached-token counts, because that is the step that made each serving path reproduce exactly in this study. It is not the only intervention that worked: disabling llama.cpp’s host-memory prompt-cache layer cut repeated-run divergence to 1 of 80 episodes, and launching a fresh vLLM server for each pass produced 0 of 80. Note also what was held out: batch size one, serial requests, FP16 cache. Continuous batching and cache-precision effects were deliberately excluded so they could not confound the measurement, so these numbers describe that controlled regime rather than the broader serving setups those factors belong to.
Correct answers can become less grounded
The second paper (Faithfulness Is Not Free) asks a different question about a different cache. In offline RAG you can prefill document contexts once, write the KV cache to disk, and reuse it. Compressing that cache saves storage. The usual check is whether accuracy holds. This study asks whether that check is sufficient.
The experimental design matters before the numbers do, because it determines what the result can attribute. For each query the system concatenates the system prompt with the top-K retrieved chunks and runs a single causal prefill, producing one unified, position-consistent cache. That choice removes a known confound: caching chunks independently creates them at local token positions but consumes them at global positions during generation, and that mismatch degrades output on its own. With one prefill pass, bit width is the only variable changed among the cached BF16, INT8 and INT4 conditions. Four conditions run over the same retrieved chunks and greedy decoding — an uncached oracle, a BF16 cache round trip, INT8, and INT4 — on Qwen2.5-7B-Instruct across RGB and HotpotQA at K in {1, 3, 5}. A stage-0 gate required the BF16 cache to reproduce the oracle on 50 held-out examples before any quantized run, which is what licenses attributing later differences to precision rather than to the round trip itself.
INT8 stayed close to BF16. INT4 did not, and the interesting part is where the damage hides. Accuracy dropped: correct→wrong flips outnumbered wrong→correct by 133 to 10 on RGB and 117 to 24 on HotpotQA. But restrict attention to the accuracy-preserved subset — the examples where containment exact match is unchanged between BF16 and INT4 — and an LLM judge still records 231 worsenings against 24 improvements on RGB and 173 against 31 on HotpotQA. Those are answers that would pass a containment exact-match regression while becoming less grounded in the retrieved passages. INT4 was also the only condition producing degenerate output, empty generations or repetition loops reaching 6% on RGB and 3% on HotpotQA at K=5; BF16 and INT8 never degenerated. Separately, non-finite logits appeared under INT4 in under 0.1% of examples and were aborted, which is why the paper reads degeneration as semantic failure rather than numerical explosion.
One result is easy to misread as an improvement. On HotpotQA at K=1, INT4 dropped the refusal rate from 0.513 to 0.317. Fewer refusals mechanically inflate containment-EM and NLI entailment, so the apparent gain on those metrics is an artifact of the model answering when it previously declined. That is why the LLM judge is the primary signal in that regime — and the judge’s agreement with human labels was 76% on HotpotQA versus 92% on RGB, so the strongest evidence and the weakest measurement instrument sit in different places. The three faithfulness signals were also chosen to be complementary, and they capture distinct failure modes: HHEM and NLI entailment correlate only moderately on RGB (r = 0.44), which is the point of running more than one.
The trend toward worse faithfulness at deeper retrieval is real in direction and weak in statistics: the INT4–BF16 hallucination gap rises monotonically from K=1 to K=5 on both benchmarks, but the slope test over three depths is underpowered (p = 0.12), and the authors treat it as suggestive. Meanwhile the storage payoff is smaller than the label implies — roughly 3.6× rather than 4× at K=5, because per-block scales and zero-points consume about 11% of the INT4 footprint.
So the audit this suggests is narrow and concrete. If you are shipping a low-bit offline cache, compare it against a full-precision cache under identical retrieved evidence and identical decoding rather than against your previous production system, verify the full-precision round trip before introducing quantization so you know your baseline is clean, and score grounding, refusal calibration, and degenerate outputs at the retrieval depths you actually plan to serve — not just at K=1. These checks follow from the paper’s controls; they do not establish a general threshold for INT4. The evidence covers one model family, two QA benchmarks, three retrieval depths, and an offline unified-cache regime that says nothing about online prefix caching.
Measure the property you need to preserve
Both papers treat a cache as part of the specification of a system rather than as an invisible accelerator, and both find that the standard measurement misses something. Beyond that, they are separate results about separate machinery: one studies mutable server state in a live serving loop, the other studies static compressed caches written to disk. Nothing here shows they compound.
What they jointly clarify is that three endpoints people routinely substitute for each other are distinct. Token identity is a reproducibility question and can diverge without any quality change. Aggregate accuracy is a task question and can hold steady while grounding erodes underneath it. Evidence faithfulness is a third thing that needs its own instrument, sometimes several. Whichever cache you are touching, the useful move is to decide which of those three you are actually protecting, and then measure that one directly.