XConf estimates whether an LLM’s answer is correct by looking up the model’s own graded past attempts on similar tasks and reading off how often it was actually right, matching or beating ten-sample Self-consistency at roughly a tenth of the generation cost.
Suppose you’ve deployed an agent that writes multi-step code patches, or a QA system that answers medical questions. For each output, you need one number: the probability it’s correct. That number decides whether you ship it, escalate to a human, or spend money on a retry. Get it wrong and you either ship broken outputs or waste review budget on things that were fine.
Today’s ways of producing that number all read only the current attempt. The model either states a confidence in words (Verbalized confidence), scores its own token probabilities, or gets resampled ten-plus times to measure agreement (self-consistency). Sampling-based methods are the strongest black-box baseline when answers can be compared, but they cost 10x more generations and don’t work naturally on code or on 30-step agent rollouts where there’s no clean way to “vote” across samples.
The authors’ claim: none of these consults the one thing humans use, the record of how similar past attempts turned out. A student who has done a hundred determinant problems trusts their answer without rechecking. That trust is data, not feeling.
The mechanism is a lookup, not a smarter prompt. After the model answers a task, it also writes a short self-reflection and states a confidence. Once the answer gets graded (by unit tests, a verifier, or an environment), the paper stores an episode: the task, the reflection, the stated confidence, the graded outcome, and a one-sentence lesson written after the grade arrived. This pile of episodes is the experience bank. It’s a byproduct of evaluation and deployment, not a separate dataset build.
When a new task comes in, XConf runs two readings of the bank:
•
Recall is statistical. It embeds the new task, looks up the 50 nearest past episodes in a space that has been rescaled so “similar” means fails for similar reasons (fit by a small logistic probe on the bank’s own outcomes), and returns the hit rate of those neighbors. If the model historically got problems like this right 44% of the time when it felt this confident, that’s the Recall estimate.
•
Reflect is verbal. It shows the model those retrieved episodes as short cards (task, prior stated confidence, outcome, lesson), asks it to name any recurring failure mode it sees, and then asks for a restated confidence.
The final estimate is a simple average of the two. Crucially, the retrieval key uses the task text and the reflection but never the answer string, which is why the same pipeline works for a multiple-choice letter, a 200-line program, or a 30-step trajectory. No weights are updated and no token probabilities are needed.
def xconf(task, bank):
answer, reflection, v = model.solve_and_reflect(task)
key = supervised_embed(task, reflection, v)
neighbors = bank.top_k(key, k=50) # past graded episodes
recall = weighted_hit_rate(neighbors, v) # confidence-conditioned
cards = render_as_context(neighbors[:8])
reflect = model.restate_confidence(task, reflection, cards)
return 0.5 * (recall + reflect), answer
Across 9 benchmarks and 4 models (two Gemini generations, Claude Sonnet 4.6, and Qwen3.5-397B-A17B), XConf beats or matches ten-sample self-consistency on 23 of 24 reasoning/code/multimodal cells in ranking quality (AUROC), and its calibration error (ECE) is much lower, on MMLU-Pro by 3-8x. Cost is one answer generation plus one short recalibration call, versus ten full generations for the sampling baseline.
The margin is largest where introspection struggles most:
•
On AppWorld, where failed agent rollouts often look clean, plain verbalized confidence averages .72 AUROC across the four models while XConf reaches .86.
•
On SWE-bench Verified, even a supervised trained verifier (HTC (trained verifier)) built on patch features stays below XConf on every model column.
•
On LiveCodeBench, self-consistency needs a code-similarity heuristic to work at all, and even then trails XConf by .06-.13 AUROC.
Selective prediction is the practical payoff. Abstaining on the least-confident 10% of episodes lifts delivered accuracy by 4.8 points on average across all 36 cells, and by up to 8.7 points on AppWorld with Gemini 3.5 Flash. The most-confident decile is 98% accurate; the least-confident decile is 28% accurate. The tail really is where the errors live.
•
Calibration keeps improving as the bank grows, with no saturation on the domains where episodes are scarcest.
•
Holding question difficulty exactly constant (via other models’ success rates on the same items), XConf still ranks correct vs incorrect at .79 AUROC and still beats verbalized confidence by 5 points. So it isn’t just a difficulty predictor.
•
The estimate survives the actor shrinking from 397B to 4B: Recall is nearly size-invariant, but Reflect degrades below 27B because small models are worse at reading their own track record.
•
Removing the stated confidence from the retrieval key costs .08-.12 AUROC; every other key element is roughly optional.
One loss to name: on short factual recall in the SimpleQA style, sampling-based voting remains stronger.
•
If you already grade outputs in production or evaluation, start the bank. The paper’s setup requires no training and no logit access; you need only an embedder and stored (task, reflection, stated confidence, graded outcome) tuples. A few hundred graded episodes with roughly 20 minority-class (failure) examples is enough for the key to become useful. This is the concrete lift the paper demonstrates.
•
The grading signal must be independent of the model itself. A weak but independent grader (an LLM judge that agrees with gold 91% of the time) preserves most of the value. Using the model’s own self-judgments as labels fails, because self-labels are systematically wrong on exactly the confidently-wrong cases the bank exists to catch. If you don’t have a real verifier or environment score, this method doesn’t apply cleanly.
•
Best fit is silent-failure domains. Where failure is obvious from the trajectory (the paper’s ALFWorld and WebShop cases, where the final state prints the reward), plain introspection already saturates and there’s no headroom. XConf’s biggest wins are on things like agent tasks whose rollouts look successful but weren’t.
•
Banks can be shared across similar datasets but not across models. A bank from a sibling coding benchmark (R2E-Gym) calibrates SWE-bench roughly as well as SWE-bench’s own bank. But borrowing another model’s bank costs .03-.06 AUROC on the same tasks, because what a record shares is which problems are hard, not which failure modes are yours. Worth testing before you assume a shared bank across a model fleet is safe.
•
Use it for selective prediction, not for making the model smarter. The paper doesn’t claim experience improves answers; it claims it improves the decision of what to ship, retry, or escalate. That’s a routing/abstention lever, not a reasoning lever.
•
The bank is bound to the actor. If you upgrade or fine-tune the model, older episodes gradually become another model’s; the paper flags but doesn’t solve how a calibration bank should forget in step with capability growth.
•
All main results run with “thinking” modes disabled to keep cost accounting comparable. The paper reports a paired ablation where XConf still wins with thinking on, but self-consistency degrades under thinking (unparseable votes or collapsed samples), so the cost comparison in a thinking-on production setup may differ from what you’d measure otherwise.
•
On short votable factual recall, sampling-based methods remain the stronger discriminator. XConf’s pitch is breadth (code, multimodal, long agent trajectories) and cost, not universal superiority.
•
Ranking quality with question difficulty held constant is real but smaller than the headline numbers (.05 lead over verbalized confidence rather than .08). Part of what looks like a calibration win is that the estimate implicitly learns which questions are hard for this model, which is a genuine gain but worth naming.
•
The method needs an external grading signal. In domains where you can’t cheaply verify outcomes (or your only “verifier” is another LLM judging the same model), the bank’s foundation is shaky.