U-Space reads a language model’s uncertainty from a small set of directions inside the residual stream, built by contrasting doubt and certainty vocabulary, giving a per-token map of where and why uncertainty arises without extra generations or correctness labels.
You’re routing hard questions to a reasoning model in production, and some answers come back wrong with a confident tone. You want a per-answer confidence score so you can defer the risky ones to a human. The common fixes are expensive or opaque. Sampling-based methods like SelfCheckGPT or Semantic entropy need many generations per question. Self-rated confidence (asking the model “are you sure?”) tends to be overconfident. Trained probes on hidden states need labeled correct/incorrect data from your task.
Worse, a recent line of work shows these estimators often cheat on generation length: incorrect reasoning traces tend to be longer, so a score that merely tracks token count looks predictive. On this paper’s setup, raw trace length alone hits 68.6% AUROC, nearly matching a supervised probe. Once you control for length, many estimators collapse. The authors want a signal that survives length control, needs no labels, and tells you which part of the reasoning got shaky.
The intuition: inside a model’s hidden states, there’s a direction that means “I’m uncertain” and an opposite direction that means “I’m confident.” If you can find those directions, projecting a hidden state onto them tells you how uncertain the model currently is, without running it again.
To find the directions, the authors start with two published word lists: cue words of doubt (“maybe”, “unclear”) and cue words of certainty (“definitely”, “undoubtedly”). They sort the doubt words into four categories (ambiguity, incompleteness, conflicting evidence, general uncertainty) with a sentence encoder. Each word lives in output-token space, so to get its matching direction in the mid-layer residual stream they use the Jacobian Lens (J-Lens): a precomputed averaged Jacobian that maps intermediate hidden states to final-layer outputs, run in reverse. For each category they average the pulled-back doubt words, subtract the averaged certainty words, and orthogonalize the four resulting vectors. That orthonormal 4-vector basis is the U-Space.
At inference, the U-Lens reads a reasoning trace in one pass:
for t in reasoning_tokens:
h = hidden_state(layer=two_thirds, token=t)
alpha = U.T @ normalize(h) # 4 signed scores
s[t] = softplus(sqrt(4) * normalize(alpha))
cone[t] = norm(s[t]) # verbalizable uncertainty
H_mean = mean(predictive_entropy(t) for t in reasoning_tokens)
score = cone[end_of_thinking] * H_mean # final uncertainty
Two signals get combined. The cone alignment at the end-of-thinking token is a “second-order” signal, meaning uncertainty the model could in principle verbalize. Mean Predictive Entropy over the reasoning trace is a “first-order” signal from the output distribution. Multiplying them (rather than fitting a weighted sum) keeps the method label-free and preserves rankings under rescaling.
Evaluated on three open reasoning models (Gemma 4, Qwen3.5, Magistral 1.1) across MMLU-Pro, Omni-MATH, SuperGPQA taxonomy, and TriviaQA, with length-stratified metrics that bin outputs by token count before scoring:
•
U-Lens wins all nine length-controlled model-metric cells in the headline table, improving AUROC over the strongest baseline by 2.2, 1.9, and 4.7 points on the three models.
•
Under length control, several multi-pass baselines (P(True), TokUR, SentenceSAR) fall below plain predictive entropy, meaning the extra forward passes were partly buying length-correlated signal.
•
The U-Lens drops only 1.8 points when length is controlled, versus generation length collapsing from 68.6% to 52.4% AUROC (near chance).
•
The two components are complementary: on Qwen3.5 predictive entropy carries the signal, on Magistral the cone alignment does, and their product beats either alone.
•
Supervised probes trained on one benchmark lose 10 to 22 points of AUROC when transferred to another; the label-free U-Lens doesn’t have that problem since it was never fit to a benchmark.
•
A causal check: adding a category direction to the residual stream during generation makes the model hesitate or hedge on questions it would otherwise answer confidently, suggesting the directions actually drive behavior rather than just correlating with errors.
The token-level readout places each reasoning token in a 4-vertex simplex, so you can watch the dominant uncertainty category shift (e.g., ambiguity giving way to conflicting evidence) across a trace.
•
If you serve open-weight reasoning models and want a cheap, label-free confidence score, this is worth trying. It needs hidden-state access at a mid-to-late layer and a precomputed Jacobian, both of which are feasible with open weights but not through a typical hosted API. The paper’s GitHub ships code and uses publicly available Jacobians for Gemma 4 and Qwen3.5.
•
If you’re already using a sampling-based estimator, re-run your evaluation with length-stratified metrics before concluding it beats single-pass alternatives. The paper’s length-controlled table is a clean template, and several multi-pass methods lose their edge once length is matched.
•
For debugging or auditing a reasoning trace, the token-by-category simplex readout gives you a visualization of when the model’s uncertainty switches character. Useful if you’re building traces into UX surfaces or doing error analysis, less so as a safety-critical signal on its own.
•
The construction isn’t uncertainty-specific. The appendix shows the same recipe (two contrastive word lists, pull back through the Jacobian, orthogonalize) picks up emotion categories on GoEmotions and reward-hacking patterns. Worth testing on other interpretable attributes you can describe with word pairs.
•
Needs internal activations and a Jacobian. Hosted-API-only models are out.
•
Readout depth is fixed at two-thirds through the network as a heuristic, following prior J-Lens observations. Other layers were worse in the ablation but the choice isn’t tuned per task.
•
The uncertainty taxonomy (four categories) is a design choice, not derived from the model. The authors argue these distinctions exist internally based on contrastive activation probes, but it’s not an exhaustive carving.
•
Even the strongest numbers (around 70–74% AUROC) leave a lot of residual error. The authors explicitly caution it is not a calibrated safety detector and shouldn’t replace human review in consequential settings.
•
Evaluation covers three open reasoning models and four reasoning-heavy benchmarks. Transfer to non-reasoning tasks, closed models, or long-form generation is not established here.