This paper proposes four functional axioms (Causality (axiom), Minimality (axiom), Separability (axiom), Stability (axiom)) for evaluating “thought representations” inside LLMs without running a downstream benchmark, and finds that no candidate representation across five open-weight models beats just embedding the input prompt, especially at telling apart two different questions from the same task.
If you’re building a reasoning product, you’ve probably seen the wave of methods that replace token-level chain-of-thought with continuous “latent thoughts” to save tokens or unlock parallel reasoning. The pitch is that some internal vector now carries the reasoning state. Today, the only evidence anyone gives you is a benchmark accuracy number. That conflates two things: whether the representation is actually a good summary of the problem, and whether the rest of the model can still decode well from it. The dominant approach in this space, exemplified by COCONUT, judges these latent vectors by downstream task accuracy. This paper argues that’s the wrong instrument, and supplies four direct readouts that run on a frozen model.
The core move is to define what a “thought” representation T should functionally do, then measure each property directly. The four axioms in plain terms:
•
Causality: if you splice T into the model in place of the actual reasoning tokens, the next-token distribution shouldn’t change. Measured as KL divergence between the two continuation distributions.
•
Minimality: T should compress the input down to what’s needed to predict the output, no extra input-specific noise. Measured via a tractable cross-entropy surrogate to the Information Bottleneck objective.
•
Separability: a bounded probe on T should be able to tell two semantically different questions apart, both across tasks and within the same task. Measured as discriminator accuracy.
•
Stability: T should reflect the distribution over possible answers (including the model’s uncertainty), not just one sampled string. Measured by whether a linear probe on T can predict whether beam outputs span multiple semantic classes (Distributional Consistency Score).
The authors prove these four are logically consistent, independent (each can be violated while the other three hold), and complete. Each metric uses a frozen LLaMA-3.2-1B as a shared decoding surface with a small trained projection on top, so cost stays constant across model sizes.
The evaluation loop for one (model, candidate) pair looks like:
for problem in bbeh_tasks:
beams = source_llm.beam_search(problem, k=8, max_tokens=8192)
T = extract_candidate(source_llm, problem) # e.g. last hidden state, soft-think vector
causality[problem] = kl(P(suffix|prefix), P(suffix|project(T)))
minimality[problem] = ce(X|Y,T) - ce(Y|T)
separability[problem] = discriminator_acc(T, paired_outputs)
stability[problem] = auroc_probe(T, semantic_entropy(beams))
Candidates audited include last-input-token hidden states, Soft Thinking with and without Gumbel noise, and Latent Thinking at 1, 16, 32, 64, and 128 thinking steps, against output-embedding (upper bound) and input-embedding (the prompt itself) references.
The prevailing assumption when a new latent-reasoning method posts higher benchmark accuracy is that its internal “thought” must be a better summary of the problem. This paper shows the opposite. High downstream accuracy can coexist with a thought representation that cannot even distinguish two questions drawn from the same task; the latent vector encodes coarse task identity and little else. The cleanest evidence is the within-task discriminator collapse, not the accuracy numbers on BIG-Bench Extra Hard.
The load-bearing finding is the within-task Separability (axiom) collapse. Across all five models and every candidate except the cheating output-embedding reference, the same-task discriminator sits at roughly 50–55%, indistinguishable from random guessing, while cross-task discrimination is near-saturated at >95%. The representations know what kind of problem they’re looking at but not which specific instance. A geometry-only analysis (no probe in the loop) confirms this is structural: the within-task subspace is too narrow for any bounded classifier to recover, and a probe-capacity ablation that scales trainable parameters by ~10x doesn’t move the number.
•
No candidate beats the input embedding on any axis when averaged across models. The prompt itself is competitive with every fancy latent thought.
•
Iterative methods degrade with more steps. Soft Thinking and Latent Thinking both lose Distributional Consistency Score as thinking budget grows from 1 to 128 steps.
•
The collapse is uniform across dense, sparse-MoE, reasoning-distilled, and RL-trained models (Llama-3.1-8B, Llama-3.3-70B, DeepSeek-R1-Distill-Qwen 32B, Skywork-OR1-32B, GPT-OSS-20B), so it’s not a property of training recipe or scale.
•
Per-task within-task accuracy shows no significant correlation with BIG-Bench Extra Hard pass@1 (Spearman ρ=0.10, p=0.31), so this isn’t just “hard tasks have noisier representations.”
Reach for this framework when you’re evaluating a latent-reasoning approach for a product, say a continuous-thought variant that promises shorter context at equal accuracy. Instead of just running your benchmark and shipping, run the four probes on a held-out set. If Separability (axiom) within a task collapses to chance, your latent vector isn’t actually carrying instance-level reasoning state, even if accuracy looks fine. That’s a signal that the decoder is doing the work and the “thought” is decorative, which matters if you later want to cache, route on, or interpret that vector.
The full pipeline is released as a Hydra project at fard-lab/formalize-thoughts under MIT. It targets Python 3.12 / CUDA 12.6 and uses public HuggingFace assets for the source LLMs, the LLaMA-3.2-1B probe backbone, and the Nemotron embedder. The dataset is the public BIG-Bench Extra Hard release. The paper notes the full audit took ~3,700 H100-hours, dominated by source-LLM generation, so reproducing one model is much cheaper than reproducing all five.
Latent reasoning methods can hit the same accuracy as text CoT while their “thoughts” carry barely more information about the specific question than the input prompt itself. Before trusting a continuous-thought vector as a reasoning artifact, probe it directly. Benchmark scores will not tell you when the decoder is doing all the work.
•
The audit only covers candidates extractable from pretrained models with no extra training. A representation trained explicitly to satisfy these four axioms might pass, and that’s left to future work.
•
Lexical-invariance under Stability (axiom) is untested because every candidate evaluated produces one vector per input by construction, so paraphrase robustness is trivial here.
•
GPT-OSS-20B’s MoE routing collapses beam outputs to one semantic cluster on 99% of questions, making its Distributional Consistency Score estimates uninformative rather than a clean signal. Conclusions are anchored on the four dense models.
•
Scope is English-language reasoning tasks in BIG-Bench Extra Hard; multilingual or non-reasoning generation is out of frame.