LLMs told to answer directly still emit visible reasoning before the answer, a pattern the authors call thinking inertia. Suppression works on yes/no tasks but on open-ended tasks, strict answer-only prompts raise compliance to ~40% while cutting accuracy by ~15 points.
You’re building an agent that calls a reasoning-capable model for cheap, latency-sensitive classifications. You flip the provider’s enable_thinking=false switch or add “answer directly, no reasoning” to the prompt, expecting short responses and lower token bills. In practice, the model keeps writing a sentence or two of task-specific inference before the final token, and you still pay for it.
Most prior work decides whether a response counts as “no-thinking” using surface proxies: the thinking mode is off, the <think> block is missing, or the output is short. All three fail. A disabled thinking channel can still leak a compact derivation into the ordinary answer channel, and a short response can contain the one equation that actually solved the problem. The paper argues no-thinking needs to be measured on the response itself, not on the switch you flipped.
The setup splits every model output into two pieces: a pre-answer text T (everything visible before the final answer) and the final answer A. Then T is scored at three levels of strictness.
•
Level 1, Empty-Thinking Rate: a binary check. Did T come out empty, with A in the expected format? This is the strictest reading of “answer only.”
•
Level 2, Question–Pre-answer Relevance: how semantically related is T to the question? Computed as cosine similarity between question and pre-answer embeddings from Qwen3-Embedding-4B, using an instruction that targets reasoning-relevance rather than surface overlap.
•
Level 3, Explicit Inference Rate: a blinded LLM judge (GPT-5.5, cross-checked with Claude Opus 4.8) sees only (question, T) and picks one of five labels. Only explicit reasoning (deduction, computation, comparison, elimination) counts toward EIR. Paraphrases and relevant assertions do not.
The authors then sweep six intervention modes on the same tasks: thinking on (M1), native off (M2), chain-of-thought re-elicitation (M3), soft answer-only (M4), strict answer-only (M5), and prefix-forcing the answer template (M6). They run this across six models including Qwen3 at three sizes, DeepSeek-V4-flash, GPT-5.4, and Gemini-3-Flash-Preview, on Boolean (BoolQ, StrategyQA), multiple-choice (MMLU, MMLU-Pro), and open-ended (GSM8K, MATH) tasks.
for response in model_outputs:
T, A = split_pre_answer_and_final(response)
etr_i = int(len(T) == 0 and is_parseable(A))
qrel_i = 0 if len(T) == 0 else cosine(embed(Q, instr), embed(T))
label = blinded_judge(Q, T) # 5-way rubric
eir_i = int(label == "explicit_reasoning")
acc_i = check_answer(A, gold)
The thinking switch is not a reasoning switch. Even with thinking disabled, DeepSeek-V4-Flash shows visible explicit inference in 99.9% of open-ended responses. Across models in native no-think mode, EIR climbs with answer-space openness: low on Boolean, intermediate on multiple-choice, highest on open-ended. The authors call this ordering thinking inertia, and it survives a step-by-step prompt that re-elicits visible reasoning.
Strict suppression trades accuracy for compliance on open-ended tasks. Moving from native off (M2) to strict answer-only (M5) on open-ended benchmarks raises mean ETR from near zero to ~40% and drops mean accuracy by ~15 percentage points. Across the full corpus, open-ended EIR under M5 is still 62.00% versus 99.69% at M2. Boolean and MCQ accuracy stays largely flat under the same pressure.
Prefix-forcing is not a stronger strict mode. Pre-filling "The answer is \boxed{" (M6) does not reliably produce answer-only responses. ETR actually drops versus M5 on MCQ and open-ended, and QRel. and EIR rebound toward the M2 baseline.
The answer interface matters more than the question. Rewriting the same numeric MMLU items for Qwen3-32B from MCQ into Boolean verification raises ETR from 10.5% to 62.5% with accuracy roughly unchanged (77.8% to 78.9%). Rewriting them into open generation drives ETR to 0% and accuracy down to 52.6%. Same underlying questions, very different compressibility.
Human annotators on 600 outputs agreed with the GPT-5.5 judge on 98.8% of binary explicit-inference calls and reproduced both the answer-space ordering and the M5-versus-M2 gap.
•
If you’re relying on enable_thinking=false or a “no reasoning” instruction to cut inference cost on open-ended generation, measure what actually comes back. The paper’s evidence is that provider switches do not reliably eliminate pre-answer reasoning on open tasks, and billed tokens will reflect that.
•
When the task allows it, reshape the answer interface rather than fighting the model with stricter prompts. Converting an open-ended question into Boolean verification or MCQ gave much higher answer-only rates without the accuracy hit the paper saw from strict prompting alone. Worth testing on your own classification pipelines before pushing suppression harder.
•
If you need to audit no-thinking behavior yourself, the three-level framework (empty-T rate, instruction-aware embedding similarity, blinded judge for explicit inference) is more informative than token-length heuristics. The paper validates that a short response can still contain the key inferential step.
•
Treat “the model complied with answer-only” as a visible-output claim only. The authors are explicit that hidden computation may still be happening; you cannot infer actual compute savings from the absence of a reasoning trace.
All claims are about observable response text. The paper does not attempt to characterize latent computation inside the model, and the authors note that visible compliance should not be read as evidence that internal reasoning has stopped. Decoding is deterministic and English-only; multilingual, tool-use, multi-turn, and sampled-decoding settings are not covered. EIR depends on a judge-model rubric; the authors mitigate this with an independent cross-family judge and human validation, but the metric is still rubric-bound. Frontier API models are evaluated through provider interfaces that can change, so exact rates may drift as endpoints update.