E-SFT trains reasoning models on the beginning and ending of teacher-generated reasoning traces while dropping roughly the middle 20%, on the hypothesis that intermediate steps are largely redundant and the model can fill them in from its own knowledge.
A common recipe for teaching smaller LLMs to reason: take a strong teacher like DeepSeek-R1, have it produce long chain-of-thought traces on math and science problems, then supervised-fine-tune a 4B–32B student on (question, full trace, answer) triples. Datasets like s1K / s1K-1.1 and OpenThoughts3 package thousands of such traces for exactly this purpose. This is the Supervised Fine-Tuning side of post-training, distinct from RL methods like Group Relative Policy Optimization (GRPO).
The problem: these traces are long, often 10k–30k tokens, and visibly full of detours, self-checks, and backtracking. Prior work (notably one line from s1) treated the full trace as the supervision signal, sometimes truncating only when it exceeded the context window. Nobody had cleanly asked whether the middle of the trace is actually pulling its weight during training, or whether the student would learn just as well, or better, from the setup plus the wrap-up.
The intuition: a reasoning trace has three functional zones. The beginning restates and decomposes the problem. The end consolidates and formats the answer. The middle is where the teacher explores, second-guesses, and backtracks. The authors argue the middle is where redundancy lives, and that a student LLM with reasonable pretraining can internally reconstruct that exploratory work from the endpoints.
They support this with three probes on a pretrained model, before any fine-tuning:
•
Attention analysis. When the model generates the answer, attention concentrates on tokens near the start and end of the trace. Middle tokens are weakly attended, and this pattern strengthens in later transformer layers.
•
Segment ablation. Delete a contiguous 10–20% chunk from the trace and regenerate the answer. Deleting the middle chunk gives answers most similar to the full-trace answer; deleting the start or end changes the answer more.
•
Perplexity under replacement. Replace a segment with the pretrained model’s own generation. Only middle replacement leaves answer perplexity roughly unchanged; replacing the start or end raises it.
Steps are delimited by double newlines, which the authors validate as reasonable semantic boundaries. The training recipe (called E-SFT) then keeps the first n and last n steps and drops everything between, targeting about 20% token removal overall. No scorer, no auxiliary model, no per-example tuning.
def build_esft_example(question, trace, answer, n=100):
steps = trace.split("\n\n")
if len(steps) <= 2 * n:
kept = steps # trace already short enough
else:
kept = steps[:n] + steps[-n:] # drop middle
trimmed_trace = "\n\n".join(kept)
return format_sft(question, trimmed_trace, answer)
The same idea extends to two other post-training objectives. For Group Relative Policy Optimization (GRPO), they mask the middle 20% of the rollout from the gradient (tokens still get scored by the reward but do not contribute to the update). For On-Policy Distillation, they mask the middle 20% of the student’s rollout when matching the teacher distribution.
The headline SFT result across three models (Qwen2.5-32B-Instruct on s1K / s1K-1.1, Qwen3-8B-Base and Qwen3-4B-Base on OpenThoughts3) and three benchmarks (AIME 2024, GPQA-Diamond, MATH): keeping only the prefix and suffix beats keeping the full trace on average, e.g. 75.19 vs 73.51 for the 32B model and 67.50 vs 65.78 for Qwen3-8B. Prefix-only or suffix-only alone underperforms the full-trace baseline, so it is specifically the combination of endpoints that matters.
Against other trace-reduction methods on s1K with the 32B model, E-SFT beats: random step selection (70.92), Jaccard-similarity filtering (73.92), perplexity-extreme filtering (72.80), perplexity-high filtering (74.43), LS-Mixture SFT (71.90), and, notably, external-LLM compression using Claude Sonnet 4 (60.08) or Gemini 2.5 Flash (49.55). LLM-based compression actively hurts, which the authors attribute to it disrupting the surface style the student needs to imitate.
Beyond SFT, applied as a middle-masking rule:
•
GRPO on DAPO-17k: Qwen3-1.7B-Base gains +8.7 points on average (much of it from MATH500); Qwen3-1.7B gains +1.2.
•
On-policy distillation with Qwen3-8B teacher into Qwen3-1.7B student: step-level middle masking gains +2.2 points on average over unmasked OPD, and beats token-level masking (+0.7).
Secondary analyses: an LLM-as-a-Judge evaluation with GPT-OSS-120B finds E-SFT traces preferred over the original s1K traces 22% of the time vs 8% the other way, with the rest called equivalent. Trained models shorten output on problems where the baseline over-thinks (>15k tokens) and lengthen it on short ones. Training loss converges lower with E-SFT than with full traces on OpenThoughts3, which the authors read as evidence that redundant middle tokens make optimization harder.
The authors also show E-SFT helps only once the base model is strong enough (roughly >60 on MMLU-family benchmarks for s1K; the threshold is lower for the larger OpenThoughts3). Below that, the student cannot fill in the trimmed middle, and trimming does not help.
•
If you are fine-tuning a mid-to-large open model (Qwen-3 family, Llama-3.1-8B, ~4B and up) on distilled reasoning traces and hitting context-length limits, a simple middle-trim keyed to \n\n boundaries is worth trying before reaching for perplexity scorers or LLM-based compressors. The paper’s default is retaining the first and last 100–200 steps (~80% of tokens); performance is not sharp around that ratio (see their Figure B), so you do not need to sweep it carefully.
•
If you are running Group Relative Policy Optimization (GRPO) or On-Policy Distillation on models that emit explicit <think>...</think> blocks, masking the middle 20% of the thinking segment from the gradient is a cheap ablation. It gave the largest gain on the weaker base model in their setup, so worth trying first where headroom is highest.
•
If you were considering paying a frontier API to “compress” your reasoning traces before SFT, the paper’s evidence argues against it: Claude and Gemini compression both underperformed doing nothing, and both underperformed a naive middle-cut. Worth reproducing on your own data before committing budget.
•
Prerequisite that matters: their gains require a base model already at moderate general-knowledge competence. On a very weak base, trimming the middle removes scaffolding the model cannot reconstruct. Test on your base before rolling out.
Code is available at the repo linked from the paper.
•
The largest models tested are 32B, and the strongest signal is on math and math-adjacent reasoning (AIME, GPQA-Diamond, MATH). Whether the same middle-redundancy pattern holds for code-heavy traces, agentic tool-use traces, or 70B+ students is not established here; the code-generation numbers on LiveCodeBench and CodeElo show E-SFT matches, not beats, standard SFT.
•
The 20% middle-drop is a heuristic, not a principled cutoff. The sensitivity plot shows a broad plateau but also clear degradation at aggressive trimming; if your traces have unusual structure (very short, or with the answer embedded mid-trace), the \n\n segmenter may misfire.
•
The attention and ablation analyses show correlation between low middle-attention and low causal impact on the answer, but the paper’s claim that the student “internally infers” the missing steps is an interpretation of the perplexity and loss-curve results, not a direct measurement of what the model computes in place of the trimmed content.
•
The GRPO and OPD results use 1.7B students and 100 training steps. They are suggestive of generalization beyond SFT, not a full study; larger-scale RL runs could behave differently.