Averaging the input embeddings of two unrelated texts and running one forward pass through a standard LLM produces logits that keep both streams’ next tokens near the top of the distribution, and a light Intra-loop self-distillation step plus contrastive decoding turns this into a ~2× throughput trick that emits two coherent continuations at once.
If you serve an LLM and want to answer two prompts, you either batch them (two independent sequences through the same weights, roughly doubling compute and KV-cache) or run them one after the other. Both assume the model can only carry one “train of thought” through its hidden state at a time. That assumption is baked into how people think about Transformer inference: softmax attention and MLP nonlinearities should scramble any attempt to jam two inputs into the same vector.
The authors challenge that assumption. They ask whether a decoder-only transformer is linear enough, end to end, that if you feed it the average of two input embedding sequences, the output logits look like the average of what you’d get from each input separately. Prior work by Razzhigaev and colleagues had already shown that layer-to-layer transitions inside these models are close to affine maps; this paper asks whether that local linearity survives all the way from input embeddings to output logits.
The core setup is deliberately minimal. Take two unrelated texts A and B of the same length, embed each token, and average the embeddings position by position. Feed that mixed sequence into a frozen pretrained LLM. Now look at the output distribution at each position and ask: does the token the model would have predicted for A alone still appear near the top? Does B’s token?
Empirically, yes. On models from Pythia, Qwen, and Llama families, the true next token for a single stream lands in the top-10 of the mixed distribution roughly 30–40% of the time, and in the top-100 about 60–65% of the time, against vocabularies of 50k+. A frequency-only control (predicting B’s token from A’s logits without any mixing) gets top-10 in only 2.63% of cases, so the effect isn’t just Zipf priors leaking through.
The authors call this the Superposition Linearity Hypothesis. To check whether it’s an architectural bias or something the model learned, they replay it across Pythia pretraining checkpoints. Linearity is strongest at initialization and degrades monotonically as pretraining proceeds. So it’s a property of the transformer stack itself; pretraining actively erodes it.
An attention-patching experiment isolates what is doing the work. Two controls on a single stream: (a) donor patching replaces every attention-weight matrix with those computed from an unrelated text (destroys content, preserves attention shape); (b) permutation patching shuffles attention weights within each row (destroys shape, preserves per-row weight statistics). Donor patching keeps the model’s top-1 prediction near the top; permutation collapses it entirely. So attention shape (locality bands, Attention Sinks, head specialization) plus the LM-head frequency prior together explain most of the aggregate signal on easy positions. But on hard “content” tokens (roughly 35% of positions), embedding mixing preserves more signal than donor patching does, meaning the mixed residual stream really is carrying case-specific information about both streams, not just riding the frequency prior.
Then they restore the degraded linearity with self-distillation on a small FineWeb subset. Teacher is a frozen copy of the pretrained model; student sees mixed embeddings and is trained to match the average of the teacher’s two independent output distributions:
for (x_A, x_B) in pairs:
P_A = teacher(x_A) # frozen
P_B = teacher(x_B) # frozen
P_target = 0.5 * (P_A + P_B)
z = 0.5 * (embed(x_A) + embed(x_B))
P_mix = student(z)
loss = KL(P_target || P_mix)
loss.backward()
Decoding two coherent streams from the mixed forward pass is harder than matching the mixed distribution. Averaging logits means the mixed probability of any token scales like the geometric mean of its per-stream probabilities, so tokens likely in A but unlikely in B get crushed. To pull the streams apart, they use Joint Contrastive decoding: a small auxiliary model runs each stream independently, and its logits are added (for the target stream) or subtracted (for the other) from the mixed large-model logits, with learned scalar weights.
After distillation, distributional fit tightens sharply. On Pythia-2.8B, mean KL divergence between the mixed output and the ideal average distribution drops from 1.86 to 0.27, and the probability of the true next token landing in the top-5 climbs from ~30% to over 60%. Critically, the improvement is not just on easy tokens: median rank on hard content positions goes from 284 to 5, and exact top-1 agreement jumps to 22.8%. The fine-tuned model is genuinely processing two semantic streams in parallel, not leaning harder on the frequency prior.
There is a real cost. The distillation objective degrades single-stream language modeling: Pythia-2.8B LAMBADA accuracy falls from 0.544 to 0.357, Qwen2.5-3B from 0.602 to 0.460. Full FineWeb perplexity roughly triples for the distilled backbones.
Joint Contrastive decoding, evaluated on LAMBADA where both prompts run through a superposed forward pass, lifts mean per-stream accuracy from the raw-pretrained 0.06–0.18 range to 0.11–0.43, with Llama-3.2-3B reaching 0.43 against a single-stream small-guide baseline of 0.54. Inter-stream token overlap (Jaccard) stays low, so the outputs really are two separate continuations rather than one duplicated. Throughput benchmarks show the two-stream fused forward pass runs within 3% of independent batch-size-2 decoding while producing two coherent outputs, delivering roughly 2× over sequential single-stream generation. Importantly, the guided variant that preserves single-stream quality (small guide model at inference) also incurs less single-stream degradation than the tuned-distillation-only path.
The effect extends past two streams: mixing three inputs still gives distribution ratios well below 1, with modest degradation (KL ratio worsens by +0.04 to +0.09 depending on model).
For engineers running inference on models you control (own weights, own serving stack), this is a candidate throughput lever worth prototyping when you have paired requests of similar length and don’t mind a small quality hit. The recipe requires fine-tuning the backbone and running an auxiliary small model at decode time, so it isn’t something you can bolt onto a hosted API. The single-stream quality regression matters: if the same deployment also serves solo requests, you’d need to keep an unmodified copy of the model or accept the LAMBADA-scale drop the paper reports.
For researchers, the pretraining-degrades-linearity finding is the reusable claim. If you’re studying interpretability, feature geometry, or Mechanistic Interpretability, the paper gives you a concrete probe (mixed-embedding rank survival) that behaves predictably across checkpoints and correlates with layer-wise affine approximability. The U-shaped depth profile (early and late layers near-linear, middle layers less so) is worth checking in your own models before assuming a monolithic “nonlinear stack.”
For anyone thinking about parallel decoding schemes like Superposed Decoding or DataMUX: this paper’s contribution is showing the primitive already exists inside off-the-shelf models and can be amplified rather than architected in from scratch. Worth testing whether your own multiplexing method is really adding capability or just surfacing something the pretrained stack already does.
Evaluation is short-context (mostly 128 tokens, up to 512 for some distributional metrics) and monolingual, on text-only models. Nothing is shown about long-context behavior, multilingual mixing, or multimodal embeddings.
The ~2× throughput claim assumes you actually want two parallel continuations from the same model. It doesn’t help latency for a single request, and if your two streams have very different lengths you lose the batching benefit anyway.
Joint Contrastive decoding narrows but does not close the gap to independent single-stream inference. The paper frames the residual gap as an intrinsic geometric-mean obstruction rather than a fitting failure, which means further distillation alone probably won’t fix it. Treat the decoder as a proof of concept, not a shipped system.
Finally, “the mixed distribution approximates the average of two independent distributions” is a statement about output probabilities, not a guarantee that generated sequences are semantically clean. Sampling directly from the mixed distribution without the contrastive decoder produces text that alternates incoherently between A and B.