Learnable Subspace Projections (LSP) compresses a pretrained transformer by learning which subspace to drop from each weight matrix end-to-end against the dense model’s output, with weights frozen. The gap over closed-form Singular Value Decomposition (SVD) baselines widens as compression gets more aggressive.
If you want to serve a 7B model on smaller hardware, one path is low-rank factorization: replace each weight matrix with the product of two thinner matrices. The savings are real and the result stays dense (so it’s fast on GPUs), but there’s a catch. Pretrained transformer weights don’t have a clean low-rank structure. Their singular values decay slowly, so naive SVD truncation wrecks quality.
The workaround everyone uses: don’t look at the weights, look at the activations. Activations flowing between layers do live in low-dimensional subspaces. Methods like SliceGPT, ASVD, and SVD-LLM pick which directions to discard based on activation energy or layer-wise reconstruction error, computed in closed form per layer.
The problem the authors identify: these are local criteria. A direction that carries little activation energy at layer 12 might still be the one the network’s final output depends on. And errors compound through depth. At mild compression (30%) this is tolerable. At 70% compression the models collapse.
The core move: treat “which subspace to remove” as a learnable thing, and train it against the whole network’s output.
For each linear layer, LSP attaches an orthogonal projector that kills a chosen k-dimensional subspace of the input (or output). Choosing a subspace = choosing a projector = choosing a rank-r factorization. So the search for “what to compress away” becomes gradient descent over projectors, with the pretrained weights frozen the entire time.
Three pieces make this work at billion-parameter scale:
1.
Tied groups. Layers that read the same activations (Q/K/V in attention, gate/up in MLPs) share one projector. One shared input factor instead of three separate ones. This is what later enables the KV-cache win.
2.
Whitened initialization + measured-KL rank allocation. Each projector starts from a whitened SVD truncation (a spectral starting point that accounts for activation covariance). To decide how many directions each layer gives up, they apply each candidate truncation in isolation, measure the resulting KL divergence on the dense model’s output, and greedily spend the parameter budget where KL-cost-per-saved-parameter is lowest. Sensitive layers stay dense.
3.
End-to-end training against a global objective. Default objective is distillation: match the dense model’s output distribution. Alternative (LSP T) uses the model’s original training loss. The teacher is the same network with projectors disabled, so no second copy of weights is stored.
After training, each projector folds exactly into standard low-rank factors W \u2248 B A. The deployed model has no LSP-specific ops. For tied Q/K/V groups, the shared input factor also means a KV cache can store one narrow latent z = Ax instead of full keys and values.
for layer in targeted_linears:
W_frozen = layer.weight # never updated
V = init_from_whitened_svd(W_frozen, calibration_acts)
# Allocate k per layer by measured KL cost per saved param
for step in training:
for layer in targeted_linears:
U = qr(V) # orthonormalize
P = I - alpha * U @ diag(mask) @ U.T
y = W_frozen @ (P @ x) # project in activation space
loss = KL(dense_output, compressed_output) + lambda_ort * L_ort
loss.backward() # updates V only
On four decoder LLMs (OPT (125M/1.3B)/1.3B, Qwen3-4B, Llama-2-7B) at 30/50/70% compression, one of the two LSP variants gets the lowest WikiText-2 perplexity in all 12 model-ratio cells, and the margin grows with the ratio.
The cleanest ablation is the comparison to NoLSP, which uses the same whitened initialization and the same measured-KL rank allocation but skips the training step. On Llama-2-7B at -70%, NoLSP gets 222.8 perplexity. LSP gets 10.9. The dense model is around 5.5. So the learning stage, not the initialization or rank allocation, is what makes aggressive compression usable. The strongest baseline (LoRA-recovered SVD-LLM) gets 13.3 at that ratio.
On zero-shot accuracy across six commonsense/science/math benchmarks, both LSP variants lead the best training-free low-rank baseline in mean accuracy at every ratio, by 6.7-10.5 points on Llama-2-7B and 6.2-10.8 on Qwen3-4B. So the gains aren’t just overfitting to the calibration domain.
On ViT-B/16 fine-tuned for CIFAR-100, LSP is the most robust to calibration shift: when the calibration pool is diversified across unrelated datasets, LSP gains more on downstream linear-probe transfer than every baseline, including PELA (which retrains all weights via feature distillation).
Inference wins come from two sources. At -70% on Llama-2-7B, LSP decodes 1.56\u00d7 faster than dense at small batch sizes, because tied groups collapse what would be three small matrix multiplies into one. With a latent KV cache, weights + cache at 128k context take 13.5\u00d7 less memory than dense, versus at most 6.5\u00d7 for untied factorizations. A 95.5 GiB GPU holds \u2248320k cached tokens per sequence versus at most 146k for untied baselines.
Analyzing what gets compressed: on Llama-2-7B the allocator strips Q/K/V hard (shared latent at ~1/7 of full rank at -70%) while leaving MLP down-projections comparatively dense. Learning (vs. NoLSP) specifically removes more of the leading singular directions and keeps more of the trailing ones, which is the opposite of what weight-SVD truncation does. The spectral ranking of directions is a poor proxy for functional importance.
•
If you’re deciding between low-rank compression methods for a transformer you plan to serve, the regime matters. At 30% compression, closed-form SVD methods are competitive and much cheaper (compression runs in minutes, not hours). At 50%+ compression, the paper’s evidence is that end-to-end learning of the subspace is the thing that keeps the model usable. The crossover is why they emphasize “margin grows with the ratio.”
•
If you care about long-context serving on a single GPU, the tied-input-side KV grouping is the actionable idea even separately from LSP’s training. Caching one shared latent per Q/K/V group rather than full K and V is where the 13.5\u00d7 memory win comes from. The paper notes this needs a dedicated attention implementation (similar to Multi-head Latent Attention (MLA) or Palu); it’s not free from factorization alone.
•
Compression cost is a one-time payment. For Llama-2-7B, LSP takes on the order of ~10 wall-clock hours total (including KL measurements that are reused across ratios and objectives). Worth testing against your inference-cost savings budget.
•
Choice of objective matters by scale. LSP T (task loss) converges faster and wins on smaller models and lower ratios where specialization helps. Distillation (default LSP) wins on larger models and higher ratios. If you’re compressing something Llama-2-7B-sized at aggressive ratios, start with distillation.
•
For a vision setting where calibration data and deployment data differ, LSP degraded least among compared methods. If you’re compressing a backbone you’ll use across tasks, consider calibrating on a diverse unlabeled pool rather than only in-domain data.
•
The measured-KL rank allocation measures each unit in isolation. The paper notes the joint KL of the final allocation exceeds the sum of isolated costs by 2-4\u00d7, so the allocator is a surrogate without a global-optimality guarantee. In a few settings uniform allocation is competitive or better.
•
Reported LLM results are on OPT-125M/1.3B, Qwen3-4B, and Llama-2-7B. The paper does not evaluate larger (30B+) or instruction-tuned chat models, so the ratio-dependent crossover may shift there.
•
The KV-cache memory win requires both (a) tying K/V on the input side and (b) a latent-cache attention implementation. On Qwen3-4B’s grouped-query attention, their best-accuracy configuration ties K/V on the output side instead, which gives up most of the KV-cache savings. Explicit accuracy/efficiency tradeoff.
•
Zero-shot evals cover six commonsense/science/math benchmarks; safety, long-form generation, and code weren’t tested. Standard compression caveat: lossy approximations can distribute errors unevenly across subpopulations and rare behaviors.