Bringing back Layer Dropout in LLM pretraining, when paired with the right per-rate scaling and a schedule that starts noisy and decays to zero, cuts up to 25% of training FLOPs while matching dense loss and unlocking up to 1.55× inference speedup via Self-Speculative Decoding.
You’re serving an 8B model and want three things at once: cheaper pretraining, a knob to trade latency for quality at inference, and a drafter for speculative decoding, all without maintaining separate distilled checkpoints. Today the standard recipe (post-GPT-3, LLaMA-style) drops explicit regularization entirely, on the belief that single-epoch training over trillions of tokens has no overfitting to fight. That belief killed Stochastic Depth along with activation dropout.
This paper argues the field threw out the wrong tool. Stochastic depth isn’t just a regularizer, it’s a structural training signal that makes the finished model tolerant of running with fewer layers. And once you configure it correctly, it does not hurt validation loss even in the modern single-epoch regime.
During each training step, each transformer block is skipped for some sequences with probability p. The output of the block, when kept, is multiplied by 1/(1−p) so that the expected contribution matches what inference will see with all blocks active. The paper shows this scaling choice (call it inverse-density scaling) is what lets learning rate and weight decay stay the same as you change the dropout rate, which is the reason prior work reported degradations: they were re-tuning around a broken scaling factor.
The paper then pins down three configuration axes:
•
Granularity: skip whole transformer layers (attention + FFN together), not attention and FFN independently. Coarser wins here, which is counterintuitive.
•
Distribution across depth: dropout rate increases linearly from 0 at the first layer to p_max at the last. Early layers stay reliable; late layers learn to be optional.
•
Schedule across time: start training with high dropout, decay to 0 by the end. High noise early forces the model to explore; a clean tail lets it settle.
The recommended recipe is the combination of the last two, which the paper calls ILD+DTS.
for t in range(T): # training step
p_time = (1 - t/(T-1)) # decreasing schedule
for l in range(L): # layer index
p = p_max * (l/(L-1)) * p_time # increasing across depth
keep_mask = bernoulli(1 - p, batch_size) # per-sequence
out = block_l(x)
x = x + (1/(1-p)) * keep_mask * out # scale then apply
Per-sequence masks (each sequence in a batch independently keeps or drops the block) beat per-batch masks on loss, and give the same FLOPs saving in compute-bound training.
The prevailing view is that dropout is a regularizer, and regularizers are pointless when you train once over a trillion tokens with no overfitting to prevent. This paper shows the opposite. Layer dropout in modern pretraining earns its keep as a structural training signal, not as a regularizer: it teaches the model to remain functional at reduced depth, which pays out later as free early exit, layer skipping, and speculative drafting. The load-bearing evidence is not the loss table, it’s that the 3.9B dense baseline loses badly under alternating-layer skipping (loss 6.4) while the dropout-trained model degrades to 2.1.
The finding that makes the thesis true is the inference-elasticity gap on large models. On the 3.9B run, skipping every other layer at inference gives cross-entropy 2.13 for the dropout-trained model versus 6.45 for the dense baseline. Self-Speculative Decoding gives 1.54× speedup for the dropout-trained 3.9B but only 1.02× for the dense one, meaning the dense baseline effectively cannot be sped up this way at all.
Secondary evidence, in the paper’s own framing:
•
Training FLOPs: the ILD+DTS recipe reaches dense-baseline validation loss with up to 25% fewer FLOPs at 8.2B; at 503M and 906M it slightly beats the dense baseline at 5% FLOPs saved.
•
Loss stays within ~0.50% of baseline as tokens-per-parameter is pushed well past the compute-optimal 20:1 point.
•
Bigger models tolerate more dropout: the 8.2B run uses p_max = 0.99 at its deepest layer (final layer skipped ~99% of the time early in training) and still lands at lower validation loss than the dense run.
•
The scaling factor 1/(1−p) applied to kept blocks is what makes optimal learning rate, batch size, and weight decay transfer across dropout rates. With scale factor 1, every rate needs its own hyperparameter sweep and “dropout hurts” conclusions follow.
•
Distribution-vs-schedule ablation: increasing-across-depth beats uniform and alternating; decreasing-across-time beats constant and increasing. An Alternating Layer Dropout (ALD) variant is best specifically for non-contiguous layer skipping at inference, so the right choice depends on how you’ll deploy.
Reach for this when you’re pretraining a foundation model you’ll later want to serve at multiple latency budgets, or use as its own speculative drafter. Instead of distilling separate small models or bolting on routers after the fact, apply increasing-across-depth layer dropout with a linear decay to zero over training, using per-sequence masks and the 1/(1−p) scale on kept blocks. Downstream, you get zero-shot early exit, tolerance to skipping alternate layers, and a drafter for Self-Speculative Decoding built into the same weights, along with cheaper training.
The paper does not link a released code repo or checkpoints. All pretraining runs were done on Cerebras CS-3 hardware, which matters because per-sequence structured sparsity translating cleanly to wall-clock savings depends on your kernel stack. The architecture recipe (271M through 8.2B) follows the Celerity Models family; hyperparameter transfer uses muP (Maximal Update Parameterization) and CompleteP.
Bring layer dropout back, but treat it as a depth-elasticity training signal, not a regularizer, and scale kept blocks by 1/(1−p) so your optimizer settings survive. The payoff isn’t a lower loss curve, it’s that the finished model runs at variable depth for free, which is what makes speculative decoding and elastic serving actually work on your dense checkpoint.
•
The dense-loss-neutral-or-better claim rests on the ILD+DTS configuration with the specific 1/(1−p) scaling; naive layer dropout at a constant high rate still degrades loss, which is likely why prior LLM recipes abandoned it.
•
FLOPs savings only convert to wall-clock savings if your training stack can actually skip the block computation for masked sequences. Not every framework does this cleanly, especially with per-sequence masks inside a compute-bound batch.
•
All experiments are dense decoder-only transformers up to 8.2B on Cerebras hardware. The paper explicitly does not test MoE, alternative architectures, or GPU-cluster kernel efficiency, and does not derive a scaling law for the maximum safe p_max at a given model and token budget.