Under a fixed parameter budget, whether making a Transformer deeper-and-narrower actually helps depends almost entirely on the residual connection design: swapping the standard Pre-LN Transformer residual for Hyper-Connections (HC) or AttnRes lets loss keep dropping down to aspect ratios as extreme as 9.1 (70 layers wide 640), while Pre-LN prefers shallow-wide shapes.
Say you’re picking a shape for a ~400M or ~1.6B language model and you have a fixed parameter budget. Conventional wisdom (and Kaplan-style scaling work) says width vs depth barely matters once size is fixed, so people default to shallow-wide shapes like 24 layers by 1024 hidden. But recent work on the curse of depth has shown that in standard Pre-LN Transformers, later layers contribute progressively less: their updates get weak, redundant, and similar to their neighbors. So stacking more layers gives you nominal depth without much extra useful computation.
A growing catalog of fixes has appeared (new normalizations, multi-stream residuals, cross-layer attention), and some are being adopted in frontier models. But each is reported under a different training recipe and different shapes, so you can’t tell whether the gains come from better optimization, better depth utilization, or just from being tuned harder. The practical question the paper tackles: if I actually want depth to be a useful scaling knob, which residual design should I use, and how do I know it’s converting layers into computation rather than dead weight?
The contribution is a controlled benchmark, DepthBench, not a new architecture. The setup: fix the parameter budget (roughly 400M, plus 200M–500M and 1.6B validation suites), fix a LLaMA-style backbone (RoPE, SwiGLU, RMSNorm), fix the training recipe on FineWeb-Edu with ~20 tokens per parameter, then sweep the aspect ratio (hidden dimension divided by layer count) from 76.0 (16 layers wide) down to 9.1 (70 layers narrow). Ten residual designs are compared: Pre-LN baseline, four normalization variants (Sandwich-LN, LayerNorm Scaling (LNS), DeepNorm, KEEL), two multi-stream residual schemes (Hyper-Connections and its constrained variant Manifold-Constrained Hyper-Connections (mHC)), two cross-layer attention residuals (Full and Block AttnRes), and MoDA (Mixture-of-Depth Attention).
Critically, they sweep learning rate per architecture and report each shape at its best LR, so a bad LR can’t be blamed for a bad shape. The evaluation is validation loss plus teacher-forced NLL on coding (HumanEval, MBPP), STEM, and math (GSM8K, MATH-500) tasks. Then they run three layer-level diagnostics to ask whether depth is actually being used: angular distance between layer representations (are deep layers all doing the same thing?), a causal score (how much does skipping layer s perturb a later layer?), and a permutation score (does swapping two layers hurt the loss?). High scores on the latter two mean layers are doing distinct, order-dependent work.
The benchmark loop, roughly:
for arch in architectures: # 10 residual designs
lr = best_lr_for(arch) # from per-arch sweep
for (L, d) in shapes: # fix ~400M params
model = build(arch, L, d)
train(model, fineweb_edu, tokens=20*params, lr=lr)
record(val_loss, nll_by_domain)
record(angular_dist, causal_score, perm_score)
rank_architectures_by_depth_scaling()
At 400M, Pre-LN loss gets worse as you go deeper: 2.759 at L=16 rises to 2.782 at L=32. All four normalization variants (Sandwich-LN, LNS, DeepNorm, KEEL) and MoDA show similarly flat or non-monotonic curves, with optima at aspect ratios in the 42.7–76.0 range. Full AttnRes and HC show the opposite pattern: Full AttnRes drops from 2.751 to 2.718, HC drops from 2.729 to 2.699 across the same range, and both keep improving out to L=70 (aspect ratio 9.1) under both fixed-total-params and fixed-backbone-params controls. The trend replicates on the 200M–500M ladder and on 1.6B for Full AttnRes; HC is less stable at 1.6B (one run hit gradient explosion), which the authors link to its optimization sensitivity. The domain-specific NLL results agree, particularly on STEM and math.
The derived variants disappoint: Block AttnRes peaks at L=20 then degrades, and mHC peaks at L=24. The layer diagnostics on L=32 models tell a consistent story. For Pre-LN, only about 1% of late-layer pairs have a causal score above 0.45 and only ~8% exceed the permutation threshold. For Full AttnRes it’s roughly 16% and 46%; for HC, 7% and 35%. Angular-distance maps show Pre-LN and its normalization cousins collapse toward smooth, uniformly similar deep-layer representations, while HC and AttnRes keep heterogeneous, layer-specific transformations. So the winners aren’t just training better, their later layers are doing genuinely different, order-dependent work.
The authors also diagnose why the derived variants fail. Block AttnRes freezes sub-layer outputs into a running sum at block boundaries; before the first boundary, the softmax puts near-zero weight on the running sum, so the network effectively re-reads the embedding. That dead segment grows with depth (up to 8 layers, a quarter of L=32). mHC’s Sinkhorn-normalized doubly-stochastic constraint makes its composed residual maps concentrate toward the uniform mixing matrix; effective rank of the propagated stream is 1.44–1.65 versus 2.53–2.81 for HC, so different long-range routes collapse into essentially the same average.
There is a real cost. As aspect ratio drops from 76.0 to 9.1, prefill FLOPs go from ~3.0 to ~4.4 TFLOPs per sequence, KV cache more than doubles, and measured GPU-hours on A100s rise sharply. Full AttnRes has the largest peak memory (it stores per-sublayer sources); HC has the highest wall-clock cost despite modest FLOP growth, because deeper models mean smaller matmuls and more sequential execution.
•
If you’re picking a shape for a Pre-LN-family model today, the paper is direct evidence that going deeper-narrower under fixed params will probably hurt you. Stick near aspect ratios of 40–76, which matches what most released industry models already do.
•
If you want depth to actually pay off, HC and Full AttnRes are the two designs the evidence supports. Full AttnRes has cleaner scaling behavior and held up at 1.6B in this study; HC did better at 400M but showed optimizer instability at 1.6B, so budget time for LR tuning if you use it. The paper does not claim either is production-ready at the scales frontier labs train at.
•
Be skeptical of derived variants that look like efficiency improvements over a working design. Block AttnRes and mHC each trade something (source access, stream diversity) that turns out to matter for depth scaling. Worth testing your own variants on a small aspect-ratio sweep before committing.
•
If you’re doing your own architecture comparisons, the DepthBench recipe (fix params, fix recipe, per-arch LR sweep, aspect-ratio sweep, plus causal/permutation/angular-distance diagnostics) is a reusable protocol. The layer-level diagnostics are the part worth borrowing even if you don’t run the full sweep: they tell you whether your deep layers are doing anything.
•
Before you commit to a deep-narrow architecture in production, price out the systems cost. KV cache growth is intrinsic to the shape, not the residual design, and wall-clock overhead can exceed the FLOP delta because of lower hardware utilization. The authors release code and checkpoints across the aspect-ratio range.
The main study is 400M with a 1.6B validation. HC’s 1.6B result did not clearly improve with depth and one run diverged, so the strong depth-scaling claim for HC is best supported at smaller scales. Aspect ratios below ~9 and models above 1.6B are not tested. All comparisons hold the training recipe fixed on FineWeb-Edu at Chinchilla-optimal token counts; behavior under longer training, different data mixes, or post-training is not evaluated. “Effective computational depth” is inferred from layer-level diagnostics on validation data and predictive NLL on a handful of tasks, not from downstream fine-tuning or instruction-following benchmarks. Finally, causal and permutation scores use fixed thresholds (0.45 and 0.15); the qualitative gap between architectures is large enough that this probably does not flip the ranking, but the exact percentages are threshold-dependent.