Get Started
Home
Topics
Search
Library
Research questionHow can multi-stream Transformer residual mixing remain stable without amplifying signals or collapsing stream diversity?Unconstrained Hyper-Connections can amplify residual streams layer after layer, while constraints that prevent amplification may progressively erase differences between streams. The underlying challenge is controlling both norm growth and stream diversity during deep-network training.
AI
Machine Learning
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.oHC: Orthogonal Hyper-Connections on SO(4) via QuaternionsThe source studies Hyper-Connections and proposes orthogonal residual mixing in the four-stream case using a parameter-free unit-quaternion construction. It reports downstream-task evaluations against single-stream residuals, manifold-constrained mixing, and identity mixing, but the evidence is specific to the described Transformer settings.research paper · Sep 2, 2026
Related questions
Which routing and mixing behaviors in trained four-stream residual pathways materially affect model performance?How can complete transformer layers be removed from LLMs without sacrificing quality or standard serving compatibility?How can high-resolution diffusion Transformers prune tokens without sacrificing image fidelity or predictable compute?Does reusing Transformer layers improve language-model quality when parameter, compute, and KV-cache budgets are matched?