大模型架构的下半场

Mixture-of-Depths Attention

[Attention Residuals](https://aboard-hurricane-478.notion.site/Attention-Residuals-3591baf34fa5803ba92df3422b88e64a)

[mHC: Manifold-Constrained Hyper-Connections](https://aboard-hurricane-478.notion.site/mHC-Manifold-Constrained-Hyper-Connections-3591baf34fa580d69888dfa9c803e5f7)