---
title: "On the Mathematical Relationship Between Layer Normalization"
description: "Zhu et al. (2025) proposed Dynamic Tanh (DyT), an element-wise function with a learnable parameter, as a drop-in substitute. 14 references."
authors: ["Felix Stollenwerk"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.48/"
doi: "10.18653/v1/2026.eacl-short.48"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stollenwerk-2026-mathematical-relationship-between-layer-normalization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stollenwerk-2026-mathematical-relationship-between-layer-normalization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

# On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

**Authors:** Felix Stollenwerk · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.48/) · [PDF](https://aclanthology.org/2026.eacl-short.48.pdf) · [DOI 10.18653/v1/2026.eacl-short.48](https://doi.org/10.18653/v1/2026.eacl-short.48) · [OpenAlex](https://openalex.org/W7140146590)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
Layer normalization (LN) and RMSNorm are standard in transformers. Zhu et al. (2025) proposed Dynamic Tanh (DyT), an element-wise function with a learnable parameter, as a drop-in substitute. DyT works well empirically, but nobody had explained theoretically why it mimics normalization.

This short theoretical paper fills that gap starting from RMSNorm:
- **DyT derivation:** The author computes the Jacobian of RMSNorm and decouples it. Off-diagonal entries are discarded and the input norm is swapped for a global constant. Solving the resulting single differential equation gives a scaled DyT, so DyT turns out to be an approximation.
- **DyISRU derivation:** Applying the same decoupling directly to a rearranged RMSNorm formula removes the need for any approximation. The result is an exact element-wise counterpart called Dynamic Inverse Square Root Unit (DyISRU).

A synthetic outlier simulation shows that DyISRU reproduces how RMSNorm squashes outliers far more faithfully than DyT. The work gives dynamic activation functions a mathematical grounding and points to a potentially better replacement for normalization layers. However, no model-training experiments were run.

## Key points
- **Scaled DyT:** √C · tanh(αx_i). It follows from neglecting off-diagonal Jacobian terms and fixing the norm of x as a learnable constant.
- **DyISRU:** √C · x_i / √(β + x_i²). Here β globally stands in for the squared norm of all channels except the transformed one. The function is proportional to the ISRU activation (Carlile et al., 2017).
- **Simulation setup:** C = 100 channels drawn from N(0, 2²). The largest value was raised in steps of 5, up to S = 9 steps.
- **Fit on outliers:** Optimal parameters were α = 0.049 and β = 301.1. Mean absolute residuals were 0.33 for DyT versus \< 0.01 for DyISRU.
- **Venue, code and limitations:** EACL 2026 short paper. Code is at github.com/flxst/dynamic-activation-functions. Effects on downstream performance and on sensitivity to parameter initialization remain untested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [AI Sweden](https://ror.org/04d5vd162).

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Felix Stollenwerk. 2026. On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stollenwerk2026mathematical,
  title = {{On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions}},
  author = {Felix Stollenwerk},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.48},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.48}
}
```

- [← previous OjaKV: Context-Aware Online Low-Rank KV Cache Compression](https://md.research.elamur.ai/free/en/papers/zhu-2026-ojakv-context-aware-online-low/)
- [next → On-Policy Self-Distillation for Efficient Diffusion Language Models with Early-Stage Calibration](https://md.research.elamur.ai/free/en/papers/zhu-2026-policy-self-distillation-efficient-diffusion/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. [Layer normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/).
2. Andrew Brock, Soham De, and Samuel L. Smith. 2021a. Characterizing signal propagation to close the performance gap in unnormalized resnets. [find](https://scholar.google.com/scholar?q=Andrew+Brock%2C+Soham+De%2C+and+Samuel+L%2E+Smith%2E+2021a%2E+Characterizing+signal+propagation+to+close+the+performance+gap+in+unnormalized+resnets%2E)
3. Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. 2021b. High-performance large-scale image recognition without normalization. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1059–1071. PMLR. [find](https://scholar.google.com/scholar?q=Andy+Brock%2C+Soham+De%2C+Samuel+L+Smith%2C+and+Karen+Simonyan%2E+2021b%2E+High%2Dperformance+large%2Dscale+image+recognition+without+normalization%2E+In+Proceedings+of+the+38th+International+Conference+on+Machine+Le)
4. Brad Carlile, Guy Delamarter, Paul Kinney, Akiko Marti, and Brian Whitney. 2017. Improving deep learning by inverse square root linear units (isrlus). [find](https://scholar.google.com/scholar?q=Brad+Carlile%2C+Guy+Delamarter%2C+Paul+Kinney%2C+Akiko+Marti%2C+and+Brian+Whitney%2E+2017%2E+Improving+deep+learning+by+inverse+square+root+linear+units+%28isrlus%29%2E)
5. Soham De and Sam Smith. 2020. Batch normalization biases residual blocks towards the identity function in deep networks. In Advances in Neural Information Processing Systems, volume 33, pages 19964–19975. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Soham+De+and+Sam+Smith%2E+2020%2E+Batch+normalization+biases+residual+blocks+towards+the+identity+function+in+deep+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+19964%E2%80%9319)
6. Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. 2000. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13. [find](https://scholar.google.com/scholar?q=Charles+Dugas%2C+Yoshua+Bengio%2C+Fran%C3%A7ois+B%C3%A9lisle%2C+Claude+Nadeau%2C+and+Ren%C3%A9+Garcia%2E+2000%2E+Incorporating+second%2Dorder+functional+knowledge+for+better+option+pricing%2E+Advances+in+neural+information+processi)
7. Bobby He and Thomas Hofmann. 2024. Simplifying transformer blocks. [find](https://scholar.google.com/scholar?q=Bobby+He+and+Thomas+Hofmann%2E+2024%2E+Simplifying+transformer+blocks%2E)
8. Lei Huang, Jie Qin, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. 2023. Normalization techniques in training dnns: Methodology, analysis and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10173–10196. [find](https://scholar.google.com/scholar?q=Lei+Huang%2C+Jie+Qin%2C+Yi+Zhou%2C+Fan+Zhu%2C+Li+Liu%2C+and+Ling+Shao%2E+2023%2E+Normalization+techniques+in+training+dnns%3A+Methodology%2C+analysis+and+application%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+I)
9. Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In Advances in Neural Information Yunhao Ni, Yuxin Guo, Junlong Jia, and Lei Huang. 2024. On the nonlinearity of layer normalization. In Proceedings of the 41st International Conference o… [find](https://scholar.google.com/scholar?q=G%C3%BCnter+Klambauer%2C+Thomas+Unterthiner%2C+Andreas+Mayr%2C+and+Sepp+Hochreiter%2E+2017%2E+Self%2Dnormalizing+neural+networks%2E+In+Advances+in+Neural+Information+Yunhao+Ni%2C+Yuxin+Guo%2C+Junlong+Jia%2C+and+Lei+Huang%2E+202)
10. Sho Takase, Shun Kiyono, Sosuke Kobayashi, and Jun Suzuki. 2025. Spike no more: Stabilizing the pre-training of large language models. [find](https://scholar.google.com/scholar?q=Sho+Takase%2C+Shun+Kiyono%2C+Sosuke+Kobayashi%2C+and+Jun+Suzuki%2E+2025%2E+Spike+no+more%3A+Stabilizing+the+pre%2Dtraining+of+large+language+models%2E)
11. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine… [find](https://scholar.google.com/scholar?q=Ruibin+Xiong%2C+Yunchang+Yang%2C+Di+He%2C+Kai+Zheng%2C+Shuxin+Zheng%2C+Chen+Xing%2C+Huishuai+Zhang%2C+Yanyan+Lan%2C+Liwei+Wang%2C+and+Tieyan+Liu%2E+2020%2E+On+layer+normalization+in+the+transformer+architecture%2E+In+Proceed)
12. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+mean+square+layer+normalization%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+32%2E+Curran+Associates%2C+Inc%2E)
13. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. 2019. Fixup initialization: Residual learning without normalization. [find](https://scholar.google.com/scholar?q=Hongyi+Zhang%2C+Yann+N%2E+Dauphin%2C+and+Tengyu+Ma%2E+2019%2E+Fixup+initialization%3A+Residual+learning+without+normalization%2E)
14. Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, and Zhuang Liu. 2025. Transformers without normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Jiachen+Zhu%2C+Xinlei+Chen%2C+Kaiming+He%2C+Yann+LeCun%2C+and+Zhuang+Liu%2E+2025%2E+Transformers+without+normalization%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E)

*Updated 31 October 2026*