---
title: "Transformers without Tears: Improving the Normalization of"
description: "The authors first build strong bilingual baselines and then add the modifications. 32.8 BLEU on IWSLT. 43 references, cited by 73 papers here."
authors: ["Toan Q. Nguyen", "Julian Salazar"]
year: 2019
venue: "arXiv:1910.05895"
source: "https://arxiv.org/abs/1910.05895"
doi: "10.5281/zenodo.3525484"
arxiv: "1910.05895"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2019-transformers-tears-improving-normalization-self/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2019-transformers-tears-improving-normalization-self/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Transformers without Tears: Improving the Normalization of Self-Attention

# Transformers without Tears: Improving the Normalization of Self-Attention

**Authors:** [Toan Q. Nguyen](https://md.research.elamur.ai/free/en/authors/toan-q-nguyen/) and [Julian Salazar](https://md.research.elamur.ai/free/en/authors/julian-salazar/) · **Year:** 2019 · **Venue:** arXiv:1910.05895 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.5281/zenodo.3525484) · [PDF](https://arxiv.org/pdf/1910.05895) · [DOI 10.5281/zenodo.3525484](https://doi.org/10.5281/zenodo.3525484) · [arXiv 1910.05895](https://arxiv.org/abs/1910.05895) · [OpenAlex](https://openalex.org/W2979636403)

**Citations:** 116 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/nguyen-2019-transformers-tears-improving-normalization-self/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines how normalization choices affect Transformer training for neural machine translation, especially when parallel data is scarce. The authors test three lightweight modifications. The first is PRENORM, which moves layer normalization in front of each sublayer instead of after the residual addition, combined with a smaller initialization of the attention weights. The second is SCALENORM, a new ℓ2-based normalization that rescales each activation vector to one learned length g. The third is FIXNORM, which constrains word embeddings to a fixed norm. Experiments cover five low-resource pairs drawn from the TED Talks corpus and IWSLT '15. The authors first build strong bilingual baselines and then add the modifications. Together, the changes make training converge reliably, remove the need for warmup, tolerate larger learning rates, and give consistent BLEU improvements. The work matters because it shows that small normalization adjustments can stabilize and speed up Transformer training on limited data. In the high-resource WMT '14 English-German setting, however, PRENORM hurts quality, while SCALENORM and FIXNORM remain on par. Code and preprocessing scripts are publicly released.

## Key points
- **Data and baselines:** Language pairs are gl→en, sk→en, en→he and ar→en from TED Talks (Qi et al., 2018), plus en→vi from IWSLT '15. Training sets range from about 10k to 214k sentence pairs. The tuned baselines already beat the best published bitext-only NMT results by about +4.0 BLEU on average.
- **Overall gains:** PRENORM + FIXNORM + SCALENORM add an average of +1.1 BLEU over these baselines. Each component contributes at least +0.3 BLEU. The combined system sets a new 32.8 BLEU on IWSLT '15 English-Vietnamese.
- **Initialization finding:** With default Xavier normal initialization, POSTNORM failed to converge on en→vi, reaching only 5.76 BLEU even with 16k warmup steps. Shrinking the attention-weight initialization (SMALLINIT) restored convergence, while PRENORM trained successfully under every configuration tested.
- **SCALENORM design:** It replaces LAYERNORM's 2d scale and shift parameters with a single scalar g. It performed comparably to or better than RMSNORM.
- **Learned g values:** The learned g tends to increase with layer depth and grows almost linearly across decoder layers. Removing label smoothing changes g in the upper decoder layers, which the authors link to overconfident output logits.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 128 works · 41,697 citations
- [University of Notre Dame](https://md.research.elamur.ai/free/en/universities/university-of-notre-dame/) — University, Notre Dame, United States · 59 works · 37,127 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Toan Q. Nguyen and Julian Salazar. 2019. Transformers without Tears: Improving the Normalization of Self-Attention. arXiv:1910.05895.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{nguyen2019transformers,
  title = {{Transformers without Tears: Improving the Normalization of Self-Attention}},
  author = {Toan Q. Nguyen and Julian Salazar},
  year = {2019},
  doi = {10.5281/zenodo.3525484},
  eprint = {1910.05895},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.5281/zenodo.3525484}
}
```

- [← previous Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation](https://md.research.elamur.ai/free/en/papers/loureiro-2019-language-modelling-makes-sense-propagating/)
- [next → compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://md.research.elamur.ai/free/en/papers/neubig-2019-compare-mt-tool-holistic-comparison/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/). In NAACL-HLT, pages 3874–3884.
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2015. [Layer Normalization](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/). CoRR, abs/1607.06450.
3. Mauro Cettolo, Jan Niehues, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. 2015. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Jan+Niehues%2C+Luisa+Bentivogli%2C+Roldano+Cattoni%2C+and+Marcello+Federico%2E+2015%2E)
4. The IWSLT 2015 Evaluation Campaign. In IWSLT, pages 3–4. [find](https://scholar.google.com/scholar?q=The+IWSLT+2015+Evaluation+Campaign%2E+In+IWSLT%2C+pages+3%E2%80%934%2E)
5. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Noam Shazeer, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. [The best of both worlds: Combining recent advances in ne…](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/)
6. Linhao Dong, Shuang Xu, and Bo Xu. 2018. Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition. In ICASSP, pages 5884–5888. [find](https://scholar.google.com/scholar?q=Linhao+Dong%2C+Shuang+Xu%2C+and+Bo+Xu%2E+2018%2E+Speech%2DTransformer%3A+A+No%2DRecurrence+Sequence%2Dto%2DSequence+Model+for+Speech+Recognition%2E+In+ICASSP%2C+pages+5884%E2%80%935888%2E)
7. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249–256. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+AISTATS%2C+pages+249%E2%80%93256%2E)
8. Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-Transformer. In NAACL-HLT, pages 1315–1325. [find](https://scholar.google.com/scholar?q=Qipeng+Guo%2C+Xipeng+Qiu%2C+Pengfei+Liu%2C+Yunfan+Shao%2C+Xiangyang+Xue%2C+and+Zheng+Zhang%2E+2019%2E+Star%2DTransformer%2E+In+NAACL%2DHLT%2C+pages+1315%E2%80%931325%2E)
9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In CVPR, pages 770–778.
10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b. Identity Mappings in Deep Residual Networks. In ECCV, pages 630–645. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016b%2E+Identity+Mappings+in+Deep+Residual+Networks%2E+In+ECCV%2C+pages+630%E2%80%93645%2E)
11. Felix Hieber, Tobias Domhan, Michael Denkowski, David Vilar, Artem Sokolov, Ann Clifton, and Matt Post. 2018. The Sockeye neural machine translation toolkit. In AMTA, pages 200–207. [find](https://scholar.google.com/scholar?q=Felix+Hieber%2C+Tobias+Domhan%2C+Michael+Denkowski%2C+David+Vilar%2C+Artem+Sokolov%2C+Ann+Clifton%2C+and+Matt+Post%2E+2018%2E+The+Sockeye+neural+machine+translation+toolkit%2E+In+AMTA%2C+pages+200%E2%80%93207%2E)
12. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, pages 448–456. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+2015%2E+Batch+Normalization%3A+Accelerating+Deep+Network+Training+by+Reducing+Internal+Covariate+Shift%2E+In+ICML%2C+pages+448%E2%80%93456%2E)
13. Diederik P Kingma and Jimmy Ba. 2015. [Adam: A Method for Stochastic Optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In ICLR.
14. Philipp Koehn. 2004. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In EMNLP, pages 388–395.
15. Wouter Kool, Herke Van Hoof, and Max Welling. 2019. Attention, Learn to Solve Routing Problems! In ICLR. [find](https://scholar.google.com/scholar?q=Wouter+Kool%2C+Herke+Van+Hoof%2C+and+Max+Welling%2E+2019%2E+Attention%2C+Learn+to+Solve+Routing+Problems%21+In+ICLR%2E)
16. Davis Liang, Zhiheng Huang, and Zachary C. Lipton. 2018. Learning noise-invariant representations for robust speech recognition. In SLT, pages 56–63. [find](https://scholar.google.com/scholar?q=Davis+Liang%2C+Zhiheng+Huang%2C+and+Zachary+C%2E+Lipton%2E+2018%2E+Learning+noise%2Dinvariant+representations+for+robust+speech+recognition%2E+In+SLT%2C+pages+56%E2%80%9363%2E)
17. Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. CoRR, abs/1908.03265. [find](https://scholar.google.com/scholar?q=Liyuan+Liu%2C+Haoming+Jiang%2C+Pengcheng+He%2C+Weizhu+Chen%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Jiawei+Han%2E+2019%2E+On+the+variance+of+the+adaptive+learning+rate+and+beyond%2E+CoRR%2C+abs%2F1908%2E03265%2E)
18. Chunjie Luo, Jianfeng Zhan, Lei Wang, and Qiang Yang. 2018. Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks. In ICANN, pages 382–391. [find](https://scholar.google.com/scholar?q=Chunjie+Luo%2C+Jianfeng+Zhan%2C+Lei+Wang%2C+and+Qiang+Yang%2E+2018%2E+Cosine+Normalization%3A+Using+Cosine+Similarity+Instead+of+Dot+Product+in+Neural+Networks%2E+In+ICANN%2C+pages+382%E2%80%93391%2E)
19. Rafael Müller, Simon Kornblith, and Geoffrey Hinton. 2019. When Does Label Smoothing Help? In NeurIPS. [find](https://scholar.google.com/scholar?q=Rafael+Mu%CC%88ller%2C+Simon+Kornblith%2C+and+Geoffrey+Hinton%2E+2019%2E+When+Does+Label+Smoothing+Help%3F+In+NeurIPS%2E)
20. Graham Neubig and Junjie Hu. 2018. [Rapid Adaptation of Neural Machine Translation to New Languages](https://md.research.elamur.ai/free/en/papers/neubig-2018-rapid-adaptation-neural-machine-translation/). In EMNLP, pages 875–880.
21. Toan Nguyen and David Chiang. 2018. Improving Lexical Choice in Neural Machine Translation. In NAACL-HLT, pages 334–343. [find](https://scholar.google.com/scholar?q=Toan+Nguyen+and+David+Chiang%2E+2018%2E+Improving+Lexical+Choice+in+Neural+Machine+Translation%2E+In+NAACL%2DHLT%2C+pages+334%E2%80%93343%2E)
22. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In NAACL-HLT (Demonstrations), pages 48–53.
23. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. [Scaling Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/). In WMT, pages 1–9.
24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL, pages 311– 318.
25. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In ICML, pages 1310–1318. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+ICML%2C+pages+1310%E2%80%931318%2E)
26. Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. [Regularizing neural networks by penalizing confident output distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/). In ICLR (Workshop).
27. Martin Popel and Ondej Bojar. 2018. [Training Tips for the Transformer Model](https://md.research.elamur.ai/free/en/papers/popel-2018-training-tips-transformer-model/). Prague Bull. Math. Linguistics, 110(1):43–70.
28. Ofir Press and Lior Wolf. 2017. [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In EACL, pages 157–163.
29. Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. [When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?](https://md.research.elamur.ai/free/en/papers/qi-2018-when-why-are-pre-trained/) In NAACL-HLT, pages 529–535.
30. Julian Salazar, Katrin Kirchhoff, and Zhiheng Huang. 2019. Self-attention Networks for Connectionist Temporal Classification in Speech Recognition. In ICASSP, pages 7115–7119. [find](https://scholar.google.com/scholar?q=Julian+Salazar%2C+Katrin+Kirchhoff%2C+and+Zhiheng+Huang%2E+2019%2E+Self%2Dattention+Networks+for+Connectionist+Temporal+Classification+in+Speech+Recognition%2E+In+ICASSP%2C+pages+7115%E2%80%937119%2E)
31. Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. 2018. How does batch normalization help optimization? In NeurIPS, pages 2483– 2493. [find](https://scholar.google.com/scholar?q=Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Andrew+Ilyas%2C+and+Aleksander+Madry%2E+2018%2E+How+does+batch+normalization+help+optimization%3F+In+NeurIPS%2C+pages+2483%E2%80%93+2493%2E)
32. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. [Edinburgh Neural Machine Translation Systems for WMT 16](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/). In WMT, pages 371–376.
33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In ACL.
34. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In ICML, pages 4603–4611. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+Learning+Rates+with+Sublinear+Memory+Cost%2E+In+ICML%2C+pages+4603%E2%80%934611%2E)
35. Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting Self-attention with Persistent Memory. CoRR, abs/1907.01470. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Edouard+Grave%2C+Guillaume+Lample%2C+Herve+Jegou%2C+and+Armand+Joulin%2E+2019%2E+Augmenting+Self%2Dattention+with+Persistent+Memory%2E+CoRR%2C+abs%2F1907%2E01470%2E)
36. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In CVPR, pages 2818–2826. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jon+Shlens%2C+and+Zbigniew+Wojna%2E+2016%2E+Rethinking+the+Inception+Architecture+for+Computer+Vision%2E+In+CVPR%2C+pages+2818%E2%80%932826%2E)
37. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. [Tensor2Tensor for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/). In AMTA, pages 193–199.
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is All you Need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NeurIPS, pages 5998–6008.
39. Andreas Veit, Michael Wilber, and Serge Belongie. 2016. Residual networks behave like ensembles of relatively shallow networks. NeurIPS, pages 550– 558. [find](https://scholar.google.com/scholar?q=Andreas+Veit%2C+Michael+Wilber%2C+and+Serge+Belongie%2E+2016%2E+Residual+networks+behave+like+ensembles+of+relatively+shallow+networks%2E+NeurIPS%2C+pages+550%E2%80%93+558%2E)
40. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. 2019. [Learning Deep Transformer Models for Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2019-learning-deep-transformer-models-machine/). In ACL, pages 1810–1822.
41. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. [Switchout: an efficient data augmentation algorithm for neural machine translation](https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/). In EMNLP, pages 856–861.
42. Biao Zhang and Rico Sennrich. 2019. Root Mean Square Layer Normalization. NeurIPS. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+2019%2E+Root+Mean+Square+Layer+Normalization%2E+NeurIPS%2E)
43. Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. 2019. Fixup initialization: Residual learning without normalization. In ICLR. [find](https://scholar.google.com/scholar?q=Hongyi+Zhang%2C+Yann+N%2E+Dauphin%2C+and+Tengyu+Ma%2E+2019%2E+Fixup+initialization%3A+Residual+learning+without+normalization%2E+In+ICLR%2E)

*Updated 2 November 2026*