---
title: "Norm-Based Curriculum Learning for Neural Machine Translation"
description: "Training neural machine translation (NMT) systems is slow and expensive, especially deep or… 41 references, cited by 65 papers here."
authors: ["Xuebo Liu", "Houtim Lai", "Derek F. Wong", "Lidia S. Chao"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.41/"
doi: "10.18653/v1/2020.acl-main.41"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2020-norm-based-curriculum-learning-neural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2020-norm-based-curriculum-learning-neural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Norm-Based Curriculum Learning for Neural Machine Translation

# Norm-Based Curriculum Learning for Neural Machine Translation

**Authors:** [Xuebo Liu](https://md.research.elamur.ai/free/en/authors/xuebo-liu/), Houtim Lai, [Derek F. Wong](https://md.research.elamur.ai/free/en/authors/derek-f-wong/) and [Lidia S. Chao](https://md.research.elamur.ai/free/en/authors/lidia-s-chao/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.41/) · [PDF](https://aclanthology.org/2020.acl-main.41.pdf) · [DOI 10.18653/v1/2020.acl-main.41](https://doi.org/10.18653/v1/2020.acl-main.41) · [OpenAlex](https://openalex.org/W3034938700)

**Citations:** 93 citations (OpenAlex, 2026-10-14) · 65 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Training neural machine translation (NMT) systems is slow and expensive, especially deep or large-batch Transformer models on high-resource data. Curriculum learning (CL) can help by presenting easier examples before harder ones. However, existing difficulty measures are either cheap but shallow (sentence length, word rarity) or informative but costly (uncertainty from auxiliary models). Competence-based CL also requires a task-specific curriculum length, which is tuned using a full baseline run.

The authors use vector norms in three ways:
- **Sentence difficulty:** the sum of word-vector norms from a simple word embedding model trained on the source corpus. This captures both rarity and context-specific "significant" words.
- **Model competence:** the growing norm of the NMT source embedding, which lets the curriculum set its own pace.
- **Sentence weights:** norm-based weights that rescale the loss to reduce over-training on easy data.

On WMT'14 English–German and WMT'17 Chinese–English, the approach beats strong Transformer baselines in both BLEU and training speed. It changes only the sampling pipeline and objective, adding no parameters. This makes it practical for cutting the cost of large NMT training.

## Key points
- **Main results:** +1.17 BLEU on WMT'14 English–German and +1.56 BLEU on WMT'17 Chinese–English, with training speedups of 2.22x and 3.33x respectively.
- **Difficulty measure:** sentence difficulty is d(x) = Σ‖xᵢ‖ over embeddings from a lightweight word embedding model (E_w2v) that trains quickly on CPU. On WMT'14 En–De source data, rare and context-insensitive words receive higher norms.
- **Competence schedule:** the method builds on the competence-based CL of Platanios et al. (2019), using the square-root competence function with c₀ = 0.01. It replaces the task-dependent length hyperparameter λₜ, normally set from the step where a baseline reaches 90% of its final BLEU, with the norm of the NMT source embedding E_nmt. The motivation is that this norm and dev-set BLEU both rise sharply until about 30K steps.
- **Effect of sentence weights:** the En–De test set (3,003 sentences) was split into three 1,001-sentence subsets by length-, frequency- and norm-based difficulty. Plain competence-based CL did better on easy sentences but worse on medium ones. Norm-based sentence weights removed this bias and also improved competence-based CL when combined with it.
- **Case study:** on a hard multi-clause sentence, the vanilla Transformer dropped the final clause, while the norm-based CL model translated the whole sentence.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 10 works · 1,905 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Xuebo Liu, Houtim Lai, Derek F. Wong and Lidia S. Chao. 2020. Norm-Based Curriculum Learning for Neural Machine Translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2020norm,
  title = {{Norm-Based Curriculum Learning for Neural Machine Translation}},
  author = {Xuebo Liu, Houtim Lai, Derek F. Wong and Lidia S. Chao},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.41},
  url = {https://doi.org/10.18653/v1/2020.acl-main.41}
}
```

- [← previous A Sentiment Analysis Dataset for Code-Mixed Malayalam-English](https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-sentiment-analysis-dataset-code-mixed/)
- [next → Understanding tables with intermediate pre-training](https://md.research.elamur.ai/free/en/papers/eisenschlos-2020-understanding-tables-intermediate-pre-training/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

41 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR 2015.
2. Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. 2018. Training Deeper Neural Machine Translation Models with Transparent Attention. In EMNLP 2018. [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Mia+Xu+Chen%2C+Orhan+Firat%2C+Yuan+Cao%2C+and+Yonghui+Wu%2E+2018%2E+Training+Deeper+Neural+Machine+Translation+Models+with+Transparent+Attention%2E+In+EMNLP+2018%2E)
3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. [Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/). In ICML 2009.
4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). In TACL 2017.
5. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP 2018. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+EMNLP+2018%2E)
6. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. [Representation degeneration problem in training natural language generation models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/). In ICLR 2019.
7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In ICML 2017.
8. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. [Mask-Predict: Parallel Decoding of Conditional Masked Language Models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/). In EMNLP 2019.
9. Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. [find](https://scholar.google.com/scholar?q=Chengyue+Gong%2C+Di+He%2C+Xu+Tan%2C+Tao+Qin%2C+Liwei+Wang%2C+and+Tie%2DYan+Liu%2E+2018%2E)
10. FRAGE: Frequency-Agnostic Word Representation. In NIPS 2018. [find](https://scholar.google.com/scholar?q=FRAGE%3A+Frequency%2DAgnostic+Word+Representation%2E+In+NIPS+2018%2E)
11. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. [Marian: Fast neural machine translation in C++](https://md.research.elamur.ai/free/en/papers/junczys-dowmunt-2018-marian-fast-neural-machine-translation/). In ACL 2018.
12. Adriaan M J Schakel and Benjamin J Wilson. 2015. Measuring Word Significance using Distributed Representations of Words. arXiv. [find](https://scholar.google.com/scholar?q=Adriaan+M+J+Schakel+and+Benjamin+J+Wilson%2E+2015%2E+Measuring+Word+Significance+using+Distributed+Representations+of+Words%2E+arXiv%2E)
13. Nal Kalchbrenner and Phil Blunsom. 2013. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In EMNLP 2013.
14. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR+2015%2E)
15. Tom Kocmi and Ondrej Bojar. 2017. Curriculum Learning and Minibatch Bucketing in Neural Machine Translation. In RANLP 2017. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Ondrej+Bojar%2E+2017%2E+Curriculum+Learning+and+Minibatch+Bucketing+in+Neural+Machine+Translation%2E+In+RANLP+2017%2E)
16. Gaurav Kumar, George Foster, Colin Cherry, and Maxim Krikun. 2019. Reinforcement learning based curriculum optimization for neural machine translation. In NAACL 2019. [find](https://scholar.google.com/scholar?q=Gaurav+Kumar%2C+George+Foster%2C+Colin+Cherry%2C+and+Maxim+Krikun%2E+2019%2E+Reinforcement+learning+based+curriculum+optimization+for+neural+machine+translation%2E+In+NAACL+2019%2E)
17. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In ACL 2016.
18. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NIPS 2014. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NIPS+2014%2E)
19. Joseph P Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. [Word representations: A simple and general method for semi-supervised learning](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/). In ACL 2010.
20. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. [A structured self-attentive sentence embedding](https://md.research.elamur.ai/free/en/papers/lin-2017-structured-self-attentive-sentence-embedding/). In ICLR 2017.
21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS 2017.
22. Xuebo Liu, Derek F. Wong, Lidia S. Chao, and Yang Liu. 2019a. Latent attribute based hierarchical decoder for neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(12):2103–2112. [find](https://scholar.google.com/scholar?q=Xuebo+Liu%2C+Derek+F%2E+Wong%2C+Lidia+S%2E+Chao%2C+and+Yang+Liu%2E+2019a%2E+Latent+attribute+based+hierarchical+decoder+for+neural+machine+translation%2E+IEEE%2FACM+Transactions+on+Audio%2C+Speech%2C+and+Language+Processin)
23. Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. 2019. [Learning deep transformer models for machine translation](https://md.research.elamur.ai/free/en/papers/wang-2019-learning-deep-transformer-models-machine/). In ACL 2019.
24. Xuebo Liu, Derek F. Wong, Yang Liu, Lidia S. Chao, Tong Xiao, and Jingbo Zhu. 2019b. Shared-private bilingual word embeddings for neural machine translation. In ACL 2019. [find](https://scholar.google.com/scholar?q=Xuebo+Liu%2C+Derek+F%2E+Wong%2C+Yang+Liu%2C+Lidia+S%2E+Chao%2C+Tong+Xiao%2C+and+Jingbo+Zhu%2E+2019b%2E+Shared%2Dprivate+bilingual+word+embeddings+for+neural+machine+translation%2E+In+ACL+2019%2E)
25. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159–165. [find](https://scholar.google.com/scholar?q=Hans+Peter+Luhn%2E+1958%2E+The+automatic+creation+of+literature+abstracts%2E+IBM+Journal+of+Research+and+Development%2C+2%282%29%3A159%E2%80%93165%2E)
26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). In ICLR 2013.
27. Toan Nguyen and David Chiang. 2019. Improving lexical choice in neural machine translation. In NAACL-HLT 2019. [find](https://scholar.google.com/scholar?q=Toan+Nguyen+and+David+Chiang%2E+2019%2E+Improving+lexical+choice+in+neural+machine+translation%2E+In+NAACL%2DHLT+2019%2E)
28. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. [Scaling neural machine translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/). In WMT@EMNLP 2018.
29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL 2002.
30. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. [A decomposable attention model for natural language inference](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/). In EMNLP 2016.
31. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabás Póczos, and Tom M. Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In NAACL 2019. [find](https://scholar.google.com/scholar?q=Emmanouil+Antonios+Platanios%2C+Otilia+Stretcu%2C+Graham+Neubig%2C+Barnaba%CC%81s+Po%CC%81czos%2C+and+Tom+M%2E+Mitchell%2E+2019%2E+Competence%2Dbased+curriculum+learning+for+neural+machine+translation%2E+In+NAACL+2019%2E)
32. Ofir Press and Lior Wolf. 2017. [Using the Output Embedding to Improve Language Models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In EACL 2017.
33. Benjamin J Wilson and Adriaan M J Schakel. 2015. Controlled Experiments for Word Embeddings. arXiv. [find](https://scholar.google.com/scholar?q=Benjamin+J+Wilson+and+Adriaan+M+J+Schakel%2E+2015%2E+Controlled+Experiments+for+Word+Embeddings%2E+arXiv%2E)
34. Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2019. Depth growing for neural machine translation. In ACL 2019. [find](https://scholar.google.com/scholar?q=Lijun+Wu%2C+Yiren+Wang%2C+Yingce+Xia%2C+Fei+Tian%2C+Fei+Gao%2C+Tao+Qin%2C+Jianhuang+Lai%2C+and+Tie%2DYan+Liu%2E+2019%2E+Depth+growing+for+neural+machine+translation%2E+In+ACL+2019%2E)
35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
36. Biao Zhang, Ivan Titov, and Rico Sennrich. 2019a. Improving deep transformer with depth-scaled initialization and merged attention. In EMNLP 2019. [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Ivan+Titov%2C+and+Rico+Sennrich%2E+2019a%2E+Improving+deep+transformer+with+depth%2Dscaled+initialization+and+merged+attention%2E+In+EMNLP+2019%2E)
37. Dakun Zhang, Jungi Kim, Josep Crego, and Jean Senellart. 2017. [find](https://scholar.google.com/scholar?q=Dakun+Zhang%2C+Jungi+Kim%2C+Josep+Crego%2C+and+Jean+Senellart%2E+2017%2E)
38. Boosting Neural Machine Translation. In IJCNLP 2017. [find](https://scholar.google.com/scholar?q=Boosting+Neural+Machine+Translation%2E+In+IJCNLP+2017%2E)
39. Xuan Zhang, Gaurav Kumar, Huda Khayrallah, Kenton Murray, Jeremy Gwinnup, Marianna J Martindale, Paul McNamee, Kevin Duh, and Marine Carpuat. 2018. An Empirical Exploration of Curriculum Learning for Neural Machine Translation. arXiv. [find](https://scholar.google.com/scholar?q=Xuan+Zhang%2C+Gaurav+Kumar%2C+Huda+Khayrallah%2C+Kenton+Murray%2C+Jeremy+Gwinnup%2C+Marianna+J+Martindale%2C+Paul+McNamee%2C+Kevin+Duh%2C+and+Marine+Carpuat%2E+2018%2E+An+Empirical+Exploration+of+Curriculum+Learning+for+)
40. Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul McNamee, Marine Carpuat, and Kevin Duh. 2019b. [Curriculum Learning for Domain Adaptation in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-curriculum-learning-domain-adaptation-neural/). In NAACL 2019.
41. Yikai Zhou, Baosong Yang, Derek F. Wong, Yu Wan, and Lidia S. Chao. 2020. Uncertainty-aware curriculum learning for neural machine translation. In ACL 2020. [find](https://scholar.google.com/scholar?q=Yikai+Zhou%2C+Baosong+Yang%2C+Derek+F%2E+Wong%2C+Yu+Wan%2C+and+Lidia+S%2E+Chao%2E+2020%2E+Uncertainty%2Daware+curriculum+learning+for+neural+machine+translation%2E+In+ACL+2020%2E)

*Updated 24 October 2026*