---
title: "Copied Monolingual Data Improves Low-Resource — Currey 2017"
description: "Neural MT needs large amounts of parallel text, and most language pairs have very little."
authors: ["Anna Currey", "Antonio Valerio Miceli Barone", "Kenneth Heafield"]
year: 2017
venue: "Proceedings of the Second Conference on Machine Translation"
source: "https://aclanthology.org/W17-4715/"
doi: "10.18653/v1/w17-4715"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/currey-2017-copied-monolingual-data-improves-low/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/currey-2017-copied-monolingual-data-improves-low/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2017](https://md.research.elamur.ai/free/papers/2017/) / Copied Monolingual Data Improves Low-Resource Neural Machine Translation

# Copied Monolingual Data Improves Low-Resource Neural Machine Translation

**Authors:** [Anna Currey](https://md.research.elamur.ai/free/authors/anna-currey/), [Antonio Valerio Miceli Barone](https://md.research.elamur.ai/free/authors/antonio-valerio-miceli-barone/), [Kenneth Heafield](https://md.research.elamur.ai/free/authors/kenneth-heafield/) · **Year:** 2017 · **Venue:** Proceedings of the Second Conference on Machine Translation · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W17-4715/) · [PDF](https://aclanthology.org/W17-4715.pdf) · [DOI 10.18653/v1/w17-4715](https://doi.org/10.18653/v1/w17-4715) · [OpenAlex](https://openalex.org/W2756566411)

**Citations:** 208 citations (openalex-snapshot · 2026-10-14) · [88](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Neural MT needs large amounts of parallel text, and most language pairs have very little. Target-language monolingual text is plentiful, but back-translation, the standard way to use it, depends on a reverse-direction system that is weak when bitext is scarce. Currey, Miceli Barone and Heafield propose a simple alternative. They turn target-side monolingual sentences into pseudo-bitext by using each sentence as its own source. These copied pairs are then shuffled into the real parallel data, optionally alongside back-translated data. The model architecture, the training procedure and the data tags stay unchanged. A shared BPE vocabulary lets one encoder read both languages.

Experiments with Nematus attentional GRU models covered Turkish, Romanian and German paired with English, using WMT data. The copied data raised BLEU by up to 1.2 points over a back-translation baseline for the two low-resource pairs. It left high-resource EN↔DE essentially unchanged. The gains mostly came from named entities and words that should pass through untranslated, which makes the method complementary to back-translation. It is a cheap, architecture-free boost for under-resourced translation.

## Key points
- **Data sizes:**
  - EN↔TR: about 207k parallel and 415k monolingual sentences.
  - EN↔RO: 608,320 of each.
  - EN↔DE: about 5.85M parallel and 10M monolingual.
  - Monolingual text came from News Crawl 2015/2016. Parallel data came from WMT17 (TR, DE) and WMT16 (RO).
- **Notable BLEU gains:**
  - EN→TR: 12.8→14.0 (newstest2016) and 14.2→15.2 (2017).
  - RO→EN: 34.5→35.7.
  - EN→RO: 23.8→24.5.
  - EN↔DE: changes were not significant.
- **Model setup:** Nematus with a bidirectional GRU encoder, attention, 1024-dimensional hidden layers, 512-dimensional embeddings, Adam, and a shared BPE model with 90,000 merges.
- **Amount of monolingual data:** For EN→TR, monolingual-to-parallel ratios of 1:1, 2:1 and 3:1 were tested, with no oversampling of the bitext. The copied system stayed about 1.0 BLEU ahead of the baseline at every ratio. Adding monolingual data gave 0.2–0.6 BLEU for both systems.
- **Fluency check:** KenLM 5-gram models were used to measure perplexity, testing whether the gains reflect more than better fluency.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/)

Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/)

## Cited by

Cited by

88 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 82 not yet.

- [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/papers/artetxe-2017-unsupervised-neural-machine-translation/)
- [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/papers/chu-2018-survey-domain-adaptation-neural-machine/)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://md.research.elamur.ai/free/papers/khayrallah-2018-impact-various-types-noise-neural/)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/papers/kocmi-2018-trivial-transfer-learning-low-resource/)
- [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)
- [Handling Syntactic Divergence in Low-resource Machine Translation](https://md.research.elamur.ai/free/papers/zhou-2019-handling-syntactic-divergence-low-resource/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{currey2017copied,
  title = {{Copied Monolingual Data Improves Low-Resource Neural Machine Translation}},
  author = {Currey, Anna and Barone, Antonio Valerio Miceli and Heafield, Kenneth},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4715},
  url = {https://doi.org/10.18653/v1/w17-4715}
}
```

- [← previous Convolutional Sequence to Sequence Learning](https://md.research.elamur.ai/free/papers/gehring-2017-convolutional-sequence-sequence-learning/)
- [next → Corpus-Driven Visualization of Textual Worlds](https://md.research.elamur.ai/free/papers/hbo-2017-kofoed-corpus-driven-visualization-textual-worlds/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/papers/2017/)

## References

13 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+3rd+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turc… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Aure%CC%81lie+Ne%CC%81ve%CC%81ol) [our card](https://md.research.elamur.ai/free/papers/bojar-2016-findings-2016-conference-machine-translation/)
3. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1700–1709. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner+and+Phil+Blunsom%2E+2013%2E+Recurrent+continuous+translation+models%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1700%E2%80%931709%2E+Associatio)
4. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+3rd+International+Conference+on+Learning+Representations%2E)
5. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In 4th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2016%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+In+4th+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/papers/luong-2015-multi-task-sequence-sequence-learning/)
6. Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the ACL, pages 1965–1974. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yong+Cheng%2C+Wei+Xu%2C+Zhongjun+He%2C+Wei+He%2C+Hua+Wu%2C+Maosong+Sun%2C+and+Yang+Liu%2E+2016%2E+Semi%2Dsupervised+learning+for+neural+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+ACL%2C+pages+1)
7. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311– 318. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+of+the+ACL%2C+pages+311%E2%80%93+318%2E) [our card](https://md.research.elamur.ai/free/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
8. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmu… [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+Van+Merrie%CC%88nboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+RNN+encoder%2Ddecoder+Rico+Sennric) [our card](https://md.research.elamur.ai/free/papers/sennrich-2017-nematus-toolkit-neural-machine-translation/)
9. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation, pages 371–376. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Edinburgh+neural+machine+translation+systems+for+WMT+16%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%2C+pages+371%E2%80%93376%2E+Associat) [our card](https://md.research.elamur.ai/free/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
10. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Improving neural machine translation models with monolingual data. In Proceedings of NAACL-HLT, pages 86–96. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016b%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+86%E2%80%9396%2E+Association+for+Computational+Ling)
11. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016c. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL, pages 1715–1725. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016c%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+ACL%2C+pages+1715%E2%80%931725%2E+Association)
12. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+27%2E)
13. Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535–1545. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiajun+Zhang+and+Chengqing+Zong%2E+2016%2E+Exploiting+source%2Dside+monolingual+data+in+neural+machine+translation%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C) [our card](https://md.research.elamur.ai/free/papers/zhang-2016-exploiting-source-side-monolingual-data/)