---
title: "Transfer Learning across Low-Resource, Related — Nguyen 2017"
description: "The paper addresses neural machine translation (NMT) for language pairs with very little parallel data."
authors: ["Toan Q. Nguyen", "David Chiang"]
year: 2017
venue: "Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)"
source: "https://aclanthology.org/I17-2050/"
arxiv: "1708.09803"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

# Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

**Authors:** [Toan Q. Nguyen](https://md.research.elamur.ai/free/en/authors/toan-q-nguyen/), [David Chiang](https://md.research.elamur.ai/free/en/authors/david-chiang/) · **Year:** 2017 · **Venue:** Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/I17-2050/) · [PDF](https://aclanthology.org/I17-2050.pdf) · [arXiv 1708.09803](https://arxiv.org/abs/1708.09803) · [OpenAlex](https://openalex.org/W2752630748)

**Citations:** 158 citations (openalex-snapshot · 2026-10-14) · [127](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses neural machine translation (NMT) for language pairs with very little parallel data. It asks whether a parent model trained on a *related* and also low-resource pair can help. The authors extend the transfer learning approach introduced by Zoph et al. (2016), where a child model starts from the weights of a trained parent model. That method maps parent source embeddings to child words arbitrarily. Here, the parent and child instead share a source vocabulary, so a subword present in both languages keeps the embedding it learned in the parent.

To create this overlap, they apply BPE to the combined source and target data of both pairs, producing one joint vocabulary. Uyghur is first transliterated from Arabic to Latin script. Experiments use LORELEI data, with Uzbek-English as the parent and Turkish-English and Uyghur-English as children, and a Luong-style attentional model. Word-level transfer gave small or insignificant gains. BPE-based transfer beat a strong BPE baseline by 0.8 BLEU for Turkish and 4.3 BLEU for Uyghur. The approach looks promising for agglutinative language families with scarce data.

## Key points
- Setup: Uzbek-English parent model, with Turkish-English and Uyghur-English child models. All corpora come from the DARPA LORELEI program and were tokenized with Moses. The model is a 2-layer, 512-unit global attentional NMT (Luong et al., 2015).
- Test BLEU for Turkish-English: BPE baseline 12.4 vs. BPE+transfer 13.2 (p \< 0.05). Word-based transfer gave 8.1 → 8.5/8.6, which was not significant.
- Test BLEU for Uyghur-English: BPE baseline 11.1 vs. BPE+transfer 15.4 (p \< 0.01). Word-based results were 8.5 → 10.6 with fine-tuned target embeddings and 8.8 with frozen ones.
- Shared BPE sharply increases overlap: the share of child training source types also found in the parent rises from 3.9% to 58.8% for Turkish and from 0.5% to 57.2% for Uyghur. The authors credit this overlap for most of the gain.
- Training details: BPE merges were tuned between 5k and 60k operations. The training data was duplicated, with rare words replaced by UNK in one copy. Optimization used Adadelta, and decoding used beam search with length normalization (α = 0.8).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

127 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 116 not yet.

- [Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/)
- [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://md.research.elamur.ai/free/en/papers/communication-2023-seamless-multilingual-expressive-streaming-speech/)
- [Trivial Transfer Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/kocmi-2018-trivial-transfer-learning-low-resource/)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://md.research.elamur.ai/free/en/papers/lin-2019-choosing-transfer-languages-cross-lingual/)
- [Tao–Filipino Neural Machine Translation: Strategies for Ultra–Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/macayan-2026-tao-filipino-neural-machine-translation/)
- [Script Correction and Synthetic Pivoting: Adapting Tencent HY-MT for Low-Resource Turkic Translation](https://md.research.elamur.ai/free/en/papers/maxim-2026-script-correction-synthetic-pivoting-adapting/)
- [Rapid Adaptation of Neural Machine Translation to New Languages](https://md.research.elamur.ai/free/en/papers/neubig-2018-rapid-adaptation-neural-machine-translation/)
- [Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages](https://md.research.elamur.ai/free/en/papers/ramesh-2022-samanantar-largest-publicly-available-parallel/)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nguyen2017transfer,
  title = {{Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation}},
  author = {Nguyen, Toan Q. and Chiang, David},
  booktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2017},
  eprint = {1708.09803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.09803}
}
```

- [← previous Towards String-To-Tree Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2017-string-tree-neural-machine-translation/)
- [next → Translation Divergences in Chinese–English Machine Translation: An Empirical Investigation](https://md.research.elamur.ai/free/en/papers/deng-2017-translation-divergences-chinese-english-machine/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
2. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrie%CC%88nboer%2C+Dzmitry+Bahdanau%2C+and+Yoshua+Bengio%2E+2014%2E+On+the+properties+of+neural+machine+translation%3A+Encoder%E2%80%93decoder+approaches%2E+In+Proc%2E+Workshop+on+Syntax%2C+Semantics+and) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
3. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+et+al%2E+2007%2E+Moses%3A+Open+source+toolkit+fo)
4. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proc. First Workshop on Neural Machine Translation. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proc%2E+First+Workshop+on+Neural+Machine+Translation%2E+Association+for+Computational+Linguistics%2E)
5. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Thang+Luong%2C+Hieu+Pham%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proc%2E+EMNLP%2E)
6. Preslav Nakov and Hwee Tou Ng. 2009. Improved statistical machine translation for resource-poor languages using related resource-rich languages. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Preslav+Nakov+and+Hwee+Tou+Ng%2E+2009%2E+Improved+statistical+machine+translation+for+resource%2Dpoor+languages+using+related+resource%2Drich+languages%2E+In+Proc%2E+EMNLP%2E)
7. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proc%2E+ACL%2E)
8. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+27%2E)
9. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
10. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv:1212.5701v1. [find](https://scholar.google.com/scholar?q=Matthew+D%2E+Zeiler%2E+2012%2E+ADADELTA%3A+An+adaptive+learning+rate+method%2E+arXiv%3A1212%2E5701v1%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
11. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Deniz+Yuret%2C+Jonathan+May%2C+and+Kevin+Knight%2E+2016%2E+Transfer+learning+for+low%2Dresource+neural+machine+translation%2E+In+Proc%2E+EMNLP%2E)