---
title: "Semi-Supervised Learning for Neural Machine Translation (2016)"
description: "The authors propose a semi-supervised scheme that trains a source-to-target model and a target-to-source model… Cited by 76 papers here."
authors: ["Yong Cheng", "Wei Xu", "Zhongjun He", "Wei He", "Hua Wu", "Maosong Sun", "Yang Liu"]
year: 2016
venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P16-1185/"
doi: "10.18653/v1/p16-1185"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cheng-2016-semi-supervised-learning-neural-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cheng-2016-semi-supervised-learning-neural-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / Semi-Supervised Learning for Neural Machine Translation

# Semi-Supervised Learning for Neural Machine Translation

**Authors:** [Yong Cheng](https://md.research.elamur.ai/free/en/authors/yong-cheng/), [Wei Xu](https://md.research.elamur.ai/free/en/authors/wei-xu/), [Zhongjun He](https://md.research.elamur.ai/free/en/authors/zhongjun-he/), [Wei He](https://md.research.elamur.ai/free/en/authors/wei-he/), [Hua Wu](https://md.research.elamur.ai/free/en/authors/hua-wu/), [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) and [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/) · **Year:** 2016 · **Venue:** [Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2016](https://md.research.elamur.ai/free/en/venues/acl/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-1185/) · [PDF](https://aclanthology.org/P16-1185.pdf) · [DOI 10.18653/v1/p16-1185](https://doi.org/10.18653/v1/p16-1185) · [OpenAlex](https://openalex.org/W2963357083)

**Citations:** 87 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/cheng-2016-semi-supervised-learning-neural-machine/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Standard NMT systems learn only from parallel text. Parallel data is scarce, narrow in domain and uneven in quality, while monolingual text is plentiful. The authors propose a semi-supervised scheme that trains a source-to-target model and a target-to-source model together. The training objective adds autoencoder-style reconstruction terms for monolingual sentences to the usual likelihood on parallel data. A monolingual sentence is "encoded" by translating it into the other language, which acts as a latent sentence. It is then "decoded" back with the reverse model, and the reconstruction probability is maximized.

Summing over all possible latent translations is intractable, so the method approximates it with a top-k candidate list, where k=10 is enough. The approach does not depend on the network architecture and can use monolingual corpora in both the source and target languages.

On Chinese-English NIST data, it significantly outperforms strong SMT and NMT baselines in both directions. It also beats the back-translation method of Sennrich et al. (2015). This makes it a simple, architecture-agnostic way to make use of unlabeled text.

## Key points
- **Training objective.** The joint objective has four terms:
  - source-to-target likelihood,
  - target-to-source likelihood,
  - a target autoencoder,
  - a source autoencoder.
  
  The autoencoder terms are weighted by hyper-parameters λ1 and λ2.
- **Data.** The parallel Chinese-English corpus has 2.56M sentence pairs (67.54M Chinese and 74.82M English words). The monolingual corpora contain 18.75M Chinese and 22.32M English sentences.
- **Approximation and batching.** The intractable sum over latent translations is replaced by a top-k list of candidates, with k=10. Each SGD step combines a parallel mini-batch with one mini-batch from each monolingual corpus.
- **Which monolingual data helps.** Target-side monolingual data helps source-to-target translation more than source-side data. Using both together brought no further significant gain.
- **Comparison with back-translation.** The method gains up to +1.8 BLEU (Chinese-to-English) and +1.0 BLEU (English-to-Chinese) over Sennrich et al. (2015) using the same corpora.

## Interesting ideas
- **Translations as a discrete latent layer.** Treating a translation as the hidden layer of an autoencoder lets two opposite-direction models supervise each other on unlabeled text. The same idea could apply to any task with a natural inverse, such as transliteration or text normalization.
- **Iterative back-translation.** The method can be read as back-translation that keeps refreshing its synthetic data. Re-translations of monolingual sentences improve as training proceeds, whereas a one-shot pseudo-parallel corpus stays fixed. This argues for regenerating synthetic data during training rather than freezing it.
- **Top-k instead of the full sum.** Approximating an exponential sum over output sequences with a small n-best list (k≈10) is a cheap, practical trick. It is reusable wherever a loss requires marginalizing over generated sequences.
- **Architecture-agnostic design.** Monolingual data enters only through the loss, not through changes to the network. The scheme can therefore be dropped into other end-to-end models without redesign, which is useful for low-resource pairs such as Ukrainian with limited parallel data.
- **Tracking pseudo-label quality over time.** Inspecting the best reconstructions of monolingual sentences at several training checkpoints is a simple qualitative diagnostic. It shows whether the pseudo-labels are actually getting better.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun and Yang Liu. 2016. Semi-Supervised Learning for Neural Machine Translation. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cheng2016semi,
  title = {{Semi-Supervised Learning for Neural Machine Translation}},
  author = {Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun and Yang Liu},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1185},
  url = {https://doi.org/10.18653/v1/p16-1185}
}
```

- [← previous Leveraging FrameNet to Improve Automatic Event Detection](https://md.research.elamur.ai/free/en/papers/liu-2016-leveraging-framenet-improve-automatic-event/)
- [next → Theano: A Python framework for fast computation of mathematical expressions](https://md.research.elamur.ai/free/en/papers/the-theano-development-team-2016-theano-python-framework-fast-computation/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

*Updated 24 October 2026*