---
title: "Variational Neural Machine Translation — Zhang 2016"
description: "Zhang et al. propose VNMT, which adds one continuous latent variable z per sentence. 23 references, cited by 71 papers here."
authors: ["Biao Zhang", "Deyi Xiong", "Jinsong Su", "Hong Duan", "Min Zhang"]
year: 2016
venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D16-1050/"
doi: "10.18653/v1/d16-1050"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2016-variational-neural-machine-translation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2016-variational-neural-machine-translation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Variational Neural Machine Translation

# Variational Neural Machine Translation

**Authors:** [Biao Zhang](https://md.research.elamur.ai/free/en/authors/biao-zhang/), [Deyi Xiong](https://md.research.elamur.ai/free/en/authors/deyi-xiong/), [Jinsong Su](https://md.research.elamur.ai/free/en/authors/jinsong-su/), [Hong Duan](https://md.research.elamur.ai/free/en/authors/hong-duan/) and [Min Zhang](https://md.research.elamur.ai/free/en/authors/min-zhang/) · **Year:** 2016 · **Venue:** [Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2016](https://md.research.elamur.ai/free/en/venues/emnlp/2016/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D16-1050/) · [PDF](https://aclanthology.org/D16-1050.pdf) · [DOI 10.18653/v1/d16-1050](https://doi.org/10.18653/v1/d16-1050) · [OpenAlex](https://openalex.org/W2963713328)

**Citations:** 208 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/zhang-2016-variational-neural-machine-translation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Attention-based encoder-decoder NMT learns the meaning of a source sentence only implicitly. When attention aligns words poorly, the context vector can miss parts of the input, and the translation drops or distorts content. Zhang et al. propose VNMT, which adds one continuous latent variable z per sentence. This variable represents the shared semantics of a sentence pair and guides decoding alongside attention. The design borrows from the variational autoencoder. A Gaussian prior over z is computed from the source alone, and an approximate Gaussian posterior is computed from both source and target. Neural networks produce both distributions from mean-pooled bidirectional GRU annotations. The reparameterization trick makes the variational lower bound trainable end-to-end with standard stochastic gradient methods. On Chinese-English and English-German translation, VNMT significantly outperforms vanilla attentional NMT baselines such as GroundHog, with especially clear gains on long sentences. The work is an early demonstration that latent-variable generative modelling can be added to NMT without losing large-scale training or efficient decoding.

## Key points
- Published at EMNLP 2016 (Austin, pages 521–530) by Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan and Min Zhang (Soochow University and Xiamen University).
- VNMT has three neural parts:
  - an encoder, which is the Bahdanau et al. bidirectional GRU encoder;
  - an inferer, which models the prior p(z|x) and the posterior q(z|x,y) as diagonal Gaussians;
  - a decoder, which conditions word prediction on z as well as attention.
- The training objective is a lower bound with two terms: the expected log-likelihood of the target given z and x, and a KL penalty between the posterior and the source-only prior.
- Experiments cover Chinese-English and English-German, comparing against conventional attention-based NMT (GroundHog).
- In a qualitative example, GroundHog skips the word "官员" and the entire final clause of a Chinese sentence, while VNMT conveys the full meaning. The authors credit the KL term with a beneficial regularizing effect.

## Interesting ideas
- **Asymmetric conditioning of prior and posterior.** The prior sees only the input, while the training-time posterior also sees the output. The KL term then pushes the input-only network to predict what the full pair reveals. This pattern fits any conditional generation task where only the input exists at inference time, such as summarization or data-to-text.
- **A global latent vector alongside attention.** Attention captures local alignment, while a sentence-level vector can carry meaning that faulty alignments miss. This could reduce omissions in other attention-based sequence models.
- **One encoder for both sides.** The same encoder architecture represents source and target sentences, and simple mean pooling turns the annotations into sentence summaries. This keeps the extra inference network cheap and easy to implement.
- **The KL term as a regularizer.** The divergence penalty in the variational objective does more than enable inference; the authors observe it also improves generalization. This suggests trying variational objectives as a regularization strategy even when latent semantics are not the main goal.
- **Omission-focused error analysis.** The qualitative evaluation checks which source fragments a system silently fails to translate. This is a useful complement to aggregate metrics when assessing under-translation, for example in Ukrainian MT work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Soochow University](https://md.research.elamur.ai/free/en/universities/soochow-university/) — University, Suzhou, China · 37 works · 4,291 citations
- [Xiamen University](https://md.research.elamur.ai/free/en/universities/xiamen-university/) — University, Xiamen, China · 10 works · 2,159 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan and Min Zhang. 2016. Variational Neural Machine Translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2016variational,
  title = {{Variational Neural Machine Translation}},
  author = {Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan and Min Zhang},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1050},
  url = {https://doi.org/10.18653/v1/d16-1050}
}
```

- [← previous Metaphor as a Medium for Emotion: An Empirical Study](https://md.research.elamur.ai/free/en/papers/mohammad-2016-metaphor-as-medium-emotion-empirical/)
- [next → Multilingual Language Processing From Bytes](https://md.research.elamur.ai/free/en/papers/gillick-2016-multilingual-language-processing-bytes/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Proc. of ICLR.
2. L. Bentivogli, A. Bisazza, M. Cettolo, and M. Federico. 2016. [Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://md.research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/). ArXiv e-prints, August.
3. S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. 2015. [Generating Sentences from a Continuous Space](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/). ArXiv e-prints, November.
4. Y. Cheng, S. Shen, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. 2015. Agreement-based Joint Training for Bidirectional Attention-based Neural Machine Translation. ArXiv e-prints, December. [find](https://scholar.google.com/scholar?q=Y%2E+Cheng%2C+S%2E+Shen%2C+Z%2E+He%2C+W%2E+He%2C+H%2E+Wu%2C+M%2E+Sun%2C+and+Y%2E+Liu%2E+2015%2E+Agreement%2Dbased+Joint+Training+for+Bidirectional+Attention%2Dbased+Neural+Machine+Translation%2E+ArXiv+e%2Dprints%2C+December%2E)
5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proc. of EMNLP, pages 1724– 1734, October. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Dzmitry+Bahdanau%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%E2%80%93decoder+for+statistic)
6. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Proc. of NIPS. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Kyle+Kastner%2C+Laurent+Dinh%2C+Kratarth+Goel%2C+Aaron+C%2E+Courville%2C+and+Yoshua+Bengio%2E+2015%2E+A+recurrent+latent+variable+model+for+sequential+data%2E+In+Proc%2E+of+NIPS%2E)
7. S. Feng, S. Liu, M. Li, and M. Zhou. 2016. Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model. ArXiv e-prints, January. [find](https://scholar.google.com/scholar?q=S%2E+Feng%2C+S%2E+Liu%2C+M%2E+Li%2C+and+M%2E+Zhou%2E+2016%2E+Implicit+Distortion+and+Fertility+Models+for+Attention%2Dbased+Encoder%2DDecoder+NMT+Model%2E+ArXiv+e%2Dprints%2C+January%2E)
8. Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. CoRR, abs/1502.04623. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. [On using very large target vocabulary for neural machine translation](https://md.research.elamur.ai/free/en/papers/jean-2015-very-large-target-vocabulary-neural/). In Proc. of…
9. Nal Kalchbrenner and Phil Blunsom. 2013. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In Proc. of EMNLP, pages 1700–1709, October.
10. Diederik P Kingma and Max Welling. 2014. [Auto-Encoding Variational Bayes](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/). In Proc. of ICLR.
11. Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Proc. of NIPS, pages 3581–3589. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, C… [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma%2C+Shakir+Mohamed%2C+Danilo+Jimenez+Rezende%2C+and+Max+Welling%2E+2014%2E+Semi%2Dsupervised+learning+with+deep+generative+models%2E+In+Proc%2E+of+NIPS%2C+pages+3581%E2%80%933589%2E+Philipp+Koehn%2C+Hieu+Hoang%2C+A)
12. Philipp Koehn. 2004. [Statistical significance tests for machine translation evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/). In Proc. of EMNLP.
13. Zhifei Li, Jason Eisner, and Sanjeev Khudanpur. 2009. Variational decoding for statistical machine translation. In Proc. of ACL, pages 593–601, August. [find](https://scholar.google.com/scholar?q=Zhifei+Li%2C+Jason+Eisner%2C+and+Sanjeev+Khudanpur%2E+2009%2E+Variational+decoding+for+statistical+machine+translation%2E+In+Proc%2E+of+ACL%2C+pages+593%E2%80%93601%2C+August%2E)
14. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proc. of EMNLP, pages 1412– 1421, September.
15. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. [Addressing the rare word problem in neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/). In Proc. of ACL-IJCNLP, pages 11–19, July.
16. F. Meng, Z. Lu, Z. Tu, H. Li, and Q. Liu. 2015. A Deep Memory-based Architecture for Sequence-to-Sequence Learning. ArXiv e-prints, June. [find](https://scholar.google.com/scholar?q=F%2E+Meng%2C+Z%2E+Lu%2C+Z%2E+Tu%2C+H%2E+Li%2C+and+Q%2E+Liu%2E+2015%2E+A+Deep+Memory%2Dbased+Architecture+for+Sequence%2Dto%2DSequence+Learning%2E+ArXiv+e%2Dprints%2C+June%2E)
17. Y. Miao, L. Yu, and P. Blunsom. 2015. Neural Variational Inference for Text Processing. ArXiv e-prints, November. [find](https://scholar.google.com/scholar?q=Y%2E+Miao%2C+L%2E+Yu%2C+and+P%2E+Blunsom%2E+2015%2E+Neural+Variational+Inference+for+Text+Processing%2E+ArXiv+e%2Dprints%2C+November%2E)
18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proc. of ACL, pages 311–318.
19. Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. [Stochastic backpropagation and approximate inference in deep generative models](https://md.research.elamur.ai/free/en/papers/rezende-2014-stochastic-backpropagation-approximate-inference-deep/). In Proc. of ICML, pages 1278–1286.
20. R. Sennrich, B. Haddow, and A. Birch. 2015. [Neural Machine Translation of Rare Words with Subword Units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). ArXiv e-prints, August.
21. S. Shen, Y. Cheng, Z. He, W. He, H. Wu, M. Sun, and Y. Liu. 2015. [Minimum Risk Training for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/). ArXiv e-prints, December.
22. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+CoRR%2C+abs%2F1409%2E3215%2E)
23. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. [Coverage-based neural machine translation](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/). CoRR, abs/1601.04811.

*Updated 24 October 2026*