---
title: "Deep Recurrent Models with Fast-Forward Connections for"
description: "The design is applied to two models: an encoder-decoder variant (Deep-ED) and an… 6.2 points. 24 references, cited by 48 papers here."
authors: ["Jie Zhou", "Ying Cao", "Xuguang Wang", "Peng Li", "Wei Xu"]
year: 2016
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q16-1027/"
doi: "10.1162/tacl_a_00105"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2016-deep-recurrent-models-fast-forward/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2016-deep-recurrent-models-fast-forward/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

# Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

**Authors:** [Jie Zhou](https://md.research.elamur.ai/free/en/authors/jie-zhou/), Ying Cao, Xuguang Wang, [Peng Li](https://md.research.elamur.ai/free/en/authors/peng-li/) and [Wei Xu](https://md.research.elamur.ai/free/en/authors/wei-xu/) · **Year:** 2016 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2016](https://md.research.elamur.ai/free/en/venues/tacl/2016/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q16-1027/) · [PDF](https://aclanthology.org/Q16-1027.pdf) · [DOI 10.1162/tacl\_a\_00105](https://doi.org/10.1162/tacl_a_00105) · [OpenAlex](https://openalex.org/W2963991316)

**Citations:** 216 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/zhou-2016-deep-recurrent-models-fast-forward/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Before this work, single neural machine translation models trailed the strongest phrase-based systems on WMT'14 English-to-French. Recurrent NMT networks were also kept shallow, at most six LSTM layers, because gradients fade as they pass through many nonlinear and recurrent steps. Zhou et al. (Baidu Research) address this with fast-forward connections. These are linear links between the feed-forward computation blocks of neighbouring LSTM layers, giving gradients a route that skips both activations and recurrence. They combine this with an interleaved bi-directional encoder, in which adjacent layers read the sequence in alternating directions. The design is applied to two models: an encoder-decoder variant (Deep-ED) and an attention variant (Deep-Att), reaching depth 16. On English-to-French, a single Deep-Att model beats the best conventional SMT system, the first time a lone NMT model has done so on this task. Adding unknown-word handling and ensembling sets a new best score. The results suggest that depth, enabled by better gradient flow, helps recurrent translation models as much as it helps vision networks.

## Key points
- **Single-model results:**
  - Deep-Att reaches BLEU 37.7. This is 6.2 points above the previous best single model (31.5, Luong et al.) and 0.7 above the SMT system of Durrani et al. (37.0).
  - Deep-ED, which has no attention, still scores 36.3.
- **Unknown words and ensembling:** Unknown-word processing adds about 1.5 BLEU, bringing a single model to 39.2. An ensemble of three Deep-Att models reaches 40.4, roughly 2.9 above the prior best result.
- **Test subset without unknown words:** On the 1705 sentences (56.8%) whose targets contain no unknown words, Deep-Att scores 40.3 and the ensemble 41.4, versus 37.5 for SMT.
- **Model size and decoding:** The deepest models have depth 16 with 25 LSTM layers. The best outputs use a beam size of only 3, compared with about 12 in earlier NMT systems.
- **Analysis:**
  - The deep models overfit less than shallow ones.
  - Deep-Att does not degrade on sentences longer than 70 words.
  - The models were also tested on the harder WMT'14 English-to-German task.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Baidu](https://md.research.elamur.ai/free/en/companies/baidu/) — Company, Beijing, China · 40 works · 14,627 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Jie Zhou, Ying Cao, Xuguang Wang, Peng Li and Wei Xu. 2016. Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{zhou2016deep,
  title = {{Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation}},
  author = {Jie Zhou, Ying Cao, Xuguang Wang, Peng Li and Wei Xu},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00105},
  url = {https://doi.org/10.1162/tacl_a_00105}
}
```

- [← previous Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://md.research.elamur.ai/free/en/papers/vijayakumar-2016-diverse-beam-search-decoding-diverse/)
- [next → Learning Global Features for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/wiseman-2016-learning-global-features-coreference-resolution/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/)

## References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Proceedings of International Conference on Learning Representations.
2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Patrice+Simard%2C+and+Paolo+Frasconi%2E+1994%2E+Learning+long%2Dterm+dependencies+with+gradient+descent+is+difficult%2E+IEEE+Transactions+on+Neural+Networks%2C+5%282%29%3A157%E2%80%93166%2E)
3. Yoshua Bengio, 2012. Practical Recommendations for Gradient-Based Training of Deep Architectures, pages 437–478. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+2012%2E+Practical+Recommendations+for+Gradient%2DBased+Training+of+Deep+Architectures%2C+pages+437%E2%80%93478%2E)
4. Springer Berlin Heidelberg, Berlin, Heidelberg. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. [N-gram counts and language models from the common crawl](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/). In Proceedings of the Language Resources and Evaluation Conference.
5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2C+Bart+van+Merrienboer%2C+Caglar+Gulcehre%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2014%2E+Learning+phrase+representations+using+rnn+encoder%2Ddecoder+for+statistical+machine+transla)
6. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. 2014. Edinburgh’s phrase-based machine translation systems for WMT-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Mikel L. Forcada and Ramón P. Ñeco. 1997. Recursive hetero-associative memories for tran… [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Kenneth+Heafield%2E+2014%2E+Edinburgh%E2%80%99s+phrase%2Dbased+machine+translation+systems+for+WMT%2D14%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Tra)
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. T… [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2E+Karl+Moritz+Hermann%2C+Toma%CC%81s%CC%8C+K)
8. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1… [find](https://scholar.google.com/scholar?q=Geoffrey+E%2E+Hinton%2C+Nitish+Srivastava%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2012%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv%3A1207%2E0580%2E+Sep)
9. Nal Kalchbrenner and Phil Blunsom. 2013. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In Proceedings of the Empirical Methods in Natural Language Processing.
10. Nal Kalchbrenner, Ivo Danihelka, and Alex Graves. 2016. Grid long short-term memory. In Proceedings of International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner%2C+Ivo+Danihelka%2C+and+Alex+Graves%2E+2016%2E+Grid+long+short%2Dterm+memory%2E+In+Proceedings+of+International+Conference+on+Learning+Representations%2E)
11. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Lei+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+Proceedings+of+International+Conference+on+Learning+Representations%2E)
12. P. Koehn, F. J. Och, and D. Marcu. 2003. Statistical phrase-based translation. In Proceedings of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning appli… [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+F%2E+J%2E+Och%2C+and+D%2E+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+on+Human+Language+Technolog)
13. Percy Liang, Ben Taskar, and Dan Klein. 2006. [Alignment by agreement](https://md.research.elamur.ai/free/en/papers/liang-2006-alignment-agreement/). In Proceedings of the North American Chapter of the Association of Computational Linguistics on Human Language Technology.
14. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. [Addressing the rare word problem in neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P…
15. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L. Yuille. 2015. Deep captioning with multimodal recurrent neural networks (m-RNN). In Proceedings of International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Junhua+Mao%2C+Wei+Xu%2C+Yi+Yang%2C+Jiang+Wang%2C+Zhiheng+Huang%2C+and+Alan+L%2E+Yuille%2E+2015%2E+Deep+captioning+with+multimodal+recurrent+neural+networks+%28m%2DRNN%29%2E+In+Proceedings+of+International+Conference+on+Learn)
16. Holger Schwenk. 2014. — [Highway Networks](https://md.research.elamur.ai/free/en/papers/srivastava-2015-highway-networks/) [http://www-lium.univ-lemans.fr/∼schwenk/cslm](http://www-lium.univ-lemans.fr/∼schwenk/cslm) joint paper [online; accessed 03-september-2014]. University Le Mans. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. In Proceedings of the 32nd International Conference on Machine Learning, Dee… [↗](http://www-lium.univ-lemans.fr/∼schwenk/cslm)
17. Ilya Sutskever, Oriol Vinyals, and Quoc Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
18. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Wei+Liu%2C+Yangqing+Jia%2C+Pierre+Sermanet%2C+Scott+Reed%2C+Dragomir+Anguelov%2C+Dumitru+Erhan%2C+Vincent+Vanhoucke%2C+and+Andrew+Rabinovich%2E+2015%2E+Going+deeper+with+convolutions%2E+In+IEEE+Confere)
19. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4. [find](https://scholar.google.com/scholar?q=Tijmen+Tieleman+and+Geoffrey+Hinton%2E+2012%2E+Lecture+6%2E5%2Drmsprop%3A+Divide+the+gradient+by+a+running+average+of+its+recent+magnitude%2E+COURSERA%3A+Neural+Networks+for+Machine+Learning%2C+4%2E)
20. Oriol Vinyals and Quoc Le. 2015. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). In Proceedings of the 32nd International Conference on Machine Learning, Deep Learning Workshop.
21. Kaisheng Yao, Trevor Cohn, Katerina Vylomova, Kevin Duh, and Chris Dyer. 2015. Depth-gated LSTM. arXiv:1508.03790. — [arXiv:1508.03790](https://arxiv.org/abs/1508.03790) [find](https://scholar.google.com/scholar?q=Kaisheng+Yao%2C+Trevor+Cohn%2C+Katerina+Vylomova%2C+Kevin+Duh%2C+and+Chris+Dyer%2E+2015%2E+Depth%2Dgated+LSTM%2E+arXiv%3A1508%2E03790%2E+%E2%80%94+%5BarXiv%3A1508%2E03790%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1508%2E03790%29)
22. Yang Yu, Wei Zhang, Chung-Wei Hang, Bing Xiang, and Bowen Zhou. 2015. Empirical study on deep learning models for QA. arXiv:1510.07526. — [arXiv:1510.07526](https://arxiv.org/abs/1510.07526) [find](https://scholar.google.com/scholar?q=Yang+Yu%2C+Wei+Zhang%2C+Chung%2DWei+Hang%2C+Bing+Xiang%2C+and+Bowen+Zhou%2E+2015%2E+Empirical+study+on+deep+learning+models+for+QA%2E+arXiv%3A1510%2E07526%2E+%E2%80%94+%5BarXiv%3A1510%2E07526%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1510%2E07526%29)
23. Matthew D. Zeiler. 2012. [ADADELTA: An adaptive learning rate method](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/). arXiv:1212.5701.
24. Jie Zhou and Wei Xu. 2015. [End-to-end learning of semantic role labeling using recurrent neural networks](https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.

*Updated 24 October 2026*