---
title: "Convolutional Sequence to Sequence Learning — Gehring 2017"
description: "Before this paper, neural sequence-to-sequence models for tasks such as machine translation and summarization were mostly recurrent encoder–decoders…"
authors: ["Jonas Gehring", "Michael Auli", "David Grangier", "Denis Yarats", "Yann N. Dauphin"]
year: 2017
venue: "arXiv:1705.03122"
source: "https://arxiv.org/abs/1705.03122"
doi: "10.48550/arxiv.1705.03122"
arxiv: "1705.03122"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Convolutional Sequence to Sequence Learning

# Convolutional Sequence to Sequence Learning

**Authors:** [Jonas Gehring](https://md.research.elamur.ai/free/en/authors/jonas-gehring/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/), [David Grangier](https://md.research.elamur.ai/free/en/authors/david-grangier/), [Denis Yarats](https://md.research.elamur.ai/free/en/authors/denis-yarats/), [Yann N. Dauphin](https://md.research.elamur.ai/free/en/authors/yann-n-dauphin/) · **Year:** 2017 · **Venue:** arXiv:1705.03122 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1705.03122) · [PDF](https://arxiv.org/pdf/1705.03122) · [DOI 10.48550/arxiv.1705.03122](https://doi.org/10.48550/arxiv.1705.03122) · [arXiv 1705.03122](https://arxiv.org/abs/1705.03122) · [OpenAlex](https://openalex.org/W2613904329)

**Citations:** 2371 citations (openalex-snapshot · 2026-10-14) · [612](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Before this paper, neural sequence-to-sequence models for tasks such as machine translation and summarization were mostly recurrent encoder–decoders (LSTM or GRU with attention). Recurrent models process tokens one step at a time, which limits how well they can use GPUs. Researchers at Facebook AI Research propose ConvS2S, an encoder–decoder built only from stacked 1-D convolutions. Each block applies a convolution, gated linear units (GLU) and a residual connection. Learned absolute position embeddings tell the model about word order. Every decoder layer has its own dot-product attention over the encoder output ("multi-step attention"), and source embeddings are added to the values it attends to. Without recurrence, all positions are computed in parallel during training, and stacking layers widens the context each layer sees. The model was tested on WMT'14 English-German, WMT'14 English-French and WMT'16 English-Romanian, and on Gigaword and DUC-2004 summarization. It beats the deep LSTM system of Wu et al. (2016) and translates about ten times faster on both GPU and CPU. This showed that convolutional models are a practical, efficient alternative to RNNs for NMT.

## Key points
- **Translation results:** sets a new state of the art on WMT'16 English-Romanian (+1.9 BLEU over the previous best). It beats Wu et al. (2016) by 0.5 BLEU on WMT'14 English-German and by 1.6 BLEU on WMT'14 English-French (compared with the likelihood-trained version of that system).
- **Speed:** translates about an order of magnitude faster than Wu et al. on GPU and CPU. Adding attention to 4 more decoder layers slows the model by only about 4%, because attention is computed for all target words in one batch.
- **Ablations:** removing both source and target position embeddings costs only about 0.5 BLEU. Encoder accuracy improves with depth up to about 9 layers, while 2 decoder layers already perform well. Narrow kernels (width 3) with more layers beat wider kernels.
- **Summarization:** a small model (6 encoder and 6 decoder layers, hidden size 256, trained overnight on one GPU) beats the likelihood-trained RNN of Shen et al. (2016) on Gigaword and DUC-2004. It comes close to the best systems, which use task-specific training or architecture.
- **Resource:** code and trained models are released in fairseq (github.com/facebookresearch/fairseq).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

612 papers of our library cite it (of the 140673 whose bibliographies we read); 60 of them have a page here, 552 not yet.

- [FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN](https://md.research.elamur.ai/free/en/papers/agarwal-2023-findings-iwslt-2023-evaluation-campaign/)
- [Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-domain-adaptation-arabic-machine-translation/)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/artetxe-2019-margin-based-parallel-corpus-mining/)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://md.research.elamur.ai/free/en/papers/beck-2018-graph-sequence-learning-gated-graph/)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/)
- [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/)
- [What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
- [Adaptively Sparse Transformers](https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/)
- [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://md.research.elamur.ai/free/en/papers/elsherief-2021-latent-hatred-benchmark-understanding-implicit/)
- [Controllable Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/)
- [ELI5: Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/)
- [APE at Scale and Its Implications on MT Evaluation Biases](https://md.research.elamur.ai/free/en/papers/freitag-2019-ape-scale-its-implications-mt/)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://md.research.elamur.ai/free/en/papers/freitag-2022-high-quality-rather-than-high/)
- [MuST-C: a Multilingual Speech Translation Corpus](https://md.research.elamur.ai/free/en/papers/gangi-2019-must-c-multilingual-speech-translation/)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://md.research.elamur.ai/free/en/papers/gao-2019-representation-degeneration-problem-training-natural/)
- [Mask-Predict: Parallel Decoding of Conditional Masked Language Models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/)
- [Non-Autoregressive Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/)
- [Meta-Learning for Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
- [Universal Neural Machine Translation for Extremely Low Resource Languages](https://md.research.elamur.ai/free/en/papers/gu-2018-universal-neural-machine-translation-extremely/)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://md.research.elamur.ai/free/en/papers/gupta-2018-semantic-parsing-task-oriented-dialog/)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
- [CTRLsum: Towards Generic Controllable Text Summarization](https://md.research.elamur.ai/free/en/papers/he-2022-ctrlsum-generic-controllable-text-summarization/)
- [Perceiver: General Perception with Iterative Attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://md.research.elamur.ai/free/en/papers/kasai-2020-deep-encoder-shallow-decoder-reevaluating/)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/)
- [A neural interlingua for multilingual machine translation](https://md.research.elamur.ai/free/en/papers/lu-2018-neural-interlingua-multilingual-machine-translation/)
- [文档级神经机器翻译综述](https://md.research.elamur.ai/free/en/papers/lv-2024-document-level-neural/)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
- [Analyzing Uncertainty in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/)
- [Scaling Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://md.research.elamur.ai/free/en/papers/perez-2017-film-visual-reasoning-general-conditioning/)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [An Analysis of Encoder Representations in Transformer-Based Machine Translation](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
- [Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog](https://md.research.elamur.ai/free/en/papers/schuster-2019-cross-lingual-transfer-learning-multilingual/)
- [MLSUM: The Multilingual Summarization Corpus](https://md.research.elamur.ai/free/en/papers/scialom-2020-mlsum-multilingual-summarization-corpus/)
- [Self-Attention with Relative Position Representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
- [Deep Semantic Role Labeling with Self-Attention](https://md.research.elamur.ai/free/en/papers/tan-2017-deep-semantic-role-labeling-self/)
- [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/)
- [Tensor2Tensor for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/vaswani-2018-tensor2tensor-neural-machine-translation/)
- [One Model to Learn Both: Zero Pronoun Prediction and Translation](https://md.research.elamur.ai/free/en/papers/wang-2019-one-model-learn-both-zero/)
- [On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wang-2020-exposure-bias-hallucination-domain-shift/)
- [Beyond BLEU: Training Neural Machine Translation with Semantic Similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/)
- [A Study of Reinforcement Learning for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wu-2018-study-reinforcement-learning-neural-machine/)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)
- [Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction](https://md.research.elamur.ai/free/en/papers/xu-2018-double-embeddings-cnn-based-sequence/)
- [GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation](https://md.research.elamur.ai/free/en/papers/xu-2022-guofeng-benchmark-zero-pronoun-recovery/)
- [视觉语言预训练综述](https://md.research.elamur.ai/free/en/papers/yin-2022-vision-language-pre/)
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)
- [Big Bird: Transformers for Longer Sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/)
- [Improving the Transformer Translation Model with Document-Level Context](https://md.research.elamur.ai/free/en/papers/zhang-2018-improving-transformer-translation-model-document/)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-bridging-gap-between-training-inference/)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2020-knowledge-grounded-dialogue-generation-pre/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gehring2017convolutional,
  title = {{Convolutional Sequence to Sequence Learning}},
  author = {Gehring, Jonas and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N.},
  year = {2017},
  doi = {10.48550/arxiv.1705.03122},
  eprint = {1705.03122},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.03122}
}
```

- [← previous Controlling Linguistic Style Aspects in Neural Language Generation](https://md.research.elamur.ai/free/en/papers/ficler-2017-controlling-linguistic-style-aspects-neural/)
- [next → Copied Monolingual Data Improves Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/currey-2017-copied-monolingual-data-improves-low/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

49 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Ba%2C+Jimmy+Lei%2C+Kiros%2C+Jamie+Ryan%2C+and+Hinton%2C+Geoffrey+E%2E+Layer+normalization%2E+arXiv+preprint+arXiv%3A1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
2. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+Dzmitry%2C+Cho%2C+Kyunghyun%2C+and+Bengio%2C+Yoshua%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+arXiv+preprint+arXiv%3A1409%2E0473%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Bojar, Ondej, Chatterjee, Rajen, Federmann, Christian, Graham, Yvette, Haddow, Barry, Huck, Matthias, Jimeno-Yepes, Antonio, Koehn, Philipp, Logacheva, Varvara, Monz, Christof, Negri, Matteo, Névéol, Aurélie, Neves, Mariana L., Popel, Martin, Post, Matt, Rubino, Raphaël, Scarton, Carolina, Speci… [find](https://scholar.google.com/scholar?q=Bojar%2C+Ondej%2C+Chatterjee%2C+Rajen%2C+Federmann%2C+Christian%2C+Graham%2C+Yvette%2C+Haddow%2C+Barry%2C+Huck%2C+Matthias%2C+Jimeno%2DYepes%2C+Antonio%2C+Koehn%2C+Philipp%2C+Logacheva%2C+Varvara%2C+Monz%2C+Christof%2C+Negri%2C+Matteo%2C+Ne%CC%81ve%CC%81ol) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
4. Bradbury, James, Merity, Stephen, Xiong, Caiming, and Socher, Richard. Quasi-Recurrent Neural Networks. arXiv preprint arXiv:1611.01576, 2016. — [arXiv:1611.01576](https://arxiv.org/abs/1611.01576) [find](https://scholar.google.com/scholar?q=Bradbury%2C+James%2C+Merity%2C+Stephen%2C+Xiong%2C+Caiming%2C+and+Socher%2C+Richard%2E+Quasi%2DRecurrent+Neural+Networks%2E+arXiv+preprint+arXiv%3A1611%2E01576%2C+2016%2E+%E2%80%94+%5BarXiv%3A1611%2E01576%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E01576%29)
5. Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proc. of EMNLP, 2014. [find](https://scholar.google.com/scholar?q=Cho%2C+Kyunghyun%2C+Van+Merrie%CC%88nboer%2C+Bart%2C+Gulcehre%2C+Caglar%2C+Bahdanau%2C+Dzmitry%2C+Bougares%2C+Fethi%2C+Schwenk%2C+Holger%2C+and+Bengio%2C+Yoshua%2E+Learning+Phrase+Representations+using+RNN+Encoder%2DDecoder+for+Statist)
6. Chorowski, Jan K, Bahdanau, Dzmitry, Serdyuk, Dmitriy, Cho, Kyunghyun, and Bengio, Yoshua. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pp. 577–585, 2015. [find](https://scholar.google.com/scholar?q=Chorowski%2C+Jan+K%2C+Bahdanau%2C+Dzmitry%2C+Serdyuk%2C+Dmitriy%2C+Cho%2C+Kyunghyun%2C+and+Bengio%2C+Yoshua%2E+Attention%2Dbased+models+for+speech+recognition%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+577%E2%80%93) [our card](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
7. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clement. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop, 2011. URL [http://torch.ch](http://torch.ch). [↗](http://torch.ch)
8. Dauphin, Yann N., Fan, Angela, Auli, Michael, and Grangier, David. Language modeling with gated linear units. arXiv preprint arXiv:1612.08083, 2016. [find](https://scholar.google.com/scholar?q=Dauphin%2C+Yann+N%2E%2C+Fan%2C+Angela%2C+Auli%2C+Michael%2C+and+Grangier%2C+David%2E+Language+modeling+with+gated+linear+units%2E+arXiv+preprint+arXiv%3A1612%2E08083%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
9. Dyer, Chris, Chahuneau, Victor, and Smith, Noah A. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proc. of ACL, 2013. [find](https://scholar.google.com/scholar?q=Dyer%2C+Chris%2C+Chahuneau%2C+Victor%2C+and+Smith%2C+Noah+A%2E+A+Simple%2C+Fast%2C+and+Effective+Reparameterization+of+IBM+Model+2%2E+In+Proc%2E+of+ACL%2C+2013%2E)
10. Elman, Jeffrey L. Finding Structure in Time. Cognitive Science, 14:179–211, 1990. [find](https://scholar.google.com/scholar?q=Elman%2C+Jeffrey+L%2E+Finding+Structure+in+Time%2E+Cognitive+Science%2C+14%3A179%E2%80%93211%2C+1990%2E)
11. Gehring, Jonas, Auli, Michael, Grangier, David, and Dauphin, Yann N. A Convolutional Encoder Model for Neural Machine Translation. arXiv preprint arXiv:1611.02344, 2016. [find](https://scholar.google.com/scholar?q=Gehring%2C+Jonas%2C+Auli%2C+Michael%2C+Grangier%2C+David%2C+and+Dauphin%2C+Yann+N%2E+A+Convolutional+Encoder+Model+for+Neural+Machine+Translation%2E+arXiv+preprint+arXiv%3A1611%2E02344%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-encoder-model-neural-machine/)
12. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. The handbook of brain theory and neural networks, 2010. [find](https://scholar.google.com/scholar?q=Glorot%2C+Xavier+and+Bengio%2C+Yoshua%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+The+handbook+of+brain+theory+and+neural+networks%2C+2010%2E)
13. Graff, David, Kong, Junbo, Chen, Ke, and Maeda, Kazuaki. English gigaword. Linguistic Data Consortium, Philadelphia, 2003. [find](https://scholar.google.com/scholar?q=Graff%2C+David%2C+Kong%2C+Junbo%2C+Chen%2C+Ke%2C+and+Maeda%2C+Kazuaki%2E+English+gigaword%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+2003%2E)
14. Ha, David, Dai, Andrew, and Le, Quoc V. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. — [arXiv:1609.09106](https://arxiv.org/abs/1609.09106) [find](https://scholar.google.com/scholar?q=Ha%2C+David%2C+Dai%2C+Andrew%2C+and+Le%2C+Quoc+V%2E+Hypernetworks%2E+arXiv+preprint+arXiv%3A1609%2E09106%2C+2016%2E+%E2%80%94+%5BarXiv%3A1609%2E09106%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1609%2E09106%29)
15. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep Residual Learning for Image Recognition. In Proc. of CVPR, 2015a. [find](https://scholar.google.com/scholar?q=He%2C+Kaiming%2C+Zhang%2C+Xiangyu%2C+Ren%2C+Shaoqing%2C+and+Sun%2C+Jian%2E+Deep+Residual+Learning+for+Image+Recognition%2E+In+Proc%2E+of+CVPR%2C+2015a%2E)
16. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034, 2015b. [find](https://scholar.google.com/scholar?q=He%2C+Kaiming%2C+Zhang%2C+Xiangyu%2C+Ren%2C+Shaoqing%2C+and+Sun%2C+Jian%2E+Delving+deep+into+rectifiers%3A+Surpassing+human%2Dlevel+performance+on+imagenet+classification%2E+In+Proceedings+of+the+IEEE+International+Confere)
17. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+Sepp+and+Schmidhuber%2C+Ju%CC%88rgen%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2C+1997%2E)
18. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of The 32nd International Conference on Machine Learning, pp. 448–456, 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+Sergey+and+Szegedy%2C+Christian%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+Proceedings+of+The+32nd+International+Conference+on+Machine+Learn)
19. Jean, Sébastien, Firat, Orhan, Cho, Kyunghyun, Memisevic, Roland, and Bengio, Yoshua. Montreal Neural Machine Translation systems for WMT15. In Proc. of WMT, pp. 134–140, 2015. [find](https://scholar.google.com/scholar?q=Jean%2C+Se%CC%81bastien%2C+Firat%2C+Orhan%2C+Cho%2C+Kyunghyun%2C+Memisevic%2C+Roland%2C+and+Bengio%2C+Yoshua%2E+Montreal+Neural+Machine+Translation+systems+for+WMT15%2E+In+Proc%2E+of+WMT%2C+pp%2E+134%E2%80%93140%2C+2015%2E)
20. Kalchbrenner, Nal, Espeholt, Lasse, Simonyan, Karen, van den Oord, Aaron, Graves, Alex, and Kavukcuoglu, Koray. Neural Machine Translation in Linear Time. arXiv, 2016. [find](https://scholar.google.com/scholar?q=Kalchbrenner%2C+Nal%2C+Espeholt%2C+Lasse%2C+Simonyan%2C+Karen%2C+van+den+Oord%2C+Aaron%2C+Graves%2C+Alex%2C+and+Kavukcuoglu%2C+Koray%2E+Neural+Machine+Translation+in+Linear+Time%2E+arXiv%2C+2016%2E)
21. LeCun, Yann and Bengio, Yoshua. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995. [find](https://scholar.google.com/scholar?q=LeCun%2C+Yann+and+Bengio%2C+Yoshua%2E+Convolutional+networks+for+images%2C+speech%2C+and+time+series%2E+The+handbook+of+brain+theory+and+neural+networks%2C+3361%2810%29%3A1995%2C+1995%2E)
22. L’Hostis, Gurvan, Grangier, David, and Auli, Michael. Vocabulary Selection Strategies for Neural Machine Translation. arXiv preprint arXiv:1610.00072, 2016. — [arXiv:1610.00072](https://arxiv.org/abs/1610.00072) [find](https://scholar.google.com/scholar?q=L%E2%80%99Hostis%2C+Gurvan%2C+Grangier%2C+David%2C+and+Auli%2C+Michael%2E+Vocabulary+Selection+Strategies+for+Neural+Machine+Translation%2E+arXiv+preprint+arXiv%3A1610%2E00072%2C+2016%2E+%E2%80%94+%5BarXiv%3A1610%2E00072%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
23. Lin, Chin-Yew. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74–81, 2004. [find](https://scholar.google.com/scholar?q=Lin%2C+Chin%2DYew%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+Summarization+Branches+Out%3A+Proceedings+of+the+ACL%2D04+Workshop%2C+pp%2E+74%E2%80%9381%2C+2004%2E)
24. Luong, Minh-Thang, Pham, Hieu, and Manning, Christopher D. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015. [find](https://scholar.google.com/scholar?q=Luong%2C+Minh%2DThang%2C+Pham%2C+Hieu%2C+and+Manning%2C+Christopher+D%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proc%2E+of+EMNLP%2C+2015%2E)
25. Meng, Fandong, Lu, Zhengdong, Wang, Mingxuan, Li, Hang, Jiang, Wenbin, and Liu, Qun. Encoding Source Language with Convolutional Neural Network for Machine Translation. In Proc. of ACL, 2015. [find](https://scholar.google.com/scholar?q=Meng%2C+Fandong%2C+Lu%2C+Zhengdong%2C+Wang%2C+Mingxuan%2C+Li%2C+Hang%2C+Jiang%2C+Wenbin%2C+and+Liu%2C+Qun%2E+Encoding+Source+Language+with+Convolutional+Neural+Network+for+Machine+Translation%2E+In+Proc%2E+of+ACL%2C+2015%2E)
26. Mi, Haitao, Wang, Zhiguo, and Ittycheriah, Abe. Vocabulary Manipulation for Neural Machine Translation. In Proc. of ACL, 2016. [find](https://scholar.google.com/scholar?q=Mi%2C+Haitao%2C+Wang%2C+Zhiguo%2C+and+Ittycheriah%2C+Abe%2E+Vocabulary+Manipulation+for+Neural+Machine+Translation%2E+In+Proc%2E+of+ACL%2C+2016%2E)
27. Miller, Alexander H., Fisch, Adam, Dodge, Jesse, Karimi, Amir-Hossein, Bordes, Antoine, and Weston, Jason. Key-value memory networks for directly reading documents. In Proc. of EMNLP, 2016. [find](https://scholar.google.com/scholar?q=Miller%2C+Alexander+H%2E%2C+Fisch%2C+Adam%2C+Dodge%2C+Jesse%2C+Karimi%2C+Amir%2DHossein%2C+Bordes%2C+Antoine%2C+and+Weston%2C+Jason%2E+Key%2Dvalue+memory+networks+for+directly+reading+documents%2E+In+Proc%2E+of+EMNLP%2C+2016%2E)
28. Nallapati, Ramesh, Zhou, Bowen, Gulcehre, Caglar, Xiang, Bing, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proc. of EMNLP, 2016. [find](https://scholar.google.com/scholar?q=Nallapati%2C+Ramesh%2C+Zhou%2C+Bowen%2C+Gulcehre%2C+Caglar%2C+Xiang%2C+Bing%2C+et+al%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+In+Proc%2E+of+EMNLP%2C+2016%2E)
29. Oord, Aaron van den, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016a. — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) [find](https://scholar.google.com/scholar?q=Oord%2C+Aaron+van+den%2C+Kalchbrenner%2C+Nal%2C+and+Kavukcuoglu%2C+Koray%2E+Pixel+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1601%2E06759%2C+2016a%2E+%E2%80%94+%5BarXiv%3A1601%2E06759%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1601%2E06759%29)
30. Oord, Aaron van den, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016b. — [arXiv:1606.05328](https://arxiv.org/abs/1606.05328) [find](https://scholar.google.com/scholar?q=Oord%2C+Aaron+van+den%2C+Kalchbrenner%2C+Nal%2C+Vinyals%2C+Oriol%2C+Espeholt%2C+Lasse%2C+Graves%2C+Alex%2C+and+Kavukcuoglu%2C+Koray%2E+Conditional+image+generation+with+pixelcnn+decoders%2E+arXiv+preprint+arXiv%3A1606%2E05328%2C+201)
31. Over, Paul, Dang, Hoa, and Harman, Donna. Duc in context. Information Processing & Management, 43(6): 1506–1520, 2007. [find](https://scholar.google.com/scholar?q=Over%2C+Paul%2C+Dang%2C+Hoa%2C+and+Harman%2C+Donna%2E+Duc+in+context%2E+Information+Processing+%26+Management%2C+43%286%29%3A+1506%E2%80%931520%2C+2007%2E)
32. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In Proceedings of The 30th International Conference on Machine Learning, pp. 1310–1318, 2013. [find](https://scholar.google.com/scholar?q=Pascanu%2C+Razvan%2C+Mikolov%2C+Tomas%2C+and+Bengio%2C+Yoshua%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proceedings+of+The+30th+International+Conference+on+Machine+Learning%2C+pp%2E+1310%E2%80%931318%2C+20)
33. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A neural attention model for abstractive sentence summarization. In Proc. of EMNLP, 2015. [find](https://scholar.google.com/scholar?q=Rush%2C+Alexander+M%2C+Chopra%2C+Sumit%2C+and+Weston%2C+Jason%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proc%2E+of+EMNLP%2C+2015%2E)
34. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) [find](https://scholar.google.com/scholar?q=Salimans%2C+Tim+and+Kingma%2C+Diederik+P%2E+Weight+normalization%3A+A+simple+reparameterization+to+accelerate+training+of+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1602%2E07868%2C+2016%2E+%E2%80%94+%5BarXiv%3A1602%2E07868%5D%28http)
35. Schuster, Mike and Nakajima, Kaisuke. Japanese and korean voice search. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, pp. 5149–5152. [find](https://scholar.google.com/scholar?q=Schuster%2C+Mike+and+Nakajima%2C+Kaisuke%2E+Japanese+and+korean+voice+search%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2012+IEEE+International+Conference+on%2C+pp%2E+5149%E2%80%935152%2E)
36. Sennrich, Rico, Haddow, Barry, and Birch, Alexandra. Neural Machine Translation of Rare Words with Subword Units. In Proc. of ACL, 2016a. [find](https://scholar.google.com/scholar?q=Sennrich%2C+Rico%2C+Haddow%2C+Barry%2C+and+Birch%2C+Alexandra%2E+Neural+Machine+Translation+of+Rare+Words+with+Subword+Units%2E+In+Proc%2E+of+ACL%2C+2016a%2E)
37. Sennrich, Rico, Haddow, Barry, and Birch, Alexandra. Edinburgh Neural Machine Translation Systems for WMT 16. In Proc. of WMT, 2016b. [find](https://scholar.google.com/scholar?q=Sennrich%2C+Rico%2C+Haddow%2C+Barry%2C+and+Birch%2C+Alexandra%2E+Edinburgh+Neural+Machine+Translation+Systems+for+WMT+16%2E+In+Proc%2E+of+WMT%2C+2016b%2E) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
38. Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc, Hinton, Geoffrey, and Dean, Jeff. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv e-prints, January 2016. [find](https://scholar.google.com/scholar?q=Shazeer%2C+Noam%2C+Mirhoseini%2C+Azalia%2C+Maziarz%2C+Krzysztof%2C+Davis%2C+Andy%2C+Le%2C+Quoc%2C+Hinton%2C+Geoffrey%2C+and+Dean%2C+Jeff%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+ArXiv+e)
39. Shen, Shiqi, Zhao, Yu, Liu, Zhiyuan, Sun, Maosong, et al. Neural headline generation with sentence-wise optimization. arXiv preprint arXiv:1604.01904, 2016. — [arXiv:1604.01904](https://arxiv.org/abs/1604.01904) [find](https://scholar.google.com/scholar?q=Shen%2C+Shiqi%2C+Zhao%2C+Yu%2C+Liu%2C+Zhiyuan%2C+Sun%2C+Maosong%2C+et+al%2E+Neural+headline+generation+with+sentence%2Dwise+optimization%2E+arXiv+preprint+arXiv%3A1604%2E01904%2C+2016%2E+%E2%80%94+%5BarXiv%3A1604%2E01904%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
40. Srivastava, Nitish, Hinton, Geoffrey E., Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent Neural Networks from overfitting. JMLR, 15:1929–1958, 2014. [find](https://scholar.google.com/scholar?q=Srivastava%2C+Nitish%2C+Hinton%2C+Geoffrey+E%2E%2C+Krizhevsky%2C+Alex%2C+Sutskever%2C+Ilya%2C+and+Salakhutdinov%2C+Ruslan%2E+Dropout%3A+a+simple+way+to+prevent+Neural+Networks+from+overfitting%2E+JMLR%2C+15%3A1929%E2%80%931958%2C+2014%2E)
41. Sukhbaatar, Sainbayar, Weston, Jason, Fergus, Rob, and Szlam, Arthur. End-to-end Memory Networks. In Proc. of NIPS, pp. 2440–2448, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+Sainbayar%2C+Weston%2C+Jason%2C+Fergus%2C+Rob%2C+and+Szlam%2C+Arthur%2E+End%2Dto%2Dend+Memory+Networks%2E+In+Proc%2E+of+NIPS%2C+pp%2E+2440%E2%80%932448%2C+2015%2E)
42. Sutskever, Ilya, Martens, James, Dahl, George E., and Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. In ICML, 2013. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Martens%2C+James%2C+Dahl%2C+George+E%2E%2C+and+Hinton%2C+Geoffrey+E%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+ICML%2C+2013%2E)
43. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS, pp. 3104–3112, 2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+Ilya%2C+Vinyals%2C+Oriol%2C+and+Le%2C+Quoc+V%2E+Sequence+to+Sequence+Learning+with+Neural+Networks%2E+In+Proc%2E+of+NIPS%2C+pp%2E+3104%E2%80%933112%2C+2014%2E)
44. Suzuki, Jun and Nagata, Masaaki. Cutting-off redundant repeating generations for neural abstractive summarization. arXiv preprint arXiv:1701.00138, 2017. [find](https://scholar.google.com/scholar?q=Suzuki%2C+Jun+and+Nagata%2C+Masaaki%2E+Cutting%2Doff+redundant+repeating+generations+for+neural+abstractive+summarization%2E+arXiv+preprint+arXiv%3A1701%2E00138%2C+2017%2E)
45. Waibel, Alex, Hanazawa, Toshiyuki, Hinton, Geoffrey, Shikano, Kiyohiro, and Lang, Kevin J. Phoneme Recognition using Time-delay Neural Networks. IEEE transactions on acoustics, speech, and signal processing, 37 (3):328–339, 1989. [find](https://scholar.google.com/scholar?q=Waibel%2C+Alex%2C+Hanazawa%2C+Toshiyuki%2C+Hinton%2C+Geoffrey%2C+Shikano%2C+Kiyohiro%2C+and+Lang%2C+Kevin+J%2E+Phoneme+Recognition+using+Time%2Ddelay+Neural+Networks%2E+IEEE+transactions+on+acoustics%2C+speech%2C+and+signal+proc)
46. Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V, Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Wu%2C+Yonghui%2C+Schuster%2C+Mike%2C+Chen%2C+Zhifeng%2C+Le%2C+Quoc+V%2C+Norouzi%2C+Mohammad%2C+Macherey%2C+Wolfgang%2C+Krikun%2C+Maxim%2C+Cao%2C+Yuan%2C+Gao%2C+Qin%2C+Macherey%2C+Klaus%2C+et+al%2E+Google%E2%80%99s+Neural+Machine+Translation+System%3A+B)
47. Yang, Zichao, Hu, Zhiting, Deng, Yuntian, Dyer, Chris, and Smola, Alex. Neural Machine Translation with Recurrent Attention Modeling. arXiv preprint arXiv:1607.05108, 2016. [find](https://scholar.google.com/scholar?q=Yang%2C+Zichao%2C+Hu%2C+Zhiting%2C+Deng%2C+Yuntian%2C+Dyer%2C+Chris%2C+and+Smola%2C+Alex%2E+Neural+Machine+Translation+with+Recurrent+Attention+Modeling%2E+arXiv+preprint+arXiv%3A1607%2E05108%2C+2016%2E)
48. Zhou, Jie, Cao, Ying, Wang, Xuguang, Li, Peng, and Xu, Wei. Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation. arXiv preprint arXiv:1606.04199, 2016. [find](https://scholar.google.com/scholar?q=Zhou%2C+Jie%2C+Cao%2C+Ying%2C+Wang%2C+Xuguang%2C+Li%2C+Peng%2C+and+Xu%2C+Wei%2E+Deep+Recurrent+Models+with+Fast%2DForward+Connections+for+Neural+Machine+Translation%2E+arXiv+preprint+arXiv%3A1606%2E04199%2C+2016%2E)
49. A. Weight Initialization We derive a weight initialization scheme tailored to the GLU activation function similar to Glorot & Bengio (2010); He et al. (2015b) by focusing on the variance of activations within the network for both forward and backward passes. We also detail how we modify the weight i… [find](https://scholar.google.com/scholar?q=A%2E+Weight+Initialization+We+derive+a+weight+initialization+scheme+tailored+to+the+GLU+activation+function+similar+to+Glorot+%26+Bengio+%282010%29%3B+He+et+al%2E+%282015b%29+by+focusing+on+the+variance+of+activation)