---
title: "Pay Less Attention with Lightweight and Dynamic — Wu 2019"
description: "Self-attention compares every position with every other position."
authors: ["Felix Wu", "Angela Fan", "Alexei Baevski", "Yann N. Dauphin", "Michael Auli"]
year: 2019
venue: "arXiv:1901.10430"
source: "https://arxiv.org/abs/1901.10430"
doi: "10.48550/arxiv.1901.10430"
arxiv: "1901.10430"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Pay Less Attention with Lightweight and Dynamic Convolutions

# Pay Less Attention with Lightweight and Dynamic Convolutions

**Authors:** [Felix Wu](https://md.research.elamur.ai/free/en/authors/felix-wu/), [Angela Fan](https://md.research.elamur.ai/free/en/authors/angela-fan/), [Alexei Baevski](https://md.research.elamur.ai/free/en/authors/alexei-baevski/), [Yann N. Dauphin](https://md.research.elamur.ai/free/en/authors/yann-n-dauphin/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/) · **Year:** 2019 · **Venue:** arXiv:1901.10430 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1901.10430) · [PDF](https://arxiv.org/pdf/1901.10430) · [DOI 10.48550/arxiv.1901.10430](https://doi.org/10.48550/arxiv.1901.10430) · [arXiv 1901.10430](https://arxiv.org/abs/1901.10430) · [OpenAlex](https://openalex.org/W2908336025)

**Citations:** 270 citations (openalex-snapshot · 2026-10-14) · [146](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Self-attention compares every position with every other position. Its cost therefore grows quadratically with sequence length, and it is not clear that such unrestricted context is needed. Wu et al. test whether much simpler convolutional operators can match it on sequence modeling. They propose two operators. LightConv is a depthwise convolution whose kernel weights are softmax-normalized over the window and shared across groups of channels. DynamicConv builds on LightConv but predicts a fresh kernel at each position, computed only from that position's own representation. This makes its cost linear in sequence length. Both operators replace self-attention inside a Transformer Big encoder-decoder implemented in fairseq. They are evaluated on machine translation, language modeling and abstractive summarization. LightConv already performs close to strong self-attention models, and DynamicConv matches or beats them. On WMT'14 English-German it reaches a new state of the art. The work matters because it shows that content-based attention over the whole sequence is not required for top results, and that a cheaper, faster alternative exists.

## Key points
- DynamicConv scores 29.7 BLEU on WMT'14 English-German, a new state of the art. It ties the best published result on WMT English-French and beats self-attention by 0.8 BLEU on IWSLT German-English.
- It runs about 20% faster than a highly optimized self-attention baseline.
- Weight sharing makes the kernels tiny. With d = 1024, k = 7 and H = 16:
  - a standard convolution needs 7,340,032 weights,
  - a depthwise convolution needs 7,168,
  - LightConv needs only 112.
- On the Billion word benchmark, DynamicConv equals or exceeds self-attention. On CNN-DailyMail summarization it outperforms a strong self-attention model.
- Kernel normalization is essential for stable training. Without it, training diverges, while softmax gives the best result, 26.9 BLEU on newstest2013. Code and pretrained models are released in fairseq.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 136 not yet.

- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/)
- [The BEA-2019 Shared Task on Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/bryant-2019-bea-2019-shared-task-grammatical/)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/)
- [Mask-Predict: Parallel Decoding of Conditional Masked Language Models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://md.research.elamur.ai/free/en/papers/gulati-2020-conformer-convolution-augmented-transformer-speech/)
- [Revealing the Dark Secrets of BERT](https://md.research.elamur.ai/free/en/papers/kovaleva-2019-revealing-dark-secrets-bert/)
- [Modeling Local Contexts for Joint Dialogue Act Recognition and Sentiment Classification with Bi-channel Dynamic Convolutions](https://md.research.elamur.ai/free/en/papers/li-2020-modeling-local-contexts-joint-dialogue/)
- [Coreference-Aware Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-coreference-aware-dialogue-summarization/)
- [Compressive Transformers for Long-Range Sequence Modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wu2019pay,
  title = {{Pay Less Attention with Lightweight and Dynamic Convolutions}},
  author = {Wu, Felix and Fan, Angela and Baevski, Alexei and Dauphin, Yann N. and Auli, Michael},
  year = {2019},
  doi = {10.48550/arxiv.1901.10430},
  eprint = {1901.10430},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.10430}
}
```

- [← previous PAWS: Paraphrase Adversaries from Word Scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/)
- [next → PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arxiv, abs/1711.02132, 2017. — [arXiv:1711.02132](https://arxiv.org/abs/1711.02132) [find](https://scholar.google.com/scholar?q=Karim+Ahmed%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+Weighted+transformer+network+for+machine+translation%2E+arxiv%2C+abs%2F1711%2E02132%2C+2017%2E+%E2%80%94+%5BarXiv%3A1711%2E02132%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1711%2E02132%29)
2. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. abs/1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Jimmy+Lei+Ba%2C+Jamie+Ryan+Kiros%2C+and+Geoffrey+E+Hinton%2E+abs%2F1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+of+ICLR%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
4. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. In Proc. of NAACL, 2018. [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Antoine+Bosselut%2C+Xiaodong+He%2C+and+Yejin+Choi%2E+Deep+communicating+agents+for+abstractive+summarization%2E+In+Proc%2E+of+NAACL%2C+2018%2E)
5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv, 2013. [find](https://scholar.google.com/scholar?q=Ciprian+Chelba%2C+Tomas+Mikolov%2C+Mike+Schuster%2C+Qi+Ge%2C+Thorsten+Brants%2C+Phillipp+Koehn%2C+and+Tony+Robinson%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+arXiv%2C+2013) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
6. Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. arxiv, abs/1804.09849, 2018. [find](https://scholar.google.com/scholar?q=Mia+Xu+Chen%2C+Orhan+Firat%2C+Ankur+Bapna%2C+Melvin+Johnson%2C+Wolfgang+Macherey%2C+George+Foster%2C+Llion+Jones%2C+Niki+Parmar%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Yonghui+Wu%2C+and+Macduff+Hughes%2E+The+best+of+both+worlds%3A) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/)
7. Yu-hsin Chen, Ignacio Lopez-Moreno, Tara N Sainath, Mirkó Visontai, Raziel Alvarez, and Carolina Parada. Locally-connected and convolutional neural networks for small footprint speaker recognition. In Proc. of Interspeech, 2015. [find](https://scholar.google.com/scholar?q=Yu%2Dhsin+Chen%2C+Ignacio+Lopez%2DMoreno%2C+Tara+N+Sainath%2C+Mirko%CC%81+Visontai%2C+Raziel+Alvarez%2C+and+Carolina+Parada%2E+Locally%2Dconnected+and+convolutional+neural+networks+for+small+footprint+speaker+recognition%2E+I)
8. Franois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proc. of CVPR, pp. 1800–1807, 2017. [find](https://scholar.google.com/scholar?q=Franois+Chollet%2E+Xception%3A+Deep+learning+with+depthwise+separable+convolutions%2E+In+Proc%2E+of+CVPR%2C+pp%2E+1800%E2%80%931807%2C+2017%2E)
9. Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, KyungHyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. arXiv, abs/1506.07503, 2015. [find](https://scholar.google.com/scholar?q=Jan+Chorowski%2C+Dzmitry+Bahdanau%2C+Dmitriy+Serdyuk%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+Attention%2Dbased+models+for+speech+recognition%2E+arXiv%2C+abs%2F1506%2E07503%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/)
10. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. of ICML, 2017. [find](https://scholar.google.com/scholar?q=Yann+N%2E+Dauphin%2C+Angela+Fan%2C+Michael+Auli%2C+and+David+Grangier%2E+Language+modeling+with+gated+convolutional+networks%2E+In+Proc%2E+of+ICML%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
11. Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M Rush. Latent alignment and variational attention. arXiv, abs/1807.03756, 2018. — [arXiv:1807.03756](https://arxiv.org/abs/1807.03756) [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yoon+Kim%2C+Justin+Chiu%2C+Demi+Guo%2C+and+Alexander+M+Rush%2E+Latent+alignment+and+variational+attention%2E+arXiv%2C+abs%2F1807%2E03756%2C+2018%2E+%E2%80%94+%5BarXiv%3A1807%2E03756%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1807%2E03756%29)
12. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proc. of NAACL, 2018. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+David+Grangier%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+Classical+structured+prediction+losses+for+sequence+to+sequence+learning%2E+In+Proc%2E+of+NAACL%2C+2018%2E)
13. Maha Elbayad, Laurent Besacier, and Jakob Verbeek. Pervasive attention: 2d convolutional neural networks for sequence-to-sequence prediction. arXiv, abs/1808.03867, 2018. [find](https://scholar.google.com/scholar?q=Maha+Elbayad%2C+Laurent+Besacier%2C+and+Jakob+Verbeek%2E+Pervasive+attention%3A+2d+convolutional+neural+networks+for+sequence%2Dto%2Dsequence+prediction%2E+arXiv%2C+abs%2F1808%2E03867%2C+2018%2E)
14. Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv, abs/1711.05217, 2017. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+David+Grangier%2C+and+Michael+Auli%2E+Controllable+abstractive+summarization%2E+arXiv%2C+abs%2F1711%2E05217%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/fan-2018-controllable-abstractive-summarization/)
15. Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. A Convolutional Encoder Model for Neural Machine Translation. arXiv, abs/1611.02344, 2016. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+and+Yann+N+Dauphin%2E+A+Convolutional+Encoder+Model+for+Neural+Machine+Translation%2E+arXiv%2C+abs%2F1611%2E02344%2C+2016%2E)
16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional Sequence to Sequence Learning. In Proc. of ICML, 2017. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+Convolutional+Sequence+to+Sequence+Learning%2E+In+Proc%2E+of+ICML%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
17. Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summarization. arXiv, abs/1808.10792, 2018. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+M+Rush%2E+Bottom%2Dup+abstractive+summarization%2E+arXiv%2C+abs%2F1808%2E10792%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
18. Jingjing Gong, Xipeng Qiu, Xinchi Chen, Dong Liang, and Xuanjing Huang. Convolutional interaction network for natural language inference. In Proc. of EMNLP, 2018. [find](https://scholar.google.com/scholar?q=Jingjing+Gong%2C+Xipeng+Qiu%2C+Xinchi+Chen%2C+Dong+Liang%2C+and+Xuanjing+Huang%2E+Convolutional+interaction+network+for+natural+language+inference%2E+In+Proc%2E+of+EMNLP%2C+2018%2E)
19. Edouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Herve Jegou. Efficient softmax approximation for gpus. arXiv, abs/1609.04309, 2016. [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Armand+Joulin%2C+Moustapha+Cisse%2C+David+Grangier%2C+and+Herve+Jegou%2E+Efficient+softmax+approximation+for+gpus%2E+arXiv%2C+abs%2F1609%2E04309%2C+2016%2E)
20. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference on Learning Representations, 2018. URL [https://openreview.net/forum?id=B1l8BtlCb](https://openreview.net/forum?id=B1l8BtlCb). [↗](https://openreview.net/forum?id=B1l8BtlCb) [our card](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/)
21. Layer normalization. arXiv, Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural machine translation with enhanced decoder input. AAAI, 2019. [find](https://scholar.google.com/scholar?q=Layer+normalization%2E+arXiv%2C+Junliang+Guo%2C+Xu+Tan%2C+Di+He%2C+Tao+Qin%2C+Linli+Xu%2C+and+Tie%2DYan+Liu%2E+Non%2Dautoregressive+neural+machine+translation+with+enhanced+decoder+input%2E+AAAI%2C+2019%2E)
22. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, Will Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhiru… [find](https://scholar.google.com/scholar?q=Hany+Hassan%2C+Anthony+Aue%2C+Chang+Chen%2C+Vishal+Chowdhary%2C+Jonathan+Clark%2C+Christian+Federmann%2C+Xuedong+Huang%2C+Marcin+Junczys%2DDowmunt%2C+Will+Lewis%2C+Mu+Li%2C+Shujie+Liu%2C+Tie%2DYan+Liu%2C+Renqian+Luo%2C+Arul+Meneze) [our card](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
23. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proc. of CVPR, 2015. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Deep+Residual+Learning+for+Image+Recognition%2E+In+Proc%2E+of+CVPR%2C+2015%2E)
24. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NIPS, 2015. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+In+Proc%2E+of+NIPS%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
25. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
26. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv, abs/1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jo%CC%81zefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv%2C+abs%2F1602%2E02410%2C+2016%2E)
27. Lukasz Kaiser, Aidan N. Gomez, and François Chollet. Depthwise separable convolutions for neural machine translation. arXiv, abs/1706.03059, 2017. — [arXiv:1706.03059](https://arxiv.org/abs/1706.03059) [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser%2C+Aidan+N%2E+Gomez%2C+and+Franc%CC%A7ois+Chollet%2E+Depthwise+separable+convolutions+for+neural+machine+translation%2E+arXiv%2C+abs%2F1706%2E03059%2C+2017%2E+%E2%80%94+%5BarXiv%3A1706%2E03059%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1706%2E0305)
28. Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 2390–2399, 2018. [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser%2C+Samy+Bengio%2C+Aurko+Roy%2C+Ashish+Vaswani%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+and+Noam+Shazeer%2E+Fast+decoding+in+sequence+models+using+discrete+latent+variables%2E+In+Proceedings+of+the+35th+Inte)
29. Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu. Neural Machine Translation in Linear Time. arXiv, 2016. [find](https://scholar.google.com/scholar?q=Nal+Kalchbrenner%2C+Lasse+Espeholt%2C+Karen+Simonyan%2C+Aaron+van+den+Oord%2C+Alex+Graves%2C+and+Koray+Kavukcuoglu%2E+Neural+Machine+Translation+in+Linear+Time%2E+arXiv%2C+2016%2E)
30. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR, 2015. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+Method+for+Stochastic+Optimization%2E+In+Proc%2E+of+ICLR%2C+2015%2E)
31. Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):2278–2324, 1998. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Leon+Bottou%2C+Yoshua+Bengio%2C+and+Patrick+Haffner%2E+Gradient%2Dbased+learning+applied+to+document+recognition%2E+Proc%2E+of+the+IEEE%2C+86%2811%29%3A2278%E2%80%932324%2C+1998%2E)
32. Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. arXiv, abs/1802.06901, 2018. [find](https://scholar.google.com/scholar?q=Jason+Lee%2C+Elman+Mansimov%2C+and+Kyunghyun+Cho%2E+Deterministic+non%2Dautoregressive+neural+sequence+modeling+by+iterative+refinement%2E+arXiv%2C+abs%2F1802%2E06901%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/)
33. Zhuohan Li, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. Hint-based training for non-autoregressive translation, 2019. URL [https://openreview.net/forum?id=r1gGpjActQ](https://openreview.net/forum?id=r1gGpjActQ). [↗](https://openreview.net/forum?id=r1gGpjActQ)
34. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out, 2004. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Workshop+on+Text+Summarization+Branches+Out%2C+2004%2E)
35. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv, abs/1801.10198, 2018. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+Generating+wikipedia+by+summarizing+long+sequences%2E+arXiv%2C+abs%2F1801%2E10198%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
36. Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv, abs/1608.03983, 2016. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts%2E+arXiv%2C+abs%2F1608%2E03983%2C+2016%2E)
37. Minh-Thang Luong, Hieu Pham, and Christopher Manning. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP, 2015. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+Manning%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proc%2E+of+EMNLP%2C+2015%2E)
38. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proc. of CONLL, 2016. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+In+Proc%2E+of+CONLL%2C+2016%2E)
39. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proc. of WMT, 2018. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+David+Grangier%2C+and+Michael+Auli%2E+Scaling+neural+machine+translation%2E+In+Proc%2E+of+WMT%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/)
40. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proc. of ICML, 2013. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+On+the+difficulty+of+training+recurrent+neural+networks%2E+In+Proc%2E+of+ICML%2C+2013%2E)
41. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv, abs/1705.04304, 2017. [find](https://scholar.google.com/scholar?q=Romain+Paulus%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+A+deep+reinforced+model+for+abstractive+summarization%2E+arXiv%2C+abs%2F1705%2E04304%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
42. Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. In Proc. of ICLR Workshop, 2017. [find](https://scholar.google.com/scholar?q=Gabriel+Pereyra%2C+George+Tucker%2C+Jan+Chorowski%2C+Lukasz+Kaiser%2C+and+Geoffrey+E%2E+Hinton%2E+Regularizing+neural+networks+by+penalizing+confident+output+distributions%2E+In+Proc%2E+of+ICLR+Workshop%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/)
43. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proc. of EACL 2017, 2017. [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+Using+the+output+embedding+to+improve+language+models%2E+In+Proc%2E+of+EACL+2017%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
44. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proc. of ACL, 2017. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proc%2E+of+ACL%2C+2017%2E)
45. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proc%2E+of+ACL%2C+2016%2E)
46. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proc. of NAACL, 2018. [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Jakob+Uszkoreit%2C+and+Ashish+Vaswani%2E+Self%2Dattention+with+relative+position+representations%2E+In+Proc%2E+of+NAACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/)
47. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Proc. of ICLR, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+In+Proc%2E+of+IC)
48. Dinghan Shen, Martin Renqiang Min, Yitong Li, and Lawrence Carin. Learning context-sensitive convolutional filters for text processing. In Proc. of EMNLP, 2018a. [find](https://scholar.google.com/scholar?q=Dinghan+Shen%2C+Martin+Renqiang+Min%2C+Yitong+Li%2C+and+Lawrence+Carin%2E+Learning+context%2Dsensitive+convolutional+filters+for+text+processing%2E+In+Proc%2E+of+EMNLP%2C+2018a%2E)
49. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Directional self-attention network for rnn/cnn-free language understanding. arXiv, abs/1709.04696, 2017. [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+Shirui+Pan%2C+and+Chengqi+Zhang%2E+Disan%3A+Directional+self%2Dattention+network+for+rnn%2Fcnn%2Dfree+language+understanding%2E+arXiv%2C+abs%2F1709%2E04696%2C+2017%2E)
50. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv, abs/1804.00857, 2018b. — [arXiv:1804.00857](https://arxiv.org/abs/1804.00857) [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+and+Chengqi+Zhang%2E+Bi%2Ddirectional+block+self%2Dattention+for+fast+and+memory%2Defficient+sequence+modeling%2E+arXiv%2C+abs%2F1804%2E00857%2C+2018b%2E+%E2%80%94+%5BarXiv%3A1804%2E008)
51. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Fast directional self-attention mechanism. arXiv, abs/1805.00912, 2018c. — [arXiv:1805.00912](https://arxiv.org/abs/1805.00912) [find](https://scholar.google.com/scholar?q=Tao+Shen%2C+Tianyi+Zhou%2C+Guodong+Long%2C+Jing+Jiang%2C+and+Chengqi+Zhang%2E+Fast+directional+self%2Dattention+mechanism%2E+arXiv%2C+abs%2F1805%2E00912%2C+2018c%2E+%E2%80%94+%5BarXiv%3A1805%2E00912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E00912%29)
52. Laurent Sifre. Rigid-motion scattering for image classification. Ph.D. thesis section 6.2, 2014. [find](https://scholar.google.com/scholar?q=Laurent+Sifre%2E+Rigid%2Dmotion+scattering+for+image+classification%2E+Ph%2ED%2E+thesis+section+6%2E2%2C+2014%2E)
53. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proc. of NIPS, 2015. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+arthur+szlam%2C+Jason+Weston%2C+and+Rob+Fergus%2E+End%2Dto%2Dend+memory+networks%2E+In+Proc%2E+of+NIPS%2C+2015%2E)
54. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+George+E%2E+Dahl%2C+and+Geoffrey+E%2E+Hinton%2E+On+the+importance+of+initialization+and+momentum+in+deep+learning%2E+In+ICML%2C+2013%2E)
55. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Networks. In Proc. of NIPS, 2014. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+Sequence+to+Sequence+Learning+with+Neural+Networks%2E+In+Proc%2E+of+NIPS%2C+2014%2E)
56. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. arXiv, abs/1512.00567, 2015. [find](https://scholar.google.com/scholar?q=Christian+Szegedy%2C+Vincent+Vanhoucke%2C+Sergey+Ioffe%2C+Jonathon+Shlens%2C+and+Zbigniew+Wojna%2E+Rethinking+the+Inception+Architecture+for+Computer+Vision%2E+arXiv%2C+abs%2F1512%2E00567%2C+2015%2E)
57. Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proc. of CVPR, 2014. [find](https://scholar.google.com/scholar?q=Yaniv+Taigman%2C+Ming+Yang%2C+Marc%E2%80%99Aurelio+Ranzato%2C+and+Lior+Wolf%2E+Deepface%3A+Closing+the+gap+to+human%2Dlevel+performance+in+face+verification%2E+In+Proc%2E+of+CVPR%2C+2014%2E)
58. Gongbo Tang, Mathias Mller, Annette Rios, and Rico Sennrich. Why self-attention? a targeted evaluation of neural machine translation architectures. In Proc. of EMNLP, 2018. [find](https://scholar.google.com/scholar?q=Gongbo+Tang%2C+Mathias+Mller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+Why+self%2Dattention%3F+a+targeted+evaluation+of+neural+machine+translation+architectures%2E+In+Proc%2E+of+EMNLP%2C+2018%2E)
59. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proc. of NIPS, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+Is+All+You+Need%2E+In+Proc%2E+of+NIPS%2C+2017%2E)
60. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proc. of ICML, 2013. [find](https://scholar.google.com/scholar?q=Li+Wan%2C+Matthew+Zeiler%2C+Sixin+Zhang%2C+Yann+Le+Cun%2C+and+Rob+Fergus%2E+Regularization+of+neural+networks+using+dropconnect%2E+In+Proc%2E+of+ICML%2C+2013%2E)
61. Zhengyang Wang and Shuiwang Ji. Smoothed dilated convolutions for improved dense prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2486–2495. ACM, 2018. [find](https://scholar.google.com/scholar?q=Zhengyang+Wang+and+Shuiwang+Ji%2E+Smoothed+dilated+convolutions+for+improved+dense+prediction%2E+In+Proceedings+of+the+24th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+%26+Data+Mining%2C+pp%2E+24)
62. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv, abs/1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+Neural+Machine+Translation+System%3A+Bridging+th)
63. Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. arXiv, abs/1805.00631, 2018. [find](https://scholar.google.com/scholar?q=Biao+Zhang%2C+Deyi+Xiong%2C+and+Jinsong+Su%2E+Accelerating+neural+transformer+via+an+average+attention+network%2E+arXiv%2C+abs%2F1805%2E00631%2C+2018%2E)