---
title: "Adaptively Sparse Transformers — Correia 2019"
description: "Standard Transformers normalize every attention head with softmax, so each head spreads at least some weight over all context tokens."
authors: ["Gonçalo M. Correia", "Vlad Niculae", "André F. T. Martins"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1223/"
doi: "10.18653/v1/d19-1223"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Adaptively Sparse Transformers

# Adaptively Sparse Transformers

**Authors:** Gonçalo M. Correia, Vlad Niculae, [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1223/) · [PDF](https://aclanthology.org/D19-1223.pdf) · [DOI 10.18653/v1/d19-1223](https://doi.org/10.18653/v1/d19-1223) · [OpenAlex](https://openalex.org/W2970777192)

**Citations:** 198 citations (openalex-snapshot · 2026-10-14) · [72](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard Transformers normalize every attention head with softmax, so each head spreads at least some weight over all context tokens. Earlier sparse variants (Sparse Transformer, adaptive span Transformer) restrict heads only to a contiguous window of preceding tokens. The authors swap softmax for α-entmax, a differentiable mapping that can give irrelevant tokens exactly zero weight. Its parameter α ranges from softmax (α = 1) to sparsemax (α = 2). Their main technical step is a closed-form derivative of the entmax output with respect to α. This makes α an ordinary trainable parameter, so each head learns its own shape by gradient descent instead of through a costly per-head search. They tested softmax, fixed 1.5-entmax and adaptive α-entmax Transformers on four machine translation tasks. The sparse models matched or slightly beat softmax accuracy. They were also easier to interpret, with heads that were more diverse and more specialized. Examples include near-deterministic positional heads and a head that joins subwords. This matters because sparsity and adaptivity can be added without any loss in quality.

## Key points
- α-entmax unifies softmax (α = 1) and sparsemax (α = 2). Values in between yield increasingly sparse attention, and the paper adds a closed-form Jacobian with respect to α (Proposition 1, derived via implicit differentiation).
- Three systems were compared on four MT tasks: softmax, fixed 1.5-entmax (its first use inside a Transformer) and adaptive α-entmax with a separately learned α for each head.
- Learned α values differ widely across heads. Examples include a previous-position head at α = 1.91 and a nearly dense head at α = 1.05 that captures the noun phrase "Armani Polo". Heads in later layers tend to be denser on average.
- Sparse heads can pick tokens that are not contiguous and not tied to position. Even a single head (α = 1.33) can vary its sparsity from one input to the next.
- Code and a pip package are released at github.com/deep-spin/entmax.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

72 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 67 not yet.

- [ETC: Encoding Long and Structured Inputs in Transformers](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/)
- [Perceiver: General Perception with Iterative Attention](https://md.research.elamur.ai/free/en/papers/jaegle-2021-perceiver-general-perception-iterative-attention/)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/)
- [Efficient Transformers: A Survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{correia2019adaptively,
  title = {{Adaptively Sparse Transformers}},
  author = {Correia, Gonçalo M. and Niculae, Vlad and Martins, André F. T.},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1223},
  url = {https://doi.org/10.18653/v1/d19-1223}
}
```

- [← previous Adaptive Attention Span in Transformers](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/)
- [next → Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modeling](https://md.research.elamur.ai/free/en/papers/acl-2019-zalmout-adversarial-multitask-learning-joint-multi/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brandon Amos and J. Zico Kolter. 2017. OptNet: Differentiable optimization as a layer in neural networks. In Proc. ICML. [find](https://scholar.google.com/scholar?q=Brandon+Amos+and+J%2E+Zico+Kolter%2E+2017%2E+OptNet%3A+Differentiable+optimization+as+a+layer+in+neural+networks%2E+In+Proc%2E+ICML%2E)
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Dzmitry+Bahdanau%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2015%2E+Neural+machine+translation+by+jointly+learning+to+align+and+translate%2E+In+Proc%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
3. Mathieu Blondel, André FT Martins, and Vlad Niculae. 2019. Learning classifiers with Fenchel-Young losses: Generalized entropies, margins, and algorithms. In Proc. AISTATS. [find](https://scholar.google.com/scholar?q=Mathieu+Blondel%2C+Andre%CC%81+FT+Martins%2C+and+Vlad+Niculae%2E+2019%2E+Learning+classifiers+with+Fenchel%2DYoung+losses%3A+Generalized+entropies%2C+margins%2C+and+algorithms%2E+In+Proc%2E+AISTATS%2E)
4. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. 2014. Findings of the 2014 workshop on statistical machine translation. In Proc. Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Ondrej+Bojar%2C+Christian+Buck%2C+Christian+Federmann%2C+Barry+Haddow%2C+Philipp+Koehn%2C+Johannes+Leveling%2C+Christof+Monz%2C+Pavel+Pecina%2C+Matt+Post%2C+Herve+Saint%2DAmand%2C+et+al%2E+2014%2E+Findings+of+the+2014+workshop) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
5. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 conference on machine translation. In Proc. WMT. [find](https://scholar.google.com/scholar?q=Ondrej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Antonio+Jimeno+Yepes%2C+Philipp+Koehn%2C+Varvara+Logacheva%2C+Christof+Monz%2C+et+al%2E+2016%2E+Findings+of+the+201) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
6. M Cettolo, M Federico, L Bentivogli, J Niehues, S Stüker, K Sudoh, K Yoshino, and C Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proc. IWSLT. [find](https://scholar.google.com/scholar?q=M+Cettolo%2C+M+Federico%2C+L+Bentivogli%2C+J+Niehues%2C+S+Stu%CC%88ker%2C+K+Sudoh%2C+K+Yoshino%2C+and+C+Federmann%2E+2017%2E+Overview+of+the+IWSLT+2017+evaluation+campaign%2E+In+Proc%2E+IWSLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/)
7. Sameen Maruf, André FT Martins, and Gholamreza Haffari. 2019. Selective attention for context-aware neural machine translation. preprint arXiv:1903.08788. [find](https://scholar.google.com/scholar?q=Sameen+Maruf%2C+Andre%CC%81+FT+Martins%2C+and+Gholamreza+Haffari%2E+2019%2E+Selective+attention+for+context%2Daware+neural+machine+translation%2E+preprint+arXiv%3A1903%2E08788%2E) [our card](https://md.research.elamur.ai/free/en/papers/maruf-2019-selective-attention-context-aware-neural/)
8. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse Transformers. preprint arXiv:1904.10509. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+long+sequences+with+sparse+Transformers%2E+preprint+arXiv%3A1904%2E10509%2E)
9. Graham Neubig. 2011. The Kyoto free translation task. [http://www.phontron.com/kftt](http://www.phontron.com/kftt). [↗](http://www.phontron.com/kftt)
10. Frank H Clarke. 1990. Optimization and Nonsmooth Analysis. SIAM. [find](https://scholar.google.com/scholar?q=Frank+H+Clarke%2E+1990%2E+Optimization+and+Nonsmooth+Analysis%2E+SIAM%2E)
11. Vlad Niculae and Mathieu Blondel. 2017. A regularized framework for sparse and structured neural attention. In Proc. NeurIPS. [find](https://scholar.google.com/scholar?q=Vlad+Niculae+and+Mathieu+Blondel%2E+2017%2E+A+regularized+framework+for+sparse+and+structured+neural+attention%2E+In+Proc%2E+NeurIPS%2E)
12. Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. 2018. Latent alignment and variational attention. In Proc. NeurIPS. [find](https://scholar.google.com/scholar?q=Yuntian+Deng%2C+Yoon+Kim%2C+Justin+Chiu%2C+Demi+Guo%2C+and+Alexander+Rush%2E+2018%2E+Latent+alignment+and+variational+attention%2E+In+Proc%2E+NeurIPS%2E)
13. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. WMT. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+David+Grangier%2C+and+Michael+Auli%2E+2018%2E+Scaling+neural+machine+translation%2E+In+Proc%2E+WMT%2E) [our card](https://md.research.elamur.ai/free/en/papers/ott-2018-scaling-neural-machine-translation/)
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proc%2E+NAACL%2DHLT%2E)
15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
16. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proc. ICML. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+N+Dauphin%2E+2017%2E+Convolutional+sequence+to+sequence+learning%2E+In+Proc%2E+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
17. Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. 2016. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. preprint arXiv:1607.05447. — [arXiv:1607.05447](https://arxiv.org/abs/1607.05447) [find](https://scholar.google.com/scholar?q=Stephen+Gould%2C+Basura+Fernando%2C+Anoop+Cherian%2C+Peter+Anderson%2C+Rodrigo+Santa+Cruz%2C+and+Edison+Guo%2E+2016%2E+On+differentiating+parameterized+argmin+and+argmax+problems+with+application+to+bi%2Dlevel+optimi)
18. Michael Held, Philip Wolfe, and Harlan P Crowder. 1974. Validation of subgradient optimization. Mathematical Programming, 6(1):62–88. [find](https://scholar.google.com/scholar?q=Michael+Held%2C+Philip+Wolfe%2C+and+Harlan+P+Crowder%2E+1974%2E+Validation+of+subgradient+optimization%2E+Mathematical+Programming%2C+6%281%29%3A62%E2%80%9388%2E)
19. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proc. NAACL-HLT. [find](https://scholar.google.com/scholar?q=Sarthak+Jain+and+Byron+C%2E+Wallace%2E+2019%2E+Attention+is+not+explanation%2E+In+Proc%2E+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/)
20. Marcin Junczys-Dowmunt, Kenneth Heafield, Hieu Hoang, Roman Grundkiewicz, and Anthony Aue. 2018. Marian: Cost-effective high-quality neural machine translation in C++. In Proc. WNMT. [find](https://scholar.google.com/scholar?q=Marcin+Junczys%2DDowmunt%2C+Kenneth+Heafield%2C+Hieu+Hoang%2C+Roman+Grundkiewicz%2C+and+Anthony+Aue%2E+2018%2E+Marian%3A+Cost%2Deffective+high%2Dquality+neural+machine+translation+in+C%2B%2B%2E+In+Proc%2E+WNMT%2E)
21. Jian Li, Zhaopeng Tu, Baosong Yang, Michael R Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Jian+Li%2C+Zhaopeng+Tu%2C+Baosong+Yang%2C+Michael+R+Lyu%2C+and+Tong+Zhang%2E+2018%2E+Multi%2DHead+Attention+with+Disagreement+Regularization%2E+In+Proc%2E+EMNLP%2E)
22. Ben Peters, Vlad Niculae, and André FT Martins. 2019. Sparse sequence-to-sequence models. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Ben+Peters%2C+Vlad+Niculae%2C+and+Andre%CC%81+FT+Martins%2E+2019%2E+Sparse+sequence%2Dto%2Dsequence+models%2E+In+Proc%2E+ACL%2E)
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. preprint. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+preprint%2E)
24. Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in Transformer-based machine translation. In Proc. BlackboxNLP. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato+and+Jo%CC%88rg+Tiedemann%2E+2018%2E+An+analysis+of+encoder+representations+in+Transformer%2Dbased+machine+translation%2E+In+Proc%2E+BlackboxNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/raganato-2018-analysis-encoder-representations-transformer-based/)
25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proc%2E+ACL%2E)
26. Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. 2019. SSN: Learning sparse switchable normalization via SparsestMax. In Proc. CVPR. [find](https://scholar.google.com/scholar?q=Wenqi+Shao%2C+Tianjian+Meng%2C+Jingyu+Li%2C+Ruimao+Zhang%2C+Yudian+Li%2C+Xiaogang+Wang%2C+and+Ping+Luo%2E+2019%2E+SSN%3A+Learning+sparse+switchable+normalization+via+SparsestMax%2E+In+Proc%2E+CVPR%2E)
27. Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive Attention Span in Transformers. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+and+Armand+Joulin%2E+2019%2E+Adaptive+Attention+Span+in+Transformers%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/)
28. Christos Louizos, Max Welling, and Diederik P Kingma. 2018. Learning sparse neural networks through L 0 regularization. Proc. ICLR. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+L+0+regularization%2E+Proc%2E+ICLR%2E)
29. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? A targeted evaluation of neural machine translation architectures. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Gongbo+Tang%2C+Mathias+Mu%CC%88ller%2C+Annette+Rios%2C+and+Rico+Sennrich%2E+2018%2E+Why+self%2Dattention%3F+A+targeted+evaluation+of+neural+machine+translation+architectures%2E+In+Proc%2E+EMNLP%2E)
30. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015%2E+Effective+approaches+to+attention%2Dbased+neural+machine+translation%2E+In+Proc%2E+EMNLP%2E)
31. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
32. Chaitanya Malaviya, Pedro Ferreira, and André FT Martins. 2018. Sparse and constrained attention for neural machine translation. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Chaitanya+Malaviya%2C+Pedro+Ferreira%2C+and+Andre%CC%81+FT+Martins%2E+2018%2E+Sparse+and+constrained+attention+for+neural+machine+translation%2E+In+Proc%2E+ACL%2E)
33. Constantino Tsallis. 1988. Possible generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52:479–487. [find](https://scholar.google.com/scholar?q=Constantino+Tsallis%2E+1988%2E+Possible+generalization+of+Boltzmann%2DGibbs+statistics%2E+Journal+of+Statistical+Physics%2C+52%3A479%E2%80%93487%2E)
34. David Mareček and Rudolf Rosa. 2018. Extracting syntactic trees from Transformer encoder self-attentions. In Proc. BlackboxNLP. [find](https://scholar.google.com/scholar?q=David+Marec%CC%8Cek+and+Rudolf+Rosa%2E+2018%2E+Extracting+syntactic+trees+from+Transformer+encoder+self%2Dattentions%2E+In+Proc%2E+BlackboxNLP%2E)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proc%2E+NeurIPS%2E)
36. André FT Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=Andre%CC%81+FT+Martins+and+Ramo%CC%81n+Fernandez+Astudillo%2E+2016%2E+From+softmax+to+sparsemax%3A+A+sparse+model+of+attention+and+multi%2Dlabel+classification%2E+In+Proc%2E+of+ICML%2E)
37. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+Pavel+Serdyukov%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2018%2E+Context%2Daware+neural+machine+translation+learns+anaphora+resolution%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/voita-2018-context-aware-neural-machine-translation/)
38. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+2019%2E+Analyzing+multi%2Dhead+self%2Dattention%3A+Specialized+heads+do+the+heavy+lifting%2C+the+rest+can+be+pruned%2E+In+Proc%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
39. Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In Proc. ICLR. [find](https://scholar.google.com/scholar?q=Felix+Wu%2C+Angela+Fan%2C+Alexei+Baevski%2C+Yann+N+Dauphin%2C+and+Michael+Auli%2E+2019%2E+Pay+less+attention+with+lightweight+and+dynamic+convolutions%2E+In+Proc%2E+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-pay-less-attention-lightweight-dynamic/)