---
title: "CoLT5 — paper by Ainslie et al. (2023) — summary and ideas"
description: "Encoding long documents with Transformers is costly. 43 references, cited by 52 papers here."
authors: ["Joshua Ainslie", "Tao Lei", "Michiel de Jong", "Santiago Ontañón", "Siddhartha Brahma", "Yury Zemlyanskiy", "David Uthus", "Mandy Guo", "James Lee-Thorp", "Yi Tay", "Yun-Hsuan Sung", "Sumit Sanghai"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.309/"
doi: "10.18653/v1/2023.emnlp-main.309"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ainslie-2023-colt5-faster-long-range-transformers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ainslie-2023-colt5-faster-long-range-transformers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / CoLT5: Faster Long-Range Transformers with Conditional Computation

# CoLT5: Faster Long-Range Transformers with Conditional Computation

**Authors:** [Joshua Ainslie](https://md.research.elamur.ai/free/en/authors/joshua-ainslie/), [Tao Lei](https://md.research.elamur.ai/free/en/authors/tao-lei/), [Michiel de Jong](https://md.research.elamur.ai/free/en/authors/michiel-de-jong/), [Santiago Ontañón](https://md.research.elamur.ai/free/en/authors/santiago-ontanon/), [Siddhartha Brahma](https://md.research.elamur.ai/free/en/authors/siddhartha-brahma/), [Yury Zemlyanskiy](https://md.research.elamur.ai/free/en/authors/yury-zemlyanskiy/), [David Uthus](https://md.research.elamur.ai/free/en/authors/david-uthus/), [Mandy Guo](https://md.research.elamur.ai/free/en/authors/mandy-guo/), [James Lee-Thorp](https://md.research.elamur.ai/free/en/authors/james-lee-thorp/), [Yi Tay](https://md.research.elamur.ai/free/en/authors/yi-tay/), [Yun-Hsuan Sung](https://md.research.elamur.ai/free/en/authors/yun-hsuan-sung/) and [Sumit Sanghai](https://md.research.elamur.ai/free/en/authors/sumit-sanghai/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.309/) · [PDF](https://aclanthology.org/2023.emnlp-main.309.pdf) · [DOI 10.18653/v1/2023.emnlp-main.309](https://doi.org/10.18653/v1/2023.emnlp-main.309) · [OpenAlex](https://openalex.org/W4389524599)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/ainslie-2023-colt5-faster-long-range-transformers/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Encoding long documents with Transformers is costly. Sparse attention reduces the quadratic attention cost, but the feedforward and projection layers still run on every token and dominate compute in larger models. CoLT5 (Conditional LongT5), from Google Research, extends LongT5 with conditional computation. Each encoder layer has two branches. A cheap branch processes all tokens with local attention and a narrow MLP. A heavier branch handles only a small set of tokens chosen by learned routers. There are separate routers for the MLP, for attention queries and for attention keys/values. The model also uses multi-query cross-attention for faster decoding and the UL2 pre-training objective, which supports few-shot use with long inputs. Across Base, Large and XL sizes, CoLT5 beats LongT5 at every speed level for both fine-tuning and inference. It sets a new SOTA on the SCROLLS benchmark, improves on arXiv summarization and TriviaQA, and keeps gaining from inputs as long as 64k tokens. This makes very long-context encoding much more affordable.

## Key points
- **Routing:** Each token gets a score from a dot product with a learned vector. Scores are normalized with a generalized softmax, the top-k tokens are kept, and the heavy branch's output is weighted by the routing score so the router receives a learning signal.
- **Feedforward cost:** The light MLP uses half the standard T5 hidden size and the heavy MLP uses four times that size. With 1/16 of tokens routed to the heavy MLP, the layer costs about 75% of a standard T5 feedforward layer's FLOPs.
- **Ablations:** Static routing (fixed, evenly spaced tokens) sharply hurts quality. Moving from 512 to 1024 routed tokens gives clear gains, and larger counts bring diminishing returns. Multi-head cross-attention is slightly better than MQA but roughly an order of magnitude slower at inference.
- **Router behaviour:** On the TriviaQA dev set, question and answer tokens are routed to heavy paths far more often than other tokens.
- **Limitations:** Conditional computation is applied only in the encoder because the router is not causal. The model must also be trained from scratch.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung and Sumit Sanghai. 2023. CoLT5: Faster Long-Range Transformers with Conditional Computation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ainslie2023colt5,
  title = {{CoLT5: Faster Long-Range Transformers with Conditional Computation}},
  author = {Joshua Ainslie, Tao Lei, Michiel {de Jong}, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung and Sumit Sanghai},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.309},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.309}
}
```

- [← previous DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://md.research.elamur.ai/free/en/papers/neeman-2023-disentqa-disentangling-parametric-contextual-knowledge/)
- [next → InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

43 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, Santiago Ontañón, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. [ETC: Encoding long and structured inputs in transformers](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/). arXiv preprint arXiv:2004.08483.
2. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
3. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. [SummScreen: A dataset for abstractive screenplay summarization](https://md.research.elamur.ai/free/en/papers/chen-2022-summscreen-dataset-abstractive-screenplay-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8602–8615, Dublin, Ireland. Association for Computati…
4. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+2019%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2E)
5. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. [A discourse-aware attention model for abstractive summarization of long documents](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational…
6. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan…
7. Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, and William Cohen. 2022. FiDO: Fusion-in-decoder optimized for stronger performance and faster inference. arXiv preprint arXiv:2212.08153. [find](https://scholar.google.com/scholar?q=Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Joshua+Ainslie%2C+Nicholas+FitzGerald%2C+Sumit+Sanghai%2C+Fei+Sha%2C+and+William+Cohen%2E+2022%2E+FiDO%3A+Fusion%2Din%2Ddecoder+optimized+for+stronger+performance+and+faster+inference)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
9. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2E)
10. Google. 2020. Profile your model with cloud tpu tools. [https://cloud.google.com/tpu/docs/](https://cloud.google.com/tpu/docs/) cloud-tpu-tools. Accessed: 2022-11-11. [↗](https://cloud.google.com/tpu/docs/)
11. Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontañón, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. [LongT5: Efficient text-to-text transformer for long sequences](https://md.research.elamur.ai/free/en/papers/guo-2022-longt5-efficient-text-text-transformer/). In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724–736, Seattle, United States. Association for…
12. Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. 2020. Flax: A neural network library and ecosystem for JAX. [find](https://scholar.google.com/scholar?q=Jonathan+Heek%2C+Anselm+Levskaya%2C+Avital+Oliver%2C+Marvin+Ritter%2C+Bertrand+Rondepierre%2C+Andreas+Steiner%2C+and+Marc+van+Zee%2E+2020%2E+Flax%3A+A+neural+network+library+and+ecosystem+for+JAX%2E)
13. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online.…
14. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Association for Computationa…
15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). CoRR, abs/2001.08361.
16. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The NarrativeQA reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317– 328.
17. Yuta Koreeda and Christopher Manning. 2021. [ContractNLI: A dataset for document-level natural language inference for contracts](https://md.research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919, Punta Cana, Dominican Republic. Association for Computational Linguistics.
18. Bernhard Kratzwald and Stefan Feuerriegel. 2018. Adaptive document retrieval for deep question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 576–581. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Bernhard+Kratzwald+and+Stefan+Feuerriegel%2E+2018%2E+Adaptive+document+retrieval+for+deep+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Bru)
19. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natur…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
20. Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-Wei Chang. 2023. Conditional adapters: Parameter-efficient transfer learning with fast inference. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tao+Lei%2C+Junwen+Bai%2C+Siddhartha+Brahma%2C+Joshua+Ainslie%2C+Kenton+Lee%2C+Yanqi+Zhou%2C+Nan+Du%2C+Vincent+Y%2E+Zhao%2C+Yuexin+Wu%2C+Bo+Li%2C+Yu+Zhang%2C+and+Ming%2DWei+Chang%2E+2023%2E+Conditional+adapters%3A+Parameter%2Defficient)
21. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Reader-guided passage reranking for [open-domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2…
22. Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2021. QuAL-ITY: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608. — [arXiv:2112.08608](https://arxiv.org/abs/2112.08608) [find](https://scholar.google.com/scholar?q=Richard+Yuanzhe+Pang%2C+Alicia+Parrish%2C+Nitish+Joshi%2C+Nikita+Nangia%2C+Jason+Phang%2C+Angelica+Chen%2C+Vishakh+Padmakumar%2C+Johnny+Ma%2C+Jana+Thompson%2C+He+He%2C+and+Samuel+R%2E+Bowman%2E+2021%2E+QuAL%2DITY%3A+Question+answe)
23. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2022. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102. [find](https://scholar.google.com/scholar?q=Reiner+Pope%2C+Sholto+Douglas%2C+Aakanksha+Chowdhery%2C+Jacob+Devlin%2C+James+Bradbury%2C+Anselm+Levskaya%2C+Jonathan+Heek%2C+Kefan+Xiao%2C+Shivani+Agrawal%2C+and+Jeff+Dean%2E+2022%2E+Efficiently+scaling+transformer+infere)
24. Yujie Qian, Jinhyuk Lee, Sai Meher Karthik Duddu, Zhuyun Dai, Siddhartha Brahma, Iftekhar Naim, Tao Lei, and Vincent Y Zhao. 2022. Multi-vector retrieval as sparse alignment. arXiv preprint arXiv:2211.01267. — [arXiv:2211.01267](https://arxiv.org/abs/2211.01267) [find](https://scholar.google.com/scholar?q=Yujie+Qian%2C+Jinhyuk+Lee%2C+Sai+Meher+Karthik+Duddu%2C+Zhuyun+Dai%2C+Siddhartha+Brahma%2C+Iftekhar+Naim%2C+Tao+Lei%2C+and+Vincent+Y+Zhao%2E+2022%2E+Multi%2Dvector+retrieval+as+sparse+alignment%2E+arXiv+preprint+arXiv%3A2211)
25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
26. Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashch… [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Hyung+Won+Chung%2C+Anselm+Levskaya%2C+Gaurav+Mishra%2C+James+Bradbury%2C+Daniel+Andor%2C+Sharan+Narang%2C+Brian+Lester%2C+Colin+Gaffney%2C+Afroz+Mohiuddin%2C+Curtis+Hawthorne%2C+Aitor+Lewkowycz%2C+Alex+Salcia)
27. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. arXiv preprint arXiv:2207.07061. — [arXiv:2207.07061](https://arxiv.org/abs/2207.07061) [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Adam+Fisch%2C+Jai+Gupta%2C+Mostafa+Dehghani%2C+Dara+Bahri%2C+Vinh+Q+Tran%2C+Yi+Tay%2C+and+Donald+Metzler%2E+2022%2E+Confident+adaptive+language+modeling%2E+arXiv+preprint+arXiv%3A2207%2E07061%2E+%E2%80%94+%5BarXiv%3A2207%2E0)
28. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. [Scrolls: Standardized comparison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). ArXiv, abs/2201.03533.
29. Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2019%2E+Fast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2E+arXiv+preprint+arXiv%3A1911%2E02150%2E)
30. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2… [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+V%2E+Le%2C+Geoffrey+E%2E+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+In)
31. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, page… [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+ICML+2018%2C+Stockholmsm%C3%A4ss)
32. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. [Long range arena : A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). In International Conference on Learning Representations.
33. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. [Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). arXiv preprint arXiv:2205.05131.
34. Neeraj Varshney, Man Luo, and Chitta Baral. 2022. Can open-domain QA reader utilize external knowledge efficiently like humans? CoRR, abs/2211.12707. — [arXiv:2211.12707](https://arxiv.org/abs/2211.12707) [find](https://scholar.google.com/scholar?q=Neeraj+Varshney%2C+Man+Luo%2C+and+Chitta+Baral%2E+2022%2E+Can+open%2Ddomain+QA+reader+utilize+external+knowledge+efficiently+like+humans%3F+CoRR%2C+abs%2F2211%2E12707%2E+%E2%80%94+%5BarXiv%3A2211%2E12707%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2211%2E12)
35. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3 : Reinforced ranker-reader for open-domain question answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th… [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Mo+Yu%2C+Xiaoxiao+Guo%2C+Zhiguo+Wang%2C+Tim+Klinger%2C+Wei+Zhang%2C+Shiyu+Chang%2C+Gerry+Tesauro%2C+Bowen+Zhou%2C+and+Jing+Jiang%2E+2018%2E+R3+%3A+Reinforced+ranker%2Dreader+for+open%2Ddomain+question+answering%2E)
36. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. [find](https://scholar.google.com/scholar?q=Sinong+Wang%2C+Belinda+Z+Li%2C+Madian+Khabsa%2C+Han+Fang%2C+and+Hao+Ma%2E+2020%2E+Linformer%3A+Self%2Dattention+with+linear+complexity%2E+arXiv+preprint+arXiv%3A2006%2E04768%2E)
37. Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2022. [Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering](https://md.research.elamur.ai/free/en/papers/yu-2022-kg-fid-infusing-knowledge-graph/). In Proceedings of the 60th Annual Meeting of the Association for Computational Lingui…
38. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in Neural Information Processing Systems, 33:17283–17297.
39. Yury Zemlyanskiy, Joshua Ainslie, Michiel de Jong, Philip Pham, Ilya Eckstein, and Fei Sha. 2021. Readtwice: Reading very large documents with memories. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Yury+Zemlyanskiy%2C+Joshua+Ainslie%2C+Michiel+de+Jong%2C+Philip+Pham%2C+Ilya+Eckstein%2C+and+Fei+Sha%2E+2021%2E+Readtwice%3A+Reading+very+large+documents+with+memories%2E+In+Proceedings+of+the+2021+Conference+of+the+No)
40. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In International Conference on Machine Learning, pages 11328–11339. PMLR.
41. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. [QMSum: A new benchmark for query-based multi-domain meeting summarization](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/). In Proceedings of the 2021 Conference of the North American Chapte…
42. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Irwan+Bello%2C+Sameer+Kumar%2C+Nan+Du%2C+Yanping+Huang%2C+Jeff+Dean%2C+Noam+Shazeer%2C+and+William+Fedus%2E+2022%2E+St%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2E+arXiv+preprint+arXiv%3A22)
43. Layers Model dim MLP light dim MLP heavy dim Heads light Heads heavy Params [find](https://scholar.google.com/scholar?q=Layers+Model+dim+MLP+light+dim+MLP+heavy+dim+Heads+light+Heads+heavy+Params)

*Updated 24 October 2026*