---
title: "Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? — Tay 2023"
description: "Most work on scaling laws studies only the standard Transformer. 35 references, cited by 46 papers here."
authors: ["Yi Tay", "Mostafa Dehghani", "Samira Abnar", "Hyung Chung", "William Fedus", "Jinfeng Rao", "Sharan Narang", "Vinh Tran", "Dani Yogatama", "Donald Metzler"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.825/"
doi: "10.18653/v1/2023.findings-emnlp.825"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

# Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?

**Authors:** [Yi Tay](https://md.research.elamur.ai/free/en/authors/yi-tay/), [Mostafa Dehghani](https://md.research.elamur.ai/free/en/authors/mostafa-dehghani/), [Samira Abnar](https://md.research.elamur.ai/free/en/authors/samira-abnar/), [Hyung Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), [William Fedus](https://md.research.elamur.ai/free/en/authors/william-fedus/), [Jinfeng Rao](https://md.research.elamur.ai/free/en/authors/jinfeng-rao/), [Sharan Narang](https://md.research.elamur.ai/free/en/authors/sharan-narang/), Vinh Tran, [Dani Yogatama](https://md.research.elamur.ai/free/en/authors/dani-yogatama/) and [Donald Metzler](https://md.research.elamur.ai/free/en/authors/donald-metzler/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.825/) · [PDF](https://aclanthology.org/2023.findings-emnlp.825.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.825](https://doi.org/10.18653/v1/2023.findings-emnlp.825) · [OpenAlex](https://openalex.org/W4389518729)

**Citations:** 33 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/tay-2023-scaling-laws-vs-model-architectures/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Most work on scaling laws studies only the standard Transformer. New architectures are also often judged at a single model size, such as "base", on the assumption that their gains carry over to other scales. This paper tests that assumption. The authors ask whether the inductive bias built into an architecture changes how it scales, both in pretraining and after finetuning.

They trained over 100 models from ten architecture families, ranging from about 15M to 40B parameters. The families include:
- Transformer variants: Switch, Universal, Evolved, Funnel, Performer, ALBERT, GLU and MoS variants
- non-Transformer designs: lightweight and dynamic convolutions, MLP-Mixer

All models use a T5-style encoder-decoder setup with span corruption and the same fixed number of pretraining tokens. They are scored on upstream negative log-perplexity and downstream SuperGLUE accuracy.

Scaling behaviour differs a lot across architectures. The vanilla Transformer scales best, even where it is not the top model in absolute terms. The best model changes depending on the compute region, and upstream gains do not always lead to downstream gains. In practice, single-size comparisons can mislead, and radical architecture changes are risky to bet on for large training runs.

## Key points
- The study covers ten architectures and more than 100 pretrained and finetuned models, from roughly 15M up to 40B parameters. All share one encoder-decoder setup with a fixed token budget.
- The fitted slope of FLOPs versus upstream performance is highest for the vanilla Transformer (about 0.54). ALBERT (about 0.08), Switch (about 0.23) and Performer (about 0.25) are far lower. These values come from the paper's Table 2, which was partly garbled in the extracted text.
- Models that change the core Transformer only slightly, such as GLU-Transformer and MoS-Transformer, keep similar scaling trends. Linear-attention models like Performer, and parameter-sharing models like ALBERT, have trouble scaling.
- Upstream perplexity does not reliably predict SuperGLUE transfer across architectures.
- Increasing depth affects downstream results much more than increasing FFN width. LConv reacts differently to depth scaling, and Evolved Transformer gains slightly more from width scaling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

Cite as: Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama and Donald Metzler. 2023. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tay2023scaling,
  title = {{Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?}},
  author = {Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama and Donald Metzler},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.825},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.825}
}
```

- [← previous D4: Improving LLM Pretraining via Document De-Duplication and Diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/)
- [next → Coreference Resolution through a seq2seq Transition-Based System](https://md.research.elamur.ai/free/en/papers/bohnet-2023-coreference-resolution-through-seq2seq-transition/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. 2021. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095. — [arXiv:2110.02095](https://arxiv.org/abs/2110.02095) [find](https://scholar.google.com/scholar?q=Samira+Abnar%2C+Mostafa+Dehghani%2C+Behnam+Neyshabur%2C+and+Hanie+Sedghi%2E+2021%2E+Exploring+the+limits+of+large+scale+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2110%2E02095%2E+%E2%80%94+%5BarXiv%3A2110%2E02095%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
2. Samira Abnar, Mostafa Dehghani, and Willem Zuidema. 2020. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006.00555. — [arXiv:2006.00555](https://arxiv.org/abs/2006.00555) [find](https://scholar.google.com/scholar?q=Samira+Abnar%2C+Mostafa+Dehghani%2C+and+Willem+Zuidema%2E+2020%2E+Transferring+inductive+biases+through+knowledge+distillation%2E+arXiv+preprint+arXiv%3A2006%2E00555%2E+%E2%80%94+%5BarXiv%3A2006%2E00555%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006)
3. Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701. — [arXiv:2102.06701](https://arxiv.org/abs/2102.06701) [find](https://scholar.google.com/scholar?q=Yasaman+Bahri%2C+Ethan+Dyer%2C+Jared+Kaplan%2C+Jaehoon+Lee%2C+and+Utkarsh+Sharma%2E+2021%2E+Explaining+neural+scaling+laws%2E+arXiv+preprint+arXiv%3A2102%2E06701%2E+%E2%80%94+%5BarXiv%3A2102%2E06701%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E06701%29)
4. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794. [find](https://scholar.google.com/scholar?q=Krzysztof+Choromanski%2C+Valerii+Likhosherstov%2C+David+Dohan%2C+Xingyou+Song%2C+Andreea+Gane%2C+Tamas+Sarlos%2C+Peter+Hawkins%2C+Jared+Davis%2C+Afroz+Mohiuddin%2C+Lukasz+Kaiser%2C+et+al%2E+2020%2E+Rethinking+attention+with+)
5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [our card](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
7. Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. arXiv preprint arXiv:2006.03236. — [arXiv:2006.03236](https://arxiv.org/abs/2006.03236) [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Guokun+Lai%2C+Yiming+Yang%2C+and+Quoc+V+Le%2E+2020%2E+Funnel%2Dtransformer%3A+Filtering+out+sequential+redundancy+for+efficient+language+processing%2E+arXiv+preprint+arXiv%3A2006%2E03236%2E+%E2%80%94+%5BarXiv%3A2006%2E0323)
8. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 933–941. PMLR. [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
9. Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. 2021a. The efficiency misnomer. arXiv preprint arXiv:2110.12894. — [arXiv:2110.12894](https://arxiv.org/abs/2110.12894) [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Anurag+Arnab%2C+Lucas+Beyer%2C+Ashish+Vaswani%2C+and+Yi+Tay%2E+2021a%2E+The+efficiency+misnomer%2E+arXiv+preprint+arXiv%3A2110%2E12894%2E+%E2%80%94+%5BarXiv%3A2110%2E12894%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2110%2E12894%29)
10. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Oriol+Vinyals%2C+Jakob+Uszkoreit%2C+and+%C5%81ukasz+Kaiser%2E+2018%2E+Universal+transformers%2E+arXiv+preprint+arXiv%3A1807%2E03819%2E)
11. Mostafa Dehghani, Yi Tay, Alexey A Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. 2021b. The benchmark lottery. arXiv preprint arXiv:2107.07002. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Yi+Tay%2C+Alexey+A+Gritsenko%2C+Zhe+Zhao%2C+Neil+Houlsby%2C+Fernando+Diaz%2C+Donald+Metzler%2C+and+Oriol+Vinyals%2E+2021b%2E+The+benchmark+lottery%2E+arXiv+preprint+arXiv%3A2107%2E07002%2E)
12. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2E)
13. Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, and Alexis Conneau. 2021. Larger-Scale Transformers for Multilingual Masked Language Modeling. arXiv e-prints, page arXiv:2105.00572. [our card](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/)
14. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701. [our card](https://md.research.elamur.ai/free/en/papers/henighan-2020-scaling-laws-autoregressive-generative-modeling/)
15. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293. [our card](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/)
16. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simo… [our card](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
17. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
18. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1)
20. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. Scaling data-constrained language models. [find](https://scholar.google.com/scholar?q=Niklas+Muennighoff%2C+Alexander+M%2E+Rush%2C+Boaz+Barak%2C+Teven+Le+Scao%2C+Aleksandra+Piktus%2C+Nouamane+Tazi%2C+Sampo+Pyysalo%2C+Thomas+Wolf%2C+and+Colin+Raffel%2E+2023%2E+Scaling+data%2Dconstrained+language+models%2E)
21. Sharan Narang, Hyung Won Chung, Yi Tay, William Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, et al. 2021. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11972. [find](https://scholar.google.com/scholar?q=Sharan+Narang%2C+Hyung+Won+Chung%2C+Yi+Tay%2C+William+Fedus%2C+Thibault+Fevry%2C+Michael+Matena%2C+Karishma+Malkan%2C+Noah+Fiedel%2C+Noam+Shazeer%2C+Zhenzhong+Lan%2C+et+al%2E+2021%2E+Do+transformer+modifications+transfer+acr)
22. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
25. Noam Shazeer. 2020. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+2020%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2E)
26. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. 2018. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423. [our card](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/)
27. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+International+Conference+on+Machine+Learning%2C+pages+4596%E2%80%934604%2E+PMLR%2E)
28. David So, Quoc Le, and Chen Liang. 2019. The evolved transformer. In International Conference on Machine Learning, pages 5877–5886. PMLR. [find](https://scholar.google.com/scholar?q=David+So%2C+Quoc+Le%2C+and+Chen+Liang%2E+2019%2E+The+evolved+transformer%2E+In+International+Conference+on+Machine+Learning%2C+pages+5877%E2%80%935886%2E+PMLR%2E)
29. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+arXiv+preprint+arXiv%3A1409%2E3215%2E)
30. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. arXiv preprint arXiv:2009.06732. [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)
31. Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. 2021a. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint arXiv:2105.03322. — [arXiv:2105.03322](https://arxiv.org/abs/2105.03322) [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jai+Gupta%2C+Dara+Bahri%2C+Vamsi+Aribandi%2C+Zhen+Qin%2C+and+Donald+Metzler%2E+2021a%2E+Are+pre%2Dtrained+convolutions+better+than+pre%2Dtrained+transformers%3F+arXiv+preprint+arXiv%3A2105%2E03322)
32. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021b. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686. — [arXiv:2109.10686](https://arxiv.org/abs/2109.10686) [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Jinfeng+Rao%2C+William+Fedus%2C+Samira+Abnar%2C+Hyung+Won+Chung%2C+Sharan+Narang%2C+Dani+Yogatama%2C+Ashish+Vaswani%2C+and+Donald+Metzler%2E+2021b%2E+Scale+efficiently%3A+Insights+from+pre%2Dtrain)
33. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131. [our card](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
34. Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601. — [arXiv:2105.01601](https://arxiv.org/abs/2105.01601) [find](https://scholar.google.com/scholar?q=Ilya+Tolstikhin%2C+Neil+Houlsby%2C+Alexander+Kolesnikov%2C+Lucas+Beyer%2C+Xiaohua+Zhai%2C+Thomas+Unterthiner%2C+Jessica+Yung%2C+Daniel+Keysers%2C+Jakob+Uszkoreit%2C+Mario+Lucic%2C+et+al%2E+2021%2E+Mlp%2Dmixer%3A+An+all%2Dmlp+archi)
35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)