---
title: "Intrinsic Dimensionality Explains the — Aghajanyan 2021"
description: "The authors explain why huge pretrained language models with hundreds of millions of parameters can be fine-tuned with ordinary gradient descent on a few…"
authors: ["Armen Aghajanyan", "Sonal Gupta", "Luke Zettlemoyer"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2021.acl-long.568/"
doi: "10.18653/v1/2021.acl-long.568"
arxiv: "2012.13255"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

# Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

**Authors:** [Armen Aghajanyan](https://md.research.elamur.ai/free/en/authors/armen-aghajanyan/), [Sonal Gupta](https://md.research.elamur.ai/free/en/authors/sonal-gupta/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.568/) · [PDF](https://aclanthology.org/2021.acl-long.568.pdf) · [DOI 10.18653/v1/2021.acl-long.568](https://doi.org/10.18653/v1/2021.acl-long.568) · [arXiv 2012.13255](https://arxiv.org/abs/2012.13255) · [OpenAlex](https://openalex.org/W3118216348)

**Citations:** 335 citations (openalex-snapshot · 2026-10-14) · [195](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors explain why huge pretrained language models with hundreds of millions of parameters can be fine-tuned with ordinary gradient descent on a few hundred or a few thousand labeled examples without overfitting. To answer this, they use the notion of intrinsic dimension. This is the smallest number of free parameters whose optimization in a random subspace yields 90% of the performance of full fine-tuning (the d90 threshold). Parameters from the small subspace are mapped into the full weight space using the Fastfood transform. The authors also propose a variant, SAID, in which a separate scale is learned for each layer. The experiments cover GLUE tasks and other datasets and more than a dozen models: BERT, RoBERTa, T5, XLM-R, Albert, XLNet, ELECTRA. They show that hundreds or thousands of parameters are usually sufficient. This dimension decreases during pre-training and is lower in larger models. A lower dimension is also accompanied by higher validation accuracy and a smaller generalization gap. Finally, the authors derive a generalization bound that depends on d rather than on the full number of parameters. The work offers a theoretical explanation for the success of fine-tuning in low-data regimes and supports parameter-efficient fine-tuning methods.

## Key points
- RoBERTa on MRPC reaches 90% of the performance of full fine-tuning when only about 200 parameters are trained. According to Table 1 (the SAID method), d90 for RoBERTa-Large is 207 on MRPC and 774 on QQP, and for BERT-Base it is 1608 and 8030.
- SAID consistently yields lower d90 estimates than the DID method, which does not take the model's division into layers into account.
- For RoBERTa, a dense projection matrix at d = 1000 would take approximately 1.3–1.4 TB of memory. The authors therefore use Fastfood: a product of Hadamard matrices, random diagonal matrices and a permutation, computed in O(D log d).
- Figure 3 compares models ranging from approximately 10⁸ to 10⁹+ parameters on MRPC: the larger the model, the smaller its d90. During RoBERTa pre-training, d90 gradually decreases.
- The generalization bound is of order √(d/m), where m is the number of examples. It extends the compression bound from Arora et al. (2018). The empirical relationship with the generalization gap is verified on MRPC, QQP, Yelp, SST-2, MNLI and ANLI.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

195 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 188 not yet.

- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/valipour-2023-dylora-parameter-efficient-tuning-pre/)
- [Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/zhao-2023-infusing-hierarchical-guidance-into-prompt/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aghajanyan2021intrinsic,
  title = {{Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning}},
  author = {Aghajanyan, Armen and Gupta, Sonal and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.568},
  eprint = {2012.13255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.568}
}
```

- [← previous Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://md.research.elamur.ai/free/en/papers/tarrant-2021-intrinsic-bias-metrics-do-not/)
- [next → Investigating Pretrained Language Models for Graph-to-Text Generation](https://md.research.elamur.ai/free/en/papers/ribeiro-2021-investigating-pretrained-language-models-graph/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020. Better fine-tuning by reducing representational collapse. arXiv preprint arXiv:2008.03156. — [arXiv:2008.03156](https://arxiv.org/abs/2008.03156) [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Akshat+Shrivastava%2C+Anchit+Gupta%2C+Naman+Goyal%2C+Luke+Zettlemoyer%2C+and+Sonal+Gupta%2E+2020%2E+Better+fine%2Dtuning+by+reducing+representational+collapse%2E+arXiv+preprint+arXiv%3A2008%2E03156%2E+%E2%80%94+%5B)
2. Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. 2018. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:1802.05296. — [arXiv:1802.05296](https://arxiv.org/abs/1802.05296) [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Rong+Ge%2C+Behnam+Neyshabur%2C+and+Yi+Zhang%2E+2018%2E+Stronger+generalization+bounds+for+deep+nets+via+a+compression+approach%2E+arXiv+preprint+arXiv%3A1802%2E05296%2E+%E2%80%94+%5BarXiv%3A1802%2E05296%5D%28https%3A%2F%2Farx)
3. Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained bert networks. Advances in Neural Information Processing Systems, 33. [find](https://scholar.google.com/scholar?q=Tianlong+Chen%2C+Jonathan+Frankle%2C+Shiyu+Chang%2C+Sijia+Liu%2C+Yang+Zhang%2C+Zhangyang+Wang%2C+and+Michael+Carbin%2E+2020%2E+The+lottery+ticket+hypothesis+for+pre%2Dtrained+bert+networks%2E+Advances+in+Neural+Informati)
4. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019a. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D+Manning%2E+2019a%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E+arXiv+preprint+arXiv%3A1906%2E04341%2E)
5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019b. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D+Manning%2E+2019b%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E+arXiv+preprint+arXiv%3A1906%2E04341%2E)
6. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2E)
7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Unsupervised+cross%2Dling)
8. Shrey Desai, Hongyuan Zhan, and Ahmed Aly. 2019. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12708. [find](https://scholar.google.com/scholar?q=Shrey+Desai%2C+Hongyuan+Zhan%2C+and+Ahmed+Aly%2E+2019%2E+Evaluating+lottery+tickets+under+distributional+shifts%2E+arXiv+preprint+arXiv%3A1910%2E12708%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
10. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
11. Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv:1908.05620. [find](https://scholar.google.com/scholar?q=Yaru+Hao%2C+Li+Dong%2C+Furu+Wei%2C+and+Ke+Xu%2E+2019%2E+Visualizing+and+understanding+the+effectiveness+of+bert%2E+arXiv+preprint+arXiv%3A1908%2E05620%2E)
12. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
13. Geoffrey E Hinton and Richard Zemel. 1993. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6:3–10. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton+and+Richard+Zemel%2E+1993%2E+Autoencoders%2C+minimum+description+length+and+helmholtz+free+energy%2E+Advances+in+neural+information+processing+systems%2C+6%3A3%E2%80%9310%2E)
14. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+arX)
15. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. 2017. First quora dataset release: Question pairs. [find](https://scholar.google.com/scholar?q=Shankar+Iyer%2C+Nikhil+Dandekar%2C+and+Kornel+Csernai%2E+2017%2E+First+quora+dataset+release%3A+Question+pairs%2E)
16. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2019%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1)
17. Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804.08838. [find](https://scholar.google.com/scholar?q=Chunyuan+Li%2C+Heerad+Farkhoor%2C+Rosanne+Liu%2C+and+Jason+Yosinski%2E+2018%2E+Measuring+the+intrinsic+dimension+of+objective+landscapes%2E+arXiv+preprint+arXiv%3A1804%2E08838%2E)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
19. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2019%2E+Adversarial+nli%3A+A+new+benchmark+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1910%2E14599%2E)
20. Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00561. [find](https://scholar.google.com/scholar?q=Sai+Prasanna%2C+Anna+Rogers%2C+and+Anna+Rumshisky%2E+2020%2E+When+bert+plays+the+lottery%2C+all+tickets+are+winning%2E+arXiv+preprint+arXiv%3A2005%2E00561%2E)
21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
25. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Marjan+Ghazvininejad%2C+Gargi+Ghosh%2C+Armen+Aghajanyan%2C+Sida+Wang%2C+and+Luke+Zettlemoyer%2E+2020%2E+Pre%2Dtraining+via+paraphrasing%2E) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/)
26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
27. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2019%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+langu) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, pages arXiv–1910. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtow%2Dicz%2C+et+al%2E+2019%2E+Huggingface%E2%80%99s+transformers%3A+State%2Dof%2Dt)
29. Quoc Le, Tamás Sarlós, and Alex Smola. 2013. Fastfood-approximating kernel expansions in loglin-ear time. In Proceedings of the international conference on machine learning, volume 85. [find](https://scholar.google.com/scholar?q=Quoc+Le%2C+Tama%CC%81s+Sarlo%CC%81s%2C+and+Alex+Smola%2E+2013%2E+Fastfood%2Dapproximating+kernel+expansions+in+loglin%2Dear+time%2E+In+Proceedings+of+the+international+conference+on+machine+learning%2C+volume+85%2E)
30. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753–5763. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+neural+inform) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
31. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. arXiv:1509.01626 [cs]. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+Convolutional+Networks+for+Text+Classification%2E+arXiv%3A1509%2E01626+%5Bcs%5D%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)