---
title: "It's Not Just Size That Matters — Schick 2021"
description: "The paper asks whether strong few-shot performance requires a giant model like GPT-3 (175 billion parameters), which learns from in-context…"
authors: ["Timo Schick", "Hinrich Schütze"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.185/"
doi: "10.18653/v1/2021.naacl-main.185"
arxiv: "2009.07118"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

# It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

**Authors:** [Timo Schick](https://md.research.elamur.ai/free/en/authors/timo-schick/), [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) · **Year:** 2021 · **Venue:** Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.185/) · [PDF](https://aclanthology.org/2021.naacl-main.185.pdf) · [DOI 10.18653/v1/2021.naacl-main.185](https://doi.org/10.18653/v1/2021.naacl-main.185) · [arXiv 2009.07118](https://arxiv.org/abs/2009.07118) · [OpenAlex](https://openalex.org/W3085177480)

**Citations:** 685 citations (openalex-snapshot · 2026-10-14) · [433](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether strong few-shot performance requires a giant model like GPT-3 (175 billion parameters), which learns from in-context demonstrations without any weight updates. The authors build on pattern-exploiting training (PET). PET rewrites each input as a cloze question that includes a task description and fine-tunes a masked language model on it with gradients. It can also use unlabeled data through distillation, and an iterative variant (iPET) adds self-training.

The original PET needed every label to map to a single vocabulary token. This work removes that restriction so labels can be multi-token. At inference, the model fills in masks one at a time, always choosing its most confident position first.

The method is tested on SuperGLUE using only 32 labeled examples per task, released as the FewGLUE subset. With ALBERT, both PET and iPET beat GPT-3 on average while using about 0.1% of its parameters. Training fits on a single GPU and needs no costly hyperparameter search. The authors argue this makes few-shot NLP cheaper and more environmentally friendly.

## Key points
- ALBERT combined with PET/iPET outperforms GPT-3 on SuperGLUE with 32 examples, using three orders of magnitude fewer parameters. Code, models and the FewGLUE data are released at github.com/timoschick/pet.
- Multi-mask decoding ("max-first") reaches an average of 87.1 on COPA, WSC and ReCoRD, compared with 72.5 for untrained ALBERT. It beats parallel decoding and is slightly better than left-to-right decoding.
- Swapping ALBERT for RoBERTa lowers the average by about 8 points, though RoBERTa still beats GPT-3 13B. GPT-2 does much worse, which the authors link partly to its unidirectionality.
- The analysis attributes PET's strength to four factors:
  - combining several patterns,
  - robustness to patterns that are hard to understand,
  - gradient updates on labeled data,
  - the choice of underlying LM.
- Using different random 32-example training sets (Σ0–Σ2) shifts results substantially; for example, CB F1 ranges from 57.4 to 84.0. Few-shot methods should therefore be compared on identical examples.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

433 papers of our library cite it (of the 140884 whose bibliographies we read); 49 of them have a page here, 384 not yet.

- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://md.research.elamur.ai/free/en/papers/chen-2023-empirical-survey-data-augmentation-limited/)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
- [Making Pre-trained Language Models Better Few-shot Learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://md.research.elamur.ai/free/en/papers/hu-2022-knowledgeable-prompt-tuning-incorporating-knowledge/)
- [Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/)
- [Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/jiang-2023-global-local-hierarchy-aware-contrastive/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
- [Pseudo Zero Pronoun Resolution Improves Zero Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/konno-2021-pseudo-zero-pronoun-resolution-improves/)
- [Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts](https://md.research.elamur.ai/free/en/papers/le-2022-few-shot-anaphora-resolution-scientific/)
- [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2021-p-tuning-v2-prompt-tuning/)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks](https://md.research.elamur.ai/free/en/papers/liu-2022-p-tuning-prompt-tuning-can/)
- [Prompt Tuning with Contradictory Intentions for Sarcasm Recognition](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-tuning-contradictory-intentions-sarcasm/)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models](https://md.research.elamur.ai/free/en/papers/prasad-2023-grips-gradient-free-edit-based/)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/reif-2021-recipe-arbitrary-text-style-transfer/)
- [How many data points is a prompt worth?](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
- [Generating Datasets with Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [Do Prompt-Based Models Really Understand the Meaning of Their Prompts?](https://md.research.elamur.ai/free/en/papers/webson-2022-do-prompt-based-models-really/)
- [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
- [Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/wu-2023-connective-prediction-implicit-discourse-relation/)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://md.research.elamur.ai/free/en/papers/yoo-2021-gpt3mix-leveraging-large-scale-language/)
- [BARTScore: Evaluating Generated Text as Text Generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)
- [Factual Probing Is MASK: Learning vs. Learning to Recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/)
- [Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/zhou-2022-prompt-based-connective-prediction-method/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schick2021it,
  title = {{It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners}},
  author = {Schick, Timo and Schütze, Hinrich},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.185},
  eprint = {2009.07118},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.185}
}
```

- [← previous Issayas Tesfamariam, ትግርኛ ንዛረብ: Let’s Speak Tigrinya, Elementary Level, A First-Year Textbook, A Multidimensional Approach to the Teaching and Learning of Tigrinya As a Foreign Language](https://md.research.elamur.ai/free/en/papers/voigt-2021-issayas-tesfamariam-lets-speak-tigrinya/)
- [next → JABER and SABER: Junior and Senior Arabic BERt](https://md.research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

60 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mark Anderson and Carlos Gómez-Rodríguez. 2020. Distilling neural networks for greener and faster dependency parsing. In Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies, pages 2–13, Online. Associa… [find](https://scholar.google.com/scholar?q=Mark+Anderson+and+Carlos+G%C3%B3mez%2DRodr%C3%ADguez%2E+2020%2E+Distilling+neural+networks+for+greener+and+faster+dependency+parsing%2E+In+Proceedings+of+the+16th+International+Conference+on+Parsing+Technologies+and+th) [our card](https://md.research.elamur.ai/free/en/papers/anderson-2020-distilling-neural-networks-greener-faster/)
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. Computing Research Repository, arXiv:2004.05150. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+Computing+Research+Repository%2C+arXiv%3A2004%2E05150%2E)
3. Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In The World Wide Web and Databases, pages 172–183, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Sergey+Brin%2E+1999%2E+Extracting+patterns+and+relations+from+the+world+wide+web%2E+In+The+World+Wide+Web+and+Databases%2C+pages+172%E2%80%93183%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+Heidelberg%2E)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
5. Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. 2019. An embarrassingly simple approach for transfer learning from pre-trained language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T… [find](https://scholar.google.com/scholar?q=Alexandra+Chronopoulou%2C+Christos+Baziotis%2C+and+Alexandros+Potamianos%2E+2019%2E+An+embarrassingly+simple+approach+for+transfer+learning+from+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2019+Confere)
6. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classificati)
8. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre-trained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–… [find](https://scholar.google.com/scholar?q=Joe+Davison%2C+Joshua+Feldman%2C+and+Alexander+Rush%2E+2019%2E+Commonsense+knowledge+mining+from+pre%2Dtrained+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+a)
9. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The CommitmentBank: Investigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung 23. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe%2C+Mandy+Simons%2C+and+Judith+Tonhauser%2E+2019%2E+The+CommitmentBank%3A+Investigating+projection+in+naturally+occurring+discourse%2E+In+Proceedings+of+Sinn+und+Bedeutung+23%2E) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
11. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. Computing Research Repository, arXiv:2002.06305. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+Smith%2E+2020%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+Com)
12. Ürün Dogan, Tobias Glasmachers, and Christian Igel. 2016. A unified view on multi-class support vector classification. J. Mach. Learn. Res., 17(45):1–32. [find](https://scholar.google.com/scholar?q=%C3%9Cr%C3%BCn+Dogan%2C+Tobias+Glasmachers%2C+and+Christian+Igel%2E+2016%2E+A+unified+view+on+multi%2Dclass+support+vector+classification%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+17%2845%29%3A1%E2%80%9332%2E)
13. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2E+2020%2E+What+BERT+is+not%3A+Lessons+from+a+new+suite+of+psycholinguistic+diagnostics+for+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A34%E2%80%9348%2E) [our card](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/)
14. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Omer+Levy%2C+Yinhan+Liu%2C+and+Luke+Zettlemoyer%2E+2019%2E+Mask%2Dpredict%3A+Parallel+decoding+of+conditional+masked+language+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Metho) [our card](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/)
15. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. Computing Research Repository, arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) [find](https://scholar.google.com/scholar?q=Yunchao+Gong%2C+Liu+Liu%2C+Ming+Yang%2C+and+Lubomir+Bourdev%2E+2014%2E+Compressing+deep+convolutional+networks+using+vector+quantization%2E+Computing+Research+Repository%2C+arXiv%3A1412%2E6115%2E+%E2%80%94+%5BarXiv%3A1412%2E6115%5D%28http)
16. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s… [find](https://scholar.google.com/scholar?q=Andrew+Gordon%2C+Zornitsa+Kozareva%2C+and+Melissa+Roemmele%2E+2012%2E+SemEval%2D2012+task+7%3A+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+%2ASEM+2012%3A+The+First+Joint+Confer) [our card](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/)
17. Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J+Dally%2E+2016%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+International+Conference+on+Learning+Represent)
18. Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+2015%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+28%2E+Curran+Ass)
19. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Computing Research Repository, arXiv:1503.02531. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+Computing+Research+Repository%2C+arXiv%3A1503%2E02531%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
20. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F%2E+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020%2E+How+can+we+know+what+language+models+know%3F+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A423%E2%80%93438%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
21. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174, Online. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+2020%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+In+Findings+of+the+Association+for+C)
22. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+2018%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
23. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nikita+Kitaev%2C+Lukasz+Kaiser%2C+and+Anselm+Levskaya%2E+2020%2E+Reformer%3A+The+efficient+transformer%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/)
24. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe)
25. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+schema+challenge%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2C+volume+46%2C+page+47%2E)
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
27. Weijie Liu, Peng Zhou, Zhiruo Wang, Zhe Zhao, Haotang Deng, and Qi Ju. 2020. FastBERT: a self-distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6035– 6044, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Weijie+Liu%2C+Peng+Zhou%2C+Zhiruo+Wang%2C+Zhe+Zhao%2C+Haotang+Deng%2C+and+Qi+Ju%2E+2020%2E+FastBERT%3A+a+self%2Ddistilling+BERT+with+adaptive+inference+time%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/)
28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pre-training approach. Computing Research Repository, arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pre%2Dtraining+approach%2E+)
29. Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. 2020. LadaBERT: Lightweight adaptation of BERT through hybrid model compression. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3225– 3234, Barcel… [find](https://scholar.google.com/scholar?q=Yihuan+Mao%2C+Yujing+Wang%2C+Chufan+Wu%2C+Chen+Zhang%2C+Yang+Wang%2C+Quanlu+Zhang%2C+Yaming+Yang%2C+Yunhai+Tong%2C+and+Jing+Bai%2E+2020%2E+LadaBERT%3A+Lightweight+adaptation+of+BERT+through+hybrid+model+compression%2E+In+Pro)
30. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152– 159, New York City, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+McClosky%2C+Eugene+Charniak%2C+and+Mark+Johnson%2E+2006%2E+Effective+self%2Dtraining+for+parsing%2E+In+Proceedings+of+the+Human+Language+Technology+Conference+of+the+NAACL%2C+Main+Conference%2C+pages+152%E2%80%93+159%2C+) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-mcclosky-effective-self-training-parsing/)
31. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 193– 202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2E+2019%2E+Argumentative+relation+classification+as+plausibility+ranking%2E+In+Preliminary+proceedings+of+the+15th+Conference+on+Natural+Language+Processing+%28KONVENS+2019%29%3A+Long+Papers%2C+pages+193)
32. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+PyTorch%2E+In+NIPS+Aut)
33. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+Proceedings+of+the+2019+Conference+on+Empiri) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
34. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar+and+Jose+Camacho%2DCollados%2E+2019%2E+WiC%3A+the+word%2Din%2Dcontext+dataset+for+evaluating+context%2Dsensitive+meaning+representations%2E+In+Proceedings+of+the+2019+Conference+of+the+North+A) [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
35. Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. Computing Research Repository, arXiv:1912.10165. — [arXiv:1912.10165](https://arxiv.org/abs/1912.10165) [find](https://scholar.google.com/scholar?q=Raul+Puri+and+Bryan+Catanzaro%2E+2019%2E+Zero%2Dshot+text+classification+with+generative+language+models%2E+Computing+Research+Repository%2C+arXiv%3A1912%2E10165%2E+%E2%80%94+%5BarXiv%3A1912%2E10165%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E101)
36. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
37. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
38. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
39. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julian+Salazar%2C+Davis+Liang%2C+Toan+Q%2E+Nguyen%2C+and+Katrin+Kirchhoff%2E+2020%2E+Masked+language+model+scoring%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page) [our card](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
40. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+In+Proceedings+of+the+5th+Workshop+on+Energy+Efficie)
41. Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In Advances in Neural Information Processing Systems, volume 33, pages 20378–20389. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Thomas+Wolf%2C+and+Alexander+Rush%2E+2020%2E+Movement+pruning%3A+Adaptive+sparsity+by+fine%2Dtuning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+33%2C+pages+20378%E2%80%9320389%2E+Curran+Assoc)
42. Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+Rare+words%3A+A+major+problem+for+contextualized+embeddings+and+how+to+fix+it+by+attentive+mimicking%2E+In+Proceedings+of+the+Thirty%2DFourth+AAAI+Conference+on+Artifi)
43. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze questions for few shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Kyiv, Ukraine (Online). International Committee on Comput… [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2021%2E+Exploiting+cloze+questions+for+few+shot+text+classification+and+natural+language+inference%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
44. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020a. Green AI. Commun. ACM, 63(12):54–63. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Jesse+Dodge%2C+Noah+A%2E+Smith%2C+and+Oren+Etzioni%2E+2020a%2E+Green+AI%2E+Commun%2E+ACM%2C+63%2812%29%3A54%E2%80%9363%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
45. Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020b. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640–6651, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Gabriel+Stanovsky%2C+Swabha+Swayamdipta%2C+Jesse+Dodge%2C+and+Noah+A%2E+Smith%2E+2020b%2E+The+right+tool+for+the+job%3A+Matching+model+and+instance+complexities%2E+In+Proceedings+of+the+58th+Annual+Meet)
46. H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371. [find](https://scholar.google.com/scholar?q=H+Scudder%2E+1965%2E+Probability+of+error+of+some+adaptive+pattern%2Drecognition+machines%2E+IEEE+Transactions+on+Information+Theory%2C+11%283%29%3A363%E2%80%93371%2E)
47. Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Pierre+Stock%2C+Angela+Fan%2C+Benjamin+Graham%2C+Edouard+Grave%2C+R%C3%A9mi+Gribonval%2C+Herve+Jegou%2C+and+Armand+Joulin%2E+2021%2E+Training+with+quantization+noise+for+extreme+model+compression%2E+In+International+Confere)
48. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emma+Strubell%2C+Ananya+Ganesh%2C+and+Andrew+McCallum%2E+2019%2E+Energy+and+policy+considerations+for+deep+learning+in+NLP%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Lingu) [our card](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/)
49. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics – on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743–758. [find](https://scholar.google.com/scholar?q=Alon+Talmor%2C+Yanai+Elazar%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2020%2E+oLMpics+%E2%80%93+on+what+language+model+pre%2Dtraining+captures%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A743%E2%80%93758%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
50. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. Computing Research Repository, arXiv:1806.02847. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+2018%2E+A+simple+method+for+commonsense+reasoning%2E+Computing+Research+Repository%2C+arXiv%3A1806%2E02847%2E)
51. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
52. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang+and+Kyunghyun+Cho%2E+2019%2E+BERT+has+a+mouth%2C+and+it+must+speak%3A+BERT+as+a+Markov+random+field+language+model%2E+In+Proceedings+of+the+Workshop+on+Methods+for+Optimizing+and+Evaluating+Neural+Lan) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)
53. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2019%2E+SuperGLUE%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understandin) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
54. Jason Weston and Chris Watkins. 1999. Support vector machines for multi-class pattern recognition. In ESANN, volume 99, pages 219–224. [find](https://scholar.google.com/scholar?q=Jason+Weston+and+Chris+Watkins%2E+1999%2E+Support+vector+machines+for+multi%2Dclass+pattern+recognition%2E+In+ESANN%2C+volume+99%2C+pages+219%E2%80%93224%2E)
55. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+Remi+Louf%2C+Morgan+Funtow%2Dicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
56. Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. 2020. Early exiting BERT for efficient document ranking. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 83–88, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Rodrigo+Nogueira%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2020%2E+Early+exiting+BERT+for+efficient+document+ranking%2E+In+Proceedings+of+SustaiNLP%3A+Workshop+on+Simple+and+Efficient+Natural+Language+Processing)
57. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Pr… [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+H%2E+Wallach%2C+H%2E+Larochelle) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
58. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189–196, Cambridge, Massachusetts, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Yarowsky%2E+1995%2E+Unsupervised+word+sense+disambiguation+rivaling+supervised+methods%2E+In+33rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+189%E2%80%93196%2C+Cambridge%2C+Massachuset) [our card](https://md.research.elamur.ai/free/en/papers/yarowsky-1995-unsupervised-word-sense-disambiguation-rivaling/)
59. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. In NeurIPS EMC2 Workshop. [find](https://scholar.google.com/scholar?q=Ofir+Zafrir%2C+Guy+Boudoukh%2C+Peter+Izsak%2C+and+Moshe+Wasserblat%2E+2019%2E+Q8BERT%3A+Quantized+8bit+BERT%2E+In+NeurIPS+EMC2+Workshop%2E)
60. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. Computing Research Repository, arXiv:1810.12885. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2018%2E+ReCoRD%3A+Bridging+the+gap+between+human+and+machine+commonsense+reading+comprehension%2E+Computing+Research) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)