---
title: "Language Models are Few-Shot Learners — Brown 2020"
description: "The standard approach in NLP has a significant drawback: after pre-training, a model is usually fine-tuned on thousands of labeled examples for each…"
authors: ["Tom B. Brown", "Benjamin Mann", "Nick Ryder", "Melanie Subbiah", "Jared Kaplan", "Prafulla Dhariwal", "Arvind Neelakantan", "Pranav Shyam", "Girish Sastry", "Amanda Askell", "Sandhini Agarwal", "Ariel Herbert-Voss", "Gretchen Krueger", "Tom Henighan", "Rewon Child", "Aditya Ramesh", "Daniel M. Ziegler", "Jeffrey Wu", "Clemens Winter", "Christopher Hesse", "Mark Chen", "Eric Sigler", "Mateusz Litwin", "Scott Gray", "Benjamin Chess", "Jack Clark", "Christopher Berner", "Sam McCandlish", "Alec Radford", "Ilya Sutskever", "Dario Amodei"]
year: 2020
venue: "arXiv:2005.14165"
source: "https://arxiv.org/abs/2005.14165"
doi: "10.48550/arxiv.2005.14165"
arxiv: "2005.14165"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / Language Models are Few-Shot Learners

# Language Models are Few-Shot Learners

**Authors:** [Tom B. Brown](https://md.research.elamur.ai/free/en/authors/tom-b-brown/), [Benjamin Mann](https://md.research.elamur.ai/free/en/authors/benjamin-mann/), Nick Ryder, Melanie Subbiah, [Jared Kaplan](https://md.research.elamur.ai/free/en/authors/jared-kaplan/), [Prafulla Dhariwal](https://md.research.elamur.ai/free/en/authors/prafulla-dhariwal/), [Arvind Neelakantan](https://md.research.elamur.ai/free/en/authors/arvind-neelakantan/), [Pranav Shyam](https://md.research.elamur.ai/free/en/authors/pranav-shyam/), [Girish Sastry](https://md.research.elamur.ai/free/en/authors/girish-sastry/), [Amanda Askell](https://md.research.elamur.ai/free/en/authors/amanda-askell/), [Sandhini Agarwal](https://md.research.elamur.ai/free/en/authors/sandhini-agarwal/), [Ariel Herbert-Voss](https://md.research.elamur.ai/free/en/authors/ariel-herbert-voss/) et al. (31 authors) · **Year:** 2020 · **Venue:** arXiv:2005.14165 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.14165) · [PDF](https://arxiv.org/pdf/2005.14165) · [DOI 10.48550/arxiv.2005.14165](https://doi.org/10.48550/arxiv.2005.14165) · [arXiv 2005.14165](https://arxiv.org/abs/2005.14165)

**Citations:** [10782](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary

The standard approach in NLP has a significant drawback: after pre-training, a model is usually fine-tuned on thousands of labeled examples for each individual task. Humans, by contrast, often need only a brief instruction or a few examples to perform a new language task. The authors from OpenAI examine whether simply scaling up a language model gives it a similar ability. They trained GPT-3, an autoregressive transformer with 175 billion parameters, on a large filtered web corpus. The model was then evaluated on more than two dozen datasets without any weight updates: the task and examples were provided simply in the text of the prompt. The authors call this mechanism in-context learning. The model was tested in three settings: zero-shot, one-shot, and few-shot. In the few-shot setting, GPT-3 sometimes reaches the level of the best fine-tuned systems, and as the model grows, the benefit from in-context examples increases faster. The paper is important because it shifts the emphasis from fine-tuning to using a single general-purpose model through prompting. At the same time, the authors candidly describe the model's weaknesses, the risk of test data "contamination," and societal risks.

## Key points

- GPT-3 has 175 billion parameters. This is roughly ten times more than any previous dense (non-sparse) language model. For comparison, GPT-2 had 1.5 billion and Turing-NLG had 17 billion.
- In the few-shot setting, the model receives as many examples as fit in the context window (typically from 10 to 100). In zero-shot, it sees only a natural-language instruction. There are no gradient updates in any setting.
- Strong results: translation, closed-book QA, cloze tasks, as well as tasks requiring on-the-fly reasoning at query time. These include unscrambling anagrams, using a novel word, and three-digit arithmetic.
- Across an aggregate of 42 benchmarks where the metric is accuracy, few-shot performance grows with model size noticeably faster than zero-shot performance. The model showed weak results on NLI (in particular ANLI) and on some reading comprehension tasks.
- Human evaluators had difficulty distinguishing news articles generated by GPT-3 from those written by humans. The authors separately discuss potential misuse, bias, and energy consumption.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{brown2020language,
  title = {{Language Models are Few-Shot Learners}},
  author = {Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},
  year = {2020},
  doi = {10.48550/arxiv.2005.14165},
  eprint = {2005.14165},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.14165}
}
```

- [← previous BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
- [next → A Simple Framework for Contrastive Learning of Visual Representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

144 references, 59 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pages 3981–3989, 2016. [find](https://scholar.google.com/scholar?q=Marcin+Andrychowicz%2C+Misha+Denil%2C+Sergio+Gomez%2C+Matthew+W+Hoffman%2C+David+Pfau%2C+Tom+Schaul%2C+Brendan+Shillingford%2C+and+Nando+De+Freitas%2E+Learning+to+learn+by+gradient+descent+by+gradient+descent%2E+In+Adv)
2. WeChat AI. Tr-mt (ensemble), December 2019. [find](https://scholar.google.com/scholar?q=WeChat+AI%2E+Tr%2Dmt+%28ensemble%29%2C+December+2019%2E)
3. Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Melvin+Johnson%2C+and+Orhan+Firat%2E+Massively+multilingual+neural+machine+translation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation) [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050, 2020. [find](https://scholar.google.com/scholar?q=Su+Lin+Blodgett%2C+Solon+Barocas%2C+Hal+Daume%CC%81+III%2C+and+Hanna+Wallach%2E+Language+%28technology%29+is+power%3A+A+critical+survey+of+%E2%80%9Cbias%E2%80%9D+in+nlp%2E+arXiv+preprint+arXiv%3A2005%2E14050%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
5. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013. [find](https://scholar.google.com/scholar?q=Jonathan+Berant%2C+Andrew+Chou%2C+Roy+Frostig%2C+and+Percy+Liang%2E+Semantic+parsing+on+freebase+from+question%2Danswer+pairs%2E+In+Proceedings+of+the+2013+conference+on+empirical+methods+in+natural+language+proc) [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
6. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+2009%2E)
7. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In Lrec, volume 10, pages 2200–2204, 2010. [find](https://scholar.google.com/scholar?q=Stefano+Baccianella%2C+Andrea+Esuli%2C+and+Fabrizio+Sebastiani%2E+Sentiwordnet+3%2E0%3A+an+enhanced+lexical+resource+for+sentiment+analysis+and+opinion+mining%2E+In+Lrec%2C+volume+10%2C+pages+2200%E2%80%932204%2C+2010%2E) [our card](https://md.research.elamur.ai/free/en/papers/baccianella-2010-sentiwordnet-3-0-enhanced-lexical/)
8. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. 2006. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E+2006%2E)
9. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language. arXiv preprint arXiv:2004.10151, 2020. [find](https://scholar.google.com/scholar?q=Yonatan+Bisk%2C+Ari+Holtzman%2C+Jesse+Thomason%2C+Jacob+Andreas%2C+Yoshua+Bengio%2C+Joyce+Chai%2C+Mirella+Lapata%2C+Angeliki+Lazaridou%2C+Jonathan+May%2C+Aleksandr+Nisnevich%2C+et+al%2E+Experience+grounds+language%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/bisk-2020-experience-grounds-language/)
10. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. Arxiv, 2013. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+Le%CC%81onard%2C+and+Aaron+C%2E+Courville%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+Arxiv%2C+2013%2E)
11. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv:1911.11641, 2019. [find](https://scholar.google.com/scholar?q=Yonatan+Bisk%2C+Rowan+Zellers%2C+Ronan+Le+Bras%2C+Jianfeng+Gao%2C+and+Yejin+Choi%2E+Piqa%3A+Reasoning+about+physical+commonsense+in+natural+language%2E+arXiv+preprint+arXiv%3A1911%2E11641%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
12. Rich Caruana. Multitask learning. Machine learning, 28(1), 1997. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+Multitask+learning%2E+Machine+learning%2C+28%281%29%2C+1997%2E)
13. Susan Carey and Elsa Bartlett. Acquiring a single new word. Proceedings of the Stanford Child Language Conference, 1978. [find](https://scholar.google.com/scholar?q=Susan+Carey+and+Elsa+Bartlett%2E+Acquiring+a+single+new+word%2E+Proceedings+of+the+Stanford+Child+Language+Conference%2C+1978%2E)
14. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+ArXiv%2C+ab)
15. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+Generating+long+sequences+with+sparse+transformers%2C+2019%2E)
16. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac : Question answering in context. Arxiv, 2018. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+He+He%2C+Mohit+Iyyer%2C+Mark+Yatskar%2C+Wen%2Dtau+Yih%2C+Yejin+Choi%2C+Percy+Liang%2C+and+Luke+Zettlemoyer%2E+Quac+%3A+Question+answering+in+context%2E+Arxiv%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
17. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+arXiv+preprint+arXiv%3A19) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
18. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740, 2019. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen%2C+Linjie+Li%2C+Licheng+Yu%2C+Ahmed+El+Kholy%2C+Faisal+Ahmed%2C+Zhe+Gan%2C+Yu+Cheng%2C+and+Jingjing+Liu%2E+Uniter%3A+Learning+universal+image%2Dtext+representations%2E+arXiv+preprint+arXiv%3A1909%2E11740%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2019-uniter-universal-image-text-representation/)
19. Kate Crawford. The trouble with bias. NIPS 2017 Keynote, 2017. [find](https://scholar.google.com/scholar?q=Kate+Crawford%2E+The+trouble+with+bias%2E+NIPS+2017+Keynote%2C+2017%2E)
20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
21. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177–190. Springer, 2006. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classification%2C+an)
22. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. Arxiv, 2018. [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Stephan+Gouws%2C+Oriol+Vinyals%2C+Jakob+Uszkoreit%2C+and+Lukasz+Kaiser%2E+Universal+transformers%2E+Arxiv%2C+2018%2E)
23. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh’s phrase-based machine translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 97–104, 2014. [find](https://scholar.google.com/scholar?q=Nadir+Durrani%2C+Barry+Haddow%2C+Philipp+Koehn%2C+and+Kenneth+Heafield%2E+Edinburgh%E2%80%99s+phrase%2Dbased+machine+translation+systems+for+wmt%2D14%2E+In+Proceedings+of+the+Ninth+Workshop+on+Statistical+Machine+Translati)
24. Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In Advances in neural information processing systems, 2015. [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai+and+Quoc+V%2E+Le%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+2015%2E)
25. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung 23. Data can be found at [https://github.com/mcdm/CommitmentBank/](https://github.com/mcdm/CommitmentBank/). [↗](https://github.com/mcdm/CommitmentBank/) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
26. Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. ArXiv, abs/1611.02779, 2016. [find](https://scholar.google.com/scholar?q=Yan+Duan%2C+John+Schulman%2C+Xi+Chen%2C+Peter+L%2E+Bartlett%2C+Ilya+Sutskever%2C+and+Pieter+Abbeel%2E+Rl+2+%3A+Fast+reinforcement+learning+via+slow+reinforcement+learning%2E+ArXiv%2C+abs%2F1611%2E02779%2C+2016%2E)
27. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+Drop%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+arXiv+preprint+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
28. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. Arxiv, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G%2E+Carbonell%2C+Quoc+V%2E+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+Arxiv%2C+2019%2E)
29. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381, 2018. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+Understanding+back%2Dtranslation+at+scale%2E+arXiv+preprint+arXiv%3A1808%2E09381%2C+2018%2E)
30. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ArXiv, abs/1703.03400, 2017. [find](https://scholar.google.com/scholar?q=Chelsea+Finn%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+ArXiv%2C+abs%2F1703%2E03400%2C+2017%2E)
31. Yaroslav Fyodorov. A natural logic inference system, 2000. [find](https://scholar.google.com/scholar?q=Yaroslav+Fyodorov%2E+A+natural+logic+inference+system%2C+2000%2E)
32. Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862, 2019. [find](https://scholar.google.com/scholar?q=Hila+Gonen+and+Yoav+Goldberg%2E+Lipstick+on+a+pig%3A+Debiasing+methods+cover+up+systematic+gender+biases+in+word+embeddings+but+do+not+remove+them%2E+arXiv+preprint+arXiv%3A1903%2E03862%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/gonen-2019-lipstick-pig-debiasing-methods-cover/)
33. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+Realm%3A+Retrieval%2Daugmented+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2002%2E08909%2C+2020%2E)
34. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics, 2007. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+The+third+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+ACL%2DPASCAL+workshop+on+textual+entailment+and+paraphra)
35. Alex Graves. Adaptive computation time for recurrent neural networks. Arxiv, 2016. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+Adaptive+computation+time+for+recurrent+neural+networks%2E+Arxiv%2C+2016%2E)
36. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324, 2018. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+R+Bowman%2C+and+Noah+A+Smith%2E+Annotation+artifacts+in+natural+language+inference+data%2E+arXiv+preprint+arXiv%3A1803%2E02324%2C+2018%2E)
37. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 1906.04043, 2019. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Hendrik+Strobelt%2C+and+Alexander+M%2E+Rush%2E+Gltr%3A+Statistical+detection+and+visualization+of+generated+text%2E+arXiv+preprint+arXiv%3A+1906%2E04043%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/)
38. Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. Meta-learning for low-resource neural machine translation. arXiv preprint arXiv:1808.08437, 2018. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Yong+Wang%2C+Yun+Chen%2C+Kyunghyun+Cho%2C+and+Victor+OK+Li%2E+Meta%2Dlearning+for+low%2Dresource+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1808%2E08437%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/)
39. Daniel Hernandez and Tom Brown. Ai and efficiency, May 2020. [find](https://scholar.google.com/scholar?q=Daniel+Hernandez+and+Tom+Brown%2E+Ai+and+efficiency%2C+May+2020%2E)
40. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. CoRR, abs/1904.09751, 2019. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+CoRR%2C+abs%2F1904%2E09751%2C+2019%2E)
41. Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out of distribution robustness. arXiv preprint arXiv:2004.06100, 2020. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Xiaoyuan+Liu%2C+Eric+Wallace%2C+Adam+Dziedzic%2C+Rishabh+Krishnan%2C+and+Dawn+Song%2E+Pretrained+transformers+improve+out+of+distribution+robustness%2E+arXiv+preprint+arXiv%3A2004%2E06100%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/)
42. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. [find](https://scholar.google.com/scholar?q=Joel+Hestness%2C+Sharan+Narang%2C+Newsha+Ardalani%2C+Gregory+Diamos%2C+Heewoo+Jun%2C+Hassan+Kianinejad%2C+Md%2E+Mostofa+Ali+Patwary%2C+Yang+Yang%2C+and+Yanqi+Zhou%2E+Deep+learning+scaling+is+predictable%2C+empirically%2E+arX) [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
43. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2C+2018%2E)
44. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
45. Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent. In International Conference on Artificial Neural Networks, pages 87–94. Springer, 2001. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter%2C+A+Steven+Younger%2C+and+Peter+R+Conwell%2E+Learning+to+Learn+Using+Gradient+Descent%2E+In+International+Conference+on+Artificial+Neural+Networks%2C+pages+87%E2%80%9394%2E+Springer%2C+2001%2E)
46. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064, 2019. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Huan+Zhang%2C+Ray+Jiang%2C+Robert+Stanforth%2C+Johannes+Welbl%2C+Jack+Rae%2C+Vishal+Maini%2C+Dani+Yogatama%2C+and+Pushmeet+Kohli%2E+Reducing+sentiment+bias+in+language+models+via+counterfactual+evaluati)
47. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. A neural network for factoid question answering over paragraphs. In Empirical Methods in Natural Language Processing, 2014. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+Jordan+Boyd%2DGraber%2C+Leonardo+Claudino%2C+Richard+Socher%2C+and+Hal+Daume%CC%81+III%2E+A+neural+network+for+factoid+question+answering+over+paragraphs%2E+In+Empirical+Methods+in+Natural+Language+Proces)
48. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019. [find](https://scholar.google.com/scholar?q=Daphne+Ippolito%2C+Daniel+Duckworth%2C+Chris+Callison%2DBurch%2C+and+Douglas+Eck%2E+Automatic+detection+of+generated+text+is+easiest+when+humans+are+fooled%2E+arXiv+preprint+arXiv%3A1911%2E00650%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/ippolito-2019-automatic-detection-generated-text-easiest/)
49. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S%2E+Weld%2C+and+Luke+Zettlemoyer%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E03551%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
50. Zheng Junyuan and Gamma Lab NYC. Numeric transformer - albert, March 2020. [find](https://scholar.google.com/scholar?q=Zheng+Junyuan+and+Gamma+Lab+NYC%2E+Numeric+transformer+%2D+albert%2C+March+2020%2E)
51. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
52. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351, 2019. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1909%2E10351%2C+2019%2E)
53. Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on conversational question answering. arXiv preprint arXiv:1909.10772, 2019. — [arXiv:1909.10772](https://arxiv.org/abs/1909.10772) [find](https://scholar.google.com/scholar?q=Ying+Ju%2C+Fubang+Zhao%2C+Shijie+Chen%2C+Bowen+Zheng%2C+Xuefeng+Yang%2C+and+Yunfeng+Liu%2E+Technical+report+on+conversational+question+answering%2E+arXiv+preprint+arXiv%3A1909%2E10772%2C+2019%2E+%E2%80%94+%5BarXiv%3A1909%2E10772%5D%28https%3A)
54. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+North+) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
55. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700, 2020. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Hannaneh+Hajishirzi%2E+Unifiedqa%3A+Crossing+format+boundaries+with+a+single+qa+system%2E+arXiv+preprint+arXiv%3A2005%2E00700%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
56. Sarah E. Kreps, Miles McCain, and Miles Brundage. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation, 2020. [find](https://scholar.google.com/scholar?q=Sarah+E%2E+Kreps%2C+Miles+McCain%2C+and+Miles+Brundage%2E+All+the+news+that%E2%80%99s+fit+to+fabricate%3A+Ai%2Dgenerated+text+as+a+tool+of+media+misinformation%2C+2020%2E)
57. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+Scaling+laws+for+neural+language+models%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
58. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Matthew+Kelcey%2C+Jacob+Devlin%2C+Kenton+Lee%2C+Kristina+N%2E+Toutanova) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
59. Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. Arxiv, 2016. [find](https://scholar.google.com/scholar?q=Yoon+Kim+and+Alexander+M%2E+Rush%2E+Sequence%2Dlevel+knowledge+distillation%2E+Arxiv%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
60. Edward Loper and Steven Bird. Nltk: The natural language toolkit, 2002. [find](https://scholar.google.com/scholar?q=Edward+Loper+and+Steven+Bird%2E+Nltk%3A+The+natural+language+toolkit%2C+2002%2E)
61. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2C+2019%2E)
62. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1909%2E11)
63. Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020. — [arXiv:2004.08994](https://arxiv.org/abs/2004.08994) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Hao+Cheng%2C+Pengcheng+He%2C+Weizhu+Chen%2C+Yu+Wang%2C+Hoifung+Poon%2C+and+Jianfeng+Gao%2E+Adversarial+training+for+large+neural+language+models%2E+arXiv+preprint+arXiv%3A2004%2E08994%2C+2020%2E+%E2%80%94+%5BarXiv%3A2004)
64. Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable bert. arXiv preprint arXiv:1905.07504, 2019. — [arXiv:1905.07504](https://arxiv.org/abs/1905.07504) [find](https://scholar.google.com/scholar?q=Zhongyang+Li%2C+Xiao+Ding%2C+and+Ting+Liu%2E+Story+ending+prediction+by+transferable+bert%2E+arXiv+preprint+arXiv%3A1905%2E07504%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E07504%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1905%2E07504%29)
65. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+Winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+2012%2E)
66. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210, 2020. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+arXiv+preprint+a)
67. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational L… [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Jianfeng+Gao%2C+Xiaodong+He%2C+Li+Deng%2C+Kevin+Duh%2C+and+Ye%2DYi+Wang%2E+Representation+learning+using+multi%2Dtask+deep+neural+networks+for+semantic+classification+and+information+retrieval%2E+In+Pro) [our card](https://md.research.elamur.ai/free/en/papers/liu-2015-representation-learning-multi-task-deep/)
68. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
69. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482, 2019. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+Improving+multi%2Dtask+deep+neural+networks+via+knowledge+distillation+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1904%2E09482%2C+201)
70. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1901%2E11504%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
71. Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint arXiv:2005.00955, 2020. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2E+How+can+we+accelerate+progress+towards+human%2Dlike+linguistic+generalization%3F+arXiv+preprint+arXiv%3A2005%2E00955%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/)
72. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Ves+Stoyanov%2C+and+Luke+Zettlemoyer%2E+Bart%3A+Denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+language+ge) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
73. Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017. — [arXiv:1703.00441](https://arxiv.org/abs/1703.00441) [find](https://scholar.google.com/scholar?q=Ke+Li+and+Jitendra+Malik%2E+Learning+to+optimize+neural+nets%2E+arXiv+preprint+arXiv%3A1703%2E00441%2C+2017%2E+%E2%80%94+%5BarXiv%3A1703%2E00441%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1703%2E00441%29)
74. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+arXiv+p)
75. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Kiela Douwe. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401, 2020. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+Ku%CC%88ttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+and+Kiela+Douwe%2E+Retrieva)
76. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. [find](https://scholar.google.com/scholar?q=Peter+J%2E+Liu%2C+Mohammad+Saleh%2C+Etienne+Pot%2C+Ben+Goodrich%2C+Ryan+Sepassi%2C+Lukasz+Kaiser%2C+and+Noam+Shazeer%2E+Generating+Wikipedia+by+summarizing+long+sequences%2E+arXiv+preprint+arXiv%3A1801%2E10198%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
77. Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020. [find](https://scholar.google.com/scholar?q=Zhuohan+Li%2C+Eric+Wallace%2C+Sheng+Shen%2C+Kevin+Lin%2C+Kurt+Keutzer%2C+Dan+Klein%2C+and+Joseph+E%2E+Gonzalez%2E+Train+large%2C+then+compress%3A+Rethinking+model+size+for+efficient+training+and+inference+of+transformers)
78. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. [find](https://scholar.google.com/scholar?q=Guokun+Lai%2C+Qizhe+Xie%2C+Hanxiao+Liu%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+Race%3A+Large%2Dscale+reading+comprehension+dataset+from+examinations%2E+arXiv+preprint+arXiv%3A1704%2E04683%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
79. Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380, 2020. — [arXiv:2003.08380](https://arxiv.org/abs/2003.08380) [find](https://scholar.google.com/scholar?q=Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+Rodrigo+Nogueira%2C+Ming%2DFeng+Tsai%2C+Chuan%2DJu+Wang%2C+and+Jimmy+Lin%2E+Tttttackling+winogrande+schemas%2E+arXiv+preprint+arXiv%3A2003%2E08380%2C+2020%2E+%E2%80%94+%5BarXiv%3A2003%2E08380%5D%28https%3A%2F%2Fa)
80. David. MacKay. Information-based objective functions for active data selection. Neural Computation, 1992. Decoupled weight decay regularization. arXiv preprint [find](https://scholar.google.com/scholar?q=David%2E+MacKay%2E+Information%2Dbased+objective+functions+for+active+data+selection%2E+Neural+Computation%2C+1992%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint)
81. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305, 2017. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+6294%E2%80%936305%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
82. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
83. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696, 2016. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+A+corpus+and+evaluation+framework+for+deeper+understanding+of+commons)
84. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. ArXiv, abs/1809.02789, 2018. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+ArXiv%2C+abs%2F1809%2E02789%2C+2018%2E)
85. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training, 2018. [find](https://scholar.google.com/scholar?q=Sam+McCandlish%2C+Jared+Kaplan%2C+Dario+Amodei%2C+and+OpenAI+Dota+Team%2E+An+empirical+model+of+large%2Dbatch+training%2C+2018%2E)
86. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114–119. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Grace+Kim%2C+Mary+Ann+Marcinkiewicz%2C+Robert+MacIntyre%2C+Ann+Bies%2C+Mark+Ferguson%2C+Karen+Katz%2C+and+Britta+Schasberger%2E+The+penn+treebank%3A+annotating+predicate+argument+structure%2E+In+Procee) [our card](https://md.research.elamur.ai/free/en/papers/bank-1994-marcus-penn-treebank-annotating-predicate-argument/)
87. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+Nitish+Shirish+Keskar%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
88. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007, 2019. [find](https://scholar.google.com/scholar?q=R+Thomas+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+Right+for+the+wrong+reasons%3A+Diagnosing+syntactic+heuristics+in+natural+language+inference%2E+arXiv+preprint+arXiv%3A1902%2E01007%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
89. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting, 2018. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+Model+cards+for+model+reporting%2C+2018%2E)
90. Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456, 2020. [find](https://scholar.google.com/scholar?q=Moin+Nadeem%2C+Anna+Bethke%2C+and+Siva+Reddy%2E+Stereoset%3A+Measuring+stereotypical+bias+in+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2004%2E09456%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/)
91. Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. arXiv preprint arXiv:1907.07355, 2019. [find](https://scholar.google.com/scholar?q=Timothy+Niven+and+Hung%2DYu+Kao%2E+Probing+neural+network+comprehension+of+natural+language+arguments%2E+arXiv+preprint+arXiv%3A1907%2E07355%2C+2019%2E)
92. Peter Norvig. Natural language corpus data, 2009. [find](https://scholar.google.com/scholar?q=Peter+Norvig%2E+Natural+language+corpus+data%2C+2009%2E)
93. Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational: Man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866, 2019. [find](https://scholar.google.com/scholar?q=Malvina+Nissim%2C+Rik+van+Noord%2C+and+Rob+van+der+Goot%2E+Fair+is+better+than+sensational%3A+Man+is+to+doctor+as+woman+is+to+doctor%2E+arXiv+preprint+arXiv%3A1905%2E09866%2C+2019%2E)
94. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+Adversarial+nli%3A+A+new+benchmark+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1910%2E14599%2C+2019%2E)
95. University of Regensburg. Fascha, 2016. [find](https://scholar.google.com/scholar?q=University+of+Regensburg%2E+Fascha%2C+2016%2E)
96. Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating context-sensitive representations. arXiv preprint arXiv:1808.09121, 2018. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar+and+Jose+Camacho%2DCollados%2E+WIC%3A+10%2C000+example+pairs+for+evaluating+context%2Dsensitive+representations%2E+arXiv+preprint+arXiv%3A1808%2E09121%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
97. Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R%2E+Bowman%2E+Sentence+encoders+on+STILTs%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
98. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of EMNLP, 2018. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+J%2E+Edward+Hu%2C+Ellie+Pavlick%2C+Aaron+Steven+White%2C+and+Benjamin+Van+Durme%2E+Collecting+diverse+natural+language+inference+problems+for+sentence+representat) [our card](https://md.research.elamur.ai/free/en/papers/poliak-2018-collecting-diverse-natural-language-inference/)
99. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. [find](https://scholar.google.com/scholar?q=Denis+Paperno%2C+Germa%CC%81n+Kruszewski%2C+Angeliki+Lazaridou%2C+Quan+Ngoc+Pham%2C+Raffaella+Bernardi%2C+Sandro+Pezzelle%2C+Marco+Baroni%2C+Gemma+Boleda%2C+and+Raquel+Ferna%CC%81ndez%2E+The+lambada+dataset%3A+Word+prediction+requ) [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
100. Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih. Dissecting contextual word embeddings: Architecture and representation, 2018. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen+tau+Yih%2E+Dissecting+contextual+word+embeddings%3A+Architecture+and+representation%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
101. Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771, 2018. [find](https://scholar.google.com/scholar?q=Matt+Post%2E+A+call+for+clarity+in+reporting+BLEU+scores%2E+arXiv+preprint+arXiv%3A1804%2E08771%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
102. Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+processing+%28E) [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
103. QIANXIN. Sa-net on albert (ensemble), April 2020. [find](https://scholar.google.com/scholar?q=QIANXIN%2E+Sa%2Dnet+on+albert+%28ensemble%29%2C+April+2020%2E)
104. Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language models with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801, 2019. [find](https://scholar.google.com/scholar?q=Yusu+Qian%2C+Urwa+Muaz%2C+Ben+Zhang%2C+and+Jae+Won+Hyun%2E+Reducing+gender+bias+in+word%2Dlevel+language+models+with+a+gender%2Dequalizing+loss+function%2E+arXiv+preprint+arXiv%3A1905%2E12801%2C+2019%2E)
105. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2C+2011%2E)
106. Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. [find](https://scholar.google.com/scholar?q=Siva+Reddy%2C+Danqi+Chen%2C+and+Christopher+D+Manning%2E+Coqa%3A+A+conversational+question+answering+challenge%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A249%E2%80%93266%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
107. Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions. arXiv preprint arXiv:1710.10304, 2017. — [arXiv:1710.10304](https://arxiv.org/abs/1710.10304) [find](https://scholar.google.com/scholar?q=Scott+Reed%2C+Yutian+Chen%2C+Thomas+Paine%2C+Aa%CC%88ron+van+den+Oord%2C+SM+Eslami%2C+Danilo+Rezende%2C+Oriol+Vinyals%2C+and+Nando+de+Freitas%2E+Few%2Dshot+autoregressive+density+estimation%3A+Towards+learning+to+learn+distri)
108. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+arXiv+preprint+arXiv%3A1806%2E03822%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
109. Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR 2017 (oral), 2016. [find](https://scholar.google.com/scholar?q=Sachin+Ravi+and+Hugo+Larochelle%2E+Optimization+as+a+model+for+few%2Dshot+learning%2E+ICLR+2017+%28oral%29%2C+2016%2E)
110. Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. NumNet: Machine reading comprehension with numerical reasoning. In Proceedings of EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Qiu+Ran%2C+Yankai+Lin%2C+Peng+Li%2C+Jie+Zhou%2C+and+Zhiyuan+Liu%2E+NumNet%3A+Machine+reading+comprehension+with+numerical+reasoning%2E+In+Proceedings+of+EMNLP%2C+2019%2E)
111. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+Gender+bias+in+coreference+resolution%2E+arXiv+preprint+arXiv%3A1804%2E09301%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
112. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
113. R.S. Ross. Guide for conducting risk assessments. NIST Special Publication, 2012. [find](https://scholar.google.com/scholar?q=R%2ES%2E+Ross%2E+Guide+for+conducting+risk+assessments%2E+NIST+Special+Publication%2C+2012%2E)
114. Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019. [find](https://scholar.google.com/scholar?q=Jonathan+S%2E+Rosenfeld%2C+Amir+Rosenfeld%2C+Yonatan+Belinkov%2C+and+Nir+Shavit%2E+A+constructive+prediction+of+the+generalization+error+across+scales%2C+2019%2E)
115. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+arXiv+preprint+arXiv%3A2002%2E08910%2C+2020%2E)
116. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+tran) [our card](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)
117. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2C+2019%2E)
118. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2C+2019%2E)
119. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. Release strategies and the social impacts of language models, 2019. [find](https://scholar.google.com/scholar?q=Irene+Solaiman%2C+Miles+Brundage%2C+Jack+Clark%2C+Amanda+Askell%2C+Ariel+Herbert%2DVoss%2C+Jeff+Wu%2C+Alec+Radford%2C+Gretchen+Krueger%2C+Jong+Wook+Kim%2C+Sarah+Kreps%2C+Miles+McCain%2C+Alex+Newhouse%2C+Jason+Blazakis%2C+Kris+Mc) [our card](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/)
120. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326, 2019. [find](https://scholar.google.com/scholar?q=Emily+Sheng%2C+Kai%2DWei+Chang%2C+Premkumar+Natarajan%2C+and+Nanyun+Peng%2E+The+woman+worked+as+a+babysitter%3A+On+biases+in+language+generation%2E+arXiv+preprint+arXiv%3A1909%2E01326%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/)
121. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2C+2019%2E)
122. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. CoRR, abs/1907.10597, 2019. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Jesse+Dodge%2C+Noah+A%2E+Smith%2C+and+Oren+Etzioni%2E+Green+AI%2E+CoRR%2C+abs%2F1907%2E10597%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
123. Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709, 2015. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2C+2015%2E)
124. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+preprint)
125. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2019%2E)
126. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint arXiv:2001.07676, 2020. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+Exploiting+cloze+questions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+arXiv+preprint+arXiv%3A2001%2E07676%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
127. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450, 2019. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+MASS%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+arXiv+preprint+arXiv%3A1905%2E02450%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
128. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017. [find](https://scholar.google.com/scholar?q=Josh+Tobin%2C+Rachel+Fong%2C+Alex+Ray%2C+Jonas+Schneider%2C+Wojciech+Zaremba%2C+and+Pieter+Abbeel%2E+Domain+randomization+for+transferring+deep+neural+networks+from+simulation+to+the+real+world%2E+In+2017+IEEE%2FRSJ+)
129. Peter D. Turney and Michael L. Littman. Corpus-based learning of analogies and semantic relations. CoRR, abs/cs/0508103, 2005. — [arXiv:cs/0508103](https://arxiv.org/abs/cs/0508103) [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney+and+Michael+L%2E+Littman%2E+Corpus%2Dbased+learning+of+analogies+and+semantic+relations%2E+CoRR%2C+abs%2Fcs%2F0508103%2C+2005%2E+%E2%80%94+%5BarXiv%3Acs%2F0508103%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2Fcs%2F0508103%29)
130. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2C+2018%2E)
131. Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. Combining independent modules to solve multiple-choice synonym and analogy problems. CoRR, cs.CL/0309035, 2003. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney%2C+Michael+L%2E+Littman%2C+Jeffrey+Bigham%2C+and+Victor+Shnayder%2E+Combining+independent+modules+to+solve+multiple%2Dchoice+synonym+and+analogy+problems%2E+CoRR%2C+cs%2ECL%2F0309035%2C+2003%2E)
132. Project Turing. Microsoft research blog, Feb 2020. [find](https://scholar.google.com/scholar?q=Project+Turing%2E+Microsoft+research+blog%2C+Feb+2020%2E)
133. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching Networks for One Shot Learning. In Advances in neural information processing systems, pages 3630–3638, 2016. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Charles+Blundell%2C+Timothy+Lillicrap%2C+Daan+Wierstra%2C+et+al%2E+Matching+Networks+for+One+Shot+Learning%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3630%E2%80%933638%2C+2016%2E)
134. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sy) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
135. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261–3275, 2019. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+syst) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
136. Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Multi-agent dual learning. ICLR 2019, 2018. [find](https://scholar.google.com/scholar?q=Yiren+Wang%2C+Yingce+Xia%2C+Tianyu+He%2C+Fei+Tian%2C+Tao+Qin%2C+ChengXiang+Zhai%2C+and+Tie%2DYan+Liu%2E+Multi%2Dagent+dual+learning%2E+ICLR+2019%2C+2018%2E)
137. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training, 2019. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Zihang+Dai%2C+Eduard+Hovy%2C+Minh%2DThang+Luong%2C+and+Quoc+V%2E+Le%2E+Unsupervised+data+augmentation+for+consistency+training%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
138. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373, 2019. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Jerome+Connor%2C+Tomas+Kocisky%2C+Mike+Chrzanowski%2C+Lingpeng+Kong%2C+Angeliki+Lazaridou%2C+Wang+Ling%2C+Lei+Yu%2C+Chris+Dyer%2C+et+al%2E+Learning+and+evaluating+general+ling)
139. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Ruslan+Salakhutdinov%2C+and+Quoc+V%2E+Le%2E+XLNet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+arXiv+preprint+arXiv%3A1906%2E08237%2C+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
140. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
141. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Hannah+Rashkin%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+Franziska+Roesner%2C+and+Yejin+Choi%2E+Defending+against+neural+fake+news%2E+arXiv+preprint+arXiv%3A1905%2E12616%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
142. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+ReCoRD%3A+Bridging+the+gap+between+human+and+machine+commonsense+reading+comprehension%2E+arXiv+preprint+arXiv%3A181) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)
143. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2C+2019%2E)
144. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+ArXiv%2C+abs%2F1909%2E08593)