---
title: "True Few-Shot Learning with Language Models — Perez 2021"
description: "Reported few-shot results for pretrained language models often depend on large held-out sets for choosing prompts, hyperparameters and other design…"
authors: ["Ethan Perez", "Douwe Kiela", "Kyunghyun Cho"]
year: 2021
venue: "arXiv:2105.11447"
source: "https://arxiv.org/abs/2105.11447"
doi: "10.48550/arxiv.2105.11447"
arxiv: "2105.11447"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / True Few-Shot Learning with Language Models

# True Few-Shot Learning with Language Models

**Authors:** [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/), [Douwe Kiela](https://md.research.elamur.ai/free/en/authors/douwe-kiela/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) · **Year:** 2021 · **Venue:** arXiv:2105.11447 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2105.11447) · [PDF](https://arxiv.org/pdf/2105.11447) · [DOI 10.48550/arxiv.2105.11447](https://doi.org/10.48550/arxiv.2105.11447) · [arXiv 2105.11447](https://arxiv.org/abs/2105.11447) · [OpenAlex](https://openalex.org/W3164972323)

**Citations:** 185 citations (OpenAlex, 2026-10-14) · [218](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Reported few-shot results for pretrained language models often depend on large held-out sets for choosing prompts, hyperparameters and other design decisions. This weakens the claim that only a handful of labeled examples were needed. Perez, Kiela and Cho distinguish three settings: multi-distribution, tuned, and true few-shot learning. In true few-shot learning, every decision must be made using only the small training set. They then test how existing methods perform when model selection is restricted to those few examples. They use two standard criteria: cross-validation (including leave-one-out) and minimum description length, estimated through online coding. They also try a more conservative CV variant that penalizes variance. For both prompt selection across many LMs and hyperparameter selection for ADAPET, the criteria beat random choice only slightly and fall far behind selection with held-out data. They also often prefer models that do worse than a random pick. The authors conclude that earlier work probably overstated LM few-shot ability, that model selection is a central obstacle, and that stricter reporting protocols are needed.

## Key points
- Prompt selection was evaluated on 9 LMs spanning about three orders of magnitude in size, using 3 classification tasks and 41 tasks from the LAMA benchmark.
- For ADAPET, a state-of-the-art few-shot method, true few-shot hyperparameter selection lowered performance by 2–10% across 8 tasks.
- The findings held when the authors changed the number of examples, the compute budget, or how conservative the selection criterion was, so the problem is not limited to one configuration.
- The authors recommend that papers report every candidate prompt and hyperparameter, count validation examples as part of the few-shot budget, run test evaluation ideally only once, and avoid reusing hyperparameters tuned on benchmark validation sets such as SuperGLUE.
- The paper also notes that CLIP's zero-shot ImageNet prompts were tuned on the training set (1.28M examples), which contributed about 5% accuracy. Code is available at github.com/ethanjperez/true_few_shot.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 106 works · 82 819 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{perez2021true,
  title = {{True Few-Shot Learning with Language Models}},
  author = {Perez, Ethan and Kiela, Douwe and Cho, Kyunghyun},
  year = {2021},
  doi = {10.48550/arxiv.2105.11447},
  eprint = {2105.11447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.11447}
}
```

- [← previous MuRIL: Multilingual Representations for Indian Languages](https://md.research.elamur.ai/free/en/papers/khanuja-2021-muril-multilingual-representations-indian-languages/)
- [next → A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/en/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

91 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2C+2019%2E)
2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. [https://github.com/openai/CLIP/blob/m…](https://github.com/openai/CLIP/blob/m…) [↗](https://github.com/openai/CLIP/blob/m…) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676, 2020. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+Exploiting+cloze+questions+for+few%2Dshot+text+classification+and+natural+language+inference%2E+Computing+Research+Repository%2C+arXiv%3A2001%2E07676%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
4. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 8:423–438, 2020. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F%2E+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+How+can+we+know+what+language+models+know%3F+TACL%2C+8%3A423%E2%80%93438%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
5. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Adam+Fisch%2C+and+Danqi+Chen%2E+Making+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2012%2E15723%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
6. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021. [find](https://scholar.google.com/scholar?q=Tony+Z%2E+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)
7. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity, 2021. [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
8. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, L. Carin, and W. Chen. What makes good in-context examples for gpt-3? arXiv, abs/2101.06804, 2021. [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+L%2E+Carin%2C+and+W%2E+Chen%2E+What+makes+good+in%2Dcontext+examples+for+gpt%2D3%3F+arXiv%2C+abs%2F2101%2E06804%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
9. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. Computing Research Repository, arXiv:2009.07118, 2020. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+It%E2%80%99s+not+just+size+that+matters%3A+Small+language+models+are+also+few%2Dshot+learners%2E+Computing+Research+Repository%2C+arXiv%3A2009%2E07118%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
10. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. Rissanen data analysis: Examining dataset characteristics via description length. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Douwe+Kiela%2C+and+Kyunghyun+Cho%2E+Rissanen+data+analysis%3A+Examining+dataset+characteristics+via+description+length%2E+In+ICML%2C+2021%2E)
11. Timo Schick and H. Schutze. Few-shot text generation with pattern-exploiting training. arXiv, abs/2012.11926, 2020. — [arXiv:2012.11926](https://arxiv.org/abs/2012.11926) [find](https://scholar.google.com/scholar?q=Timo+Schick+and+H%2E+Schutze%2E+Few%2Dshot+text+generation+with+pattern%2Dexploiting+training%2E+arXiv%2C+abs%2F2012%2E11926%2C+2020%2E+%E2%80%94+%5BarXiv%3A2012%2E11926%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E11926%29)
12. Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. Improving and simplifying pattern exploiting training. arxiv preprint arXiv:2103.11955, 2021. [find](https://scholar.google.com/scholar?q=Derek+Tam%2C+Rakesh+R+Menon%2C+Mohit+Bansal%2C+Shashank+Srivastava%2C+and+Colin+Raffel%2E+Improving+and+simplifying+pattern+exploiting+training%2E+arxiv+preprint+arXiv%3A2103%2E11955%2C+2021%2E)
13. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Aditya+Ramesh%2C+Gabriel+Goh%2C+Sandhini+Agarwal%2C+Girish+Sastry%2C+Amanda+Askell%2C+Pamela+Mishkin%2C+Jack+Clark%2C+Gretchen+Krueger%2C+and+Ilya+Sutskever%2E+Learning+trans) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
14. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. Entailment as few-shot learner, 2021. [find](https://scholar.google.com/scholar?q=Sinong+Wang%2C+Han+Fang%2C+Madian+Khabsa%2C+Hanzi+Mao%2C+and+Hao+Ma%2E+Entailment+as+few%2Dshot+learner%2C+2021%2E)
15. Teven Le Scao and Alexander M. Rush. How many data points is a prompt worth?, 2021. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao+and+Alexander+M%2E+Rush%2E+How+many+data+points+is+a+prompt+worth%3F%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/scao-2021-how-many-data-points-is/)
16. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv, abs/1910.01108, 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv%2C+abs%2F1910%2E01108%2C+2019%2E)
17. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In EMNLP, pages 2463–2473, Hong Kong, China, November 2019. ACL. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+Language+models+as+knowledge+bases%3F+In+EMNLP%2C+pages+2463%E2%80%932473%2C+Hong+Kong%2C+China%2C+Novem) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
18. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, NeuRIPS, volume 29. Curran Associates, Inc., 2016. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Charles+Blundell%2C+Timothy+Lillicrap%2C+koray+kavukcuoglu%2C+and+Daan+Wierstra%2E+Matching+networks+for+one+shot+learning%2E+In+D%2E+Lee%2C+M%2E+Sugiyama%2C+U%2E+Luxburg%2C+I%2E+Guyon%2C+and+R%2E+Garnett%2C+editors)
19. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeuRIPS, volume 30. Curran Associates, Inc., 2017. [find](https://scholar.google.com/scholar?q=Jake+Snell%2C+Kevin+Swersky%2C+and+Richard+Zemel%2E+Prototypical+networks+for+few%2Dshot+learning%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+R%2E+Fergus%2C+S%2E+Vishwanathan%2C+and+R%2E+Garnett%2C+editors%2C+NeuRI)
20. S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017. [find](https://scholar.google.com/scholar?q=S%2E+Ravi+and+H%2E+Larochelle%2E+Optimization+as+a+model+for+few%2Dshot+learning%2E+In+ICLR%2C+2017%2E)
21. Ke Li and Jitendra Malik. Learning to optimize. arXiv, abs/1606.01885, 2017. — [arXiv:1606.01885](https://arxiv.org/abs/1606.01885) [find](https://scholar.google.com/scholar?q=Ke+Li+and+Jitendra+Malik%2E+Learning+to+optimize%2E+arXiv%2C+abs%2F1606%2E01885%2C+2017%2E+%E2%80%94+%5BarXiv%3A1606%2E01885%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1606%2E01885%29)
22. David M. Allen. The relationship between variable selection and data agumentation and a method for prediction. Technometrics, 16(1):125–127, 1974. [find](https://scholar.google.com/scholar?q=David+M%2E+Allen%2E+The+relationship+between+variable+selection+and+data+agumentation+and+a+method+for+prediction%2E+Technometrics%2C+16%281%29%3A125%E2%80%93127%2C+1974%2E)
23. M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society. Series A (Methodological), 36:111–133, 1974. [find](https://scholar.google.com/scholar?q=M%2E+Stone%2E+Cross%2Dvalidatory+choice+and+assessment+of+statistical+predictions%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+A+%28Methodological%29%2C+36%3A111%E2%80%93133%2C+1974%2E)
24. Seymour Geisser. The predictive sample reuse method with applications. Journal of the American Statistical Association, 70(350):320–328, 1975. [find](https://scholar.google.com/scholar?q=Seymour+Geisser%2E+The+predictive+sample+reuse+method+with+applications%2E+Journal+of+the+American+Statistical+Association%2C+70%28350%29%3A320%E2%80%93328%2C+1975%2E)
25. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001. [find](https://scholar.google.com/scholar?q=Trevor+Hastie%2C+Robert+Tibshirani%2C+and+Jerome+Friedman%2E+The+Elements+of+Statistical+Learning%2E+Springer+Series+in+Statistics%2E+Springer+New+York+Inc%2E%2C+New+York%2C+NY%2C+USA%2C+2001%2E)
26. Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, volume 70 of ICML’17, pages 1126–1135. JMLR.org, 2017. [find](https://scholar.google.com/scholar?q=Chelsea+Finn%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+In+ICML%2C+volume+70+of+ICML%E2%80%9917%2C+pages+1126%E2%80%931135%2E+JMLR%2Eorg%2C+2017%2E)
27. Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, NeurIPS, volume 32. Curran Associates, Inc., 2019. [find](https://scholar.google.com/scholar?q=Aravind+Rajeswaran%2C+Chelsea+Finn%2C+Sham+M+Kakade%2C+and+Sergey+Levine%2E+Meta%2Dlearning+with+implicit+gradients%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%27Alch%C3%A9%2DBuc%2C+E%2E+Fox%2C+and+R%2E+Garnett%2C+editors)
28. J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465 – 471, 1978. [find](https://scholar.google.com/scholar?q=J%2E+Rissanen%2E+Modeling+by+shortest+data+description%2E+Automatica%2C+14%285%29%3A465+%E2%80%93+471%2C+1978%2E)
29. J. Rissanen. Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory, 30(4):629–636, 1984. [find](https://scholar.google.com/scholar?q=J%2E+Rissanen%2E+Universal+coding%2C+information%2C+prediction%2C+and+estimation%2E+IEEE+Transactions+on+Information+Theory%2C+30%284%29%3A629%E2%80%93636%2C+1984%2E)
30. A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach. Journal of the Royal Statistical Society. Series A (General), 147(2):278–292, 1984. [find](https://scholar.google.com/scholar?q=A%2E+P%2E+Dawid%2E+Present+position+and+potential+developments%3A+Some+personal+views%3A+Statistical+theory%3A+The+prequential+approach%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+A+%28General%29%2C+147%282%29%3A278%E2%80%9329)
31. Peter Grünwald. A tutorial introduction to the minimum description length principle. CoRR, math.ST/0406077, 06 2004. [find](https://scholar.google.com/scholar?q=Peter+Gr%C3%BCnwald%2E+A+tutorial+introduction+to+the+minimum+description+length+principle%2E+CoRR%2C+math%2EST%2F0406077%2C+06+2004%2E)
32. Léonard Blier and Yann Ollivier. The description length of deep learning models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeuRIPS, volume 31, pages 2216–2226. Curran Associates, Inc., 2018. [find](https://scholar.google.com/scholar?q=L%C3%A9onard+Blier+and+Yann+Ollivier%2E+The+description+length+of+deep+learning+models%2E+In+S%2E+Bengio%2C+H%2E+Wallach%2C+H%2E+Larochelle%2C+K%2E+Grauman%2C+N%2E+Cesa%2DBianchi%2C+and+R%2E+Garnett%2C+editors%2C+NeuRIPS%2C+volume+31%2C+page)
33. Alselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K. Warmuth. Occam’s razor. Inf. Process. Lett., 24(6):377–380, April 1987. [find](https://scholar.google.com/scholar?q=Alselm+Blumer%2C+Andrzej+Ehrenfeucht%2C+David+Haussler%2C+and+Manfred+K%2E+Warmuth%2E+Occam%E2%80%99s+razor%2E+Inf%2E+Process%2E+Lett%2E%2C+24%286%29%3A377%E2%80%93380%2C+April+1987%2E)
34. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. CoRR, abs/2104.06644, 2021. [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Robin+Jia%2C+Dieuwke+Hupkes%2C+Joelle+Pineau%2C+Adina+Williams%2C+and+Douwe+Kiela%2E+Masked+language+modeling+and+the+distributional+hypothesis%3A+Order+word+matters+pre%2Dtraining+for+little%2E+CoRR%2C+) [our card](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)
35. P. J. Phillips, Hyeonjoon Moon, S. A. Rizvi, and P. J. Rauss. The feret evaluation methodology for face-recognition algorithms. TPAMI, 22(10):1090–1104, 2000. [find](https://scholar.google.com/scholar?q=P%2E+J%2E+Phillips%2C+Hyeonjoon+Moon%2C+S%2E+A%2E+Rizvi%2C+and+P%2E+J%2E+Rauss%2E+The+feret+evaluation+methodology+for+face%2Drecognition+algorithms%2E+TPAMI%2C+22%2810%29%3A1090%E2%80%931104%2C+2000%2E)
36. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Fairness, Accountability and Transparency, volume 81 of PMLR, pages 77–91, New York, NY, USA, 23–24 Feb 2018. PMLR. [find](https://scholar.google.com/scholar?q=Joy+Buolamwini+and+Timnit+Gebru%2E+Gender+shades%3A+Intersectional+accuracy+disparities+in+commercial+gender+classification%2E+In+Sorelle+A%2E+Friedler+and+Christo+Wilson%2C+editors%2C+Fairness%2C+Accountability+an)
37. Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 123–129, New York, NY, USA, 2018. Association for Computing… [find](https://scholar.google.com/scholar?q=Peter+Henderson%2C+Koustuv+Sinha%2C+Nicolas+Angelard%2DGontier%2C+Nan+Rosemary+Ke%2C+Genevieve+Fried%2C+Ryan+Lowe%2C+and+Joelle+Pineau%2E+Ethical+challenges+in+data%2Ddriven+dialogue+systems%2E+In+AAAI%2FACM+Conference+on+)
38. Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Gottardi, Sanjeev Kwatra, Sanju Pancholi, Ming Cheng, Qinglang Chen, Lauren Stubel, Karthik Gopalakrishnan, Kate Bland, Raefer Gabriel, Arindam Mandal, Dilek Hakkani-Tür, Gene Hwang, Nate Michel, Eric King,… — [arXiv:1812.10757](https://arxiv.org/abs/1812.10757) [find](https://scholar.google.com/scholar?q=Chandra+Khatri%2C+Behnam+Hedayatnia%2C+Anu+Venkatesh%2C+Jeff+Nunn%2C+Yi+Pan%2C+Qing+Liu%2C+Han+Song%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Sanju+Pancholi%2C+Ming+Cheng%2C+Qinglang+Chen%2C+Lauren+Stubel%2C+Karthik+Gopalakrishnan)
39. Javier García, Fern, and o Fernández. A comprehensive survey on safe reinforcement learning. JMLR, 16(42):1437–1480, 2015. [find](https://scholar.google.com/scholar?q=Javier+Garc%C3%ADa%2C+Fern%2C+and+o+Fern%C3%A1ndez%2E+A+comprehensive+survey+on+safe+reinforcement+learning%2E+JMLR%2C+16%2842%29%3A1437%E2%80%931480%2C+2015%2E)
40. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. CoRR, abs/1606.06565, 2016. [find](https://scholar.google.com/scholar?q=Dario+Amodei%2C+Chris+Olah%2C+Jacob+Steinhardt%2C+Paul+F%2E+Christiano%2C+John+Schulman%2C+and+Dan+Man%C3%A9%2E+Concrete+problems+in+AI+safety%2E+CoRR%2C+abs%2F1606%2E06565%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/amodei-2016-concrete-problems-ai-safety/)
41. Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. JMLR, 11(116):3571–3594, 2010. [find](https://scholar.google.com/scholar?q=Sumio+Watanabe%2E+Asymptotic+equivalence+of+bayes+cross+validation+and+widely+applicable+information+criterion+in+singular+learning+theory%2E+JMLR%2C+11%28116%29%3A3571%E2%80%933594%2C+2010%2E)
42. H. Akaike. A new look at the statistical model identification. TACON, 19(6):716–723, 1974. [find](https://scholar.google.com/scholar?q=H%2E+Akaike%2E+A+new+look+at+the+statistical+model+identification%2E+TACON%2C+19%286%29%3A716%E2%80%93723%2C+1974%2E)
43. C. L. Mallows. Some comments on cp. Technometrics, 15(4):661–675, 1973. [find](https://scholar.google.com/scholar?q=C%2E+L%2E+Mallows%2E+Some+comments+on+cp%2E+Technometrics%2C+15%284%29%3A661%E2%80%93675%2C+1973%2E)
44. Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 507–523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Frank+Hutter%2C+Holger+H%2E+Hoos%2C+and+Kevin+Leyton%2DBrown%2E+Sequential+model%2Dbased+optimization+for+general+algorithm+configuration%2E+In+Carlos+A%2E+Coello+Coello%2C+editor%2C+Learning+and+Intelligent+Optimization)
45. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, NeuRIPS, volume 24. Curran Associates, Inc., 2011. [find](https://scholar.google.com/scholar?q=James+Bergstra%2C+R%C3%A9mi+Bardenet%2C+Yoshua+Bengio%2C+and+Bal%C3%A1zs+K%C3%A9gl%2E+Algorithms+for+hyperparameter+optimization%2E+In+J%2E+Shawe%2DTaylor%2C+R%2E+Zemel%2C+P%2E+Bartlett%2C+F%2E+Pereira%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+NeuRIPS)
46. Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, NeuRIPS, volume 25. Curran Associates, Inc., 2012. [find](https://scholar.google.com/scholar?q=Jasper+Snoek%2C+Hugo+Larochelle%2C+and+Ryan+P+Adams%2E+Practical+bayesian+optimization+of+machine+learning+algorithms%2E+In+F%2E+Pereira%2C+C%2E+J%2E+C%2E+Burges%2C+L%2E+Bottou%2C+and+K%2E+Q%2E+Weinberger%2C+editors%2C+NeuRIPS%2C+volu)
47. Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Chapter 15 - evolving deep neural networks. In Robert Kozma, Cesare Alippi, Yoonsuck Choe, and Francesco Carlo Morabito, editors,… [find](https://scholar.google.com/scholar?q=Risto+Miikkulainen%2C+Jason+Liang%2C+Elliot+Meyerson%2C+Aditya+Rawal%2C+Daniel+Fink%2C+Olivier+Francon%2C+Bala+Raju%2C+Hormoz+Shahrzad%2C+Arshak+Navruzyan%2C+Nigel+Duffy%2C+and+Babak+Hodjat%2E+Chapter+15+%2D+evolving+deep+ne)
48. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. AAAI, 33(01):4780–4789, Jul. 2019. [find](https://scholar.google.com/scholar?q=Esteban+Real%2C+Alok+Aggarwal%2C+Yanping+Huang%2C+and+Quoc+V%2E+Le%2E+Regularized+evolution+for+image+classifier+architecture+search%2E+AAAI%2C+33%2801%29%3A4780%E2%80%934789%2C+Jul%2E+2019%2E)
49. Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR. OpenReview.net, 2017. [find](https://scholar.google.com/scholar?q=Barret+Zoph+and+Quoc+V%2E+Le%2E+Neural+architecture+search+with+reinforcement+learning%2E+In+ICLR%2E+OpenReview%2Enet%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/zoph-2016-neural-architecture-search-reinforcement-learning/)
50. J. Larsen, L.K. Hansen, C. Svarer, and M. Ohlsson. Design and regularization of neural networks: the optimal use of a validation set. In Neural Networks for Signal Processing VI. IEEE Signal Processing Society Workshop, pages 62–71, 1996. [find](https://scholar.google.com/scholar?q=J%2E+Larsen%2C+L%2EK%2E+Hansen%2C+C%2E+Svarer%2C+and+M%2E+Ohlsson%2E+Design+and+regularization+of+neural+networks%3A+the+optimal+use+of+a+validation+set%2E+In+Neural+Networks+for+Signal+Processing+VI%2E+IEEE+Signal+Processin)
51. Yoshua Bengio. Gradient-Based Optimization of Hyperparameters. Neural Computation, 12(8):1889–1900, 08 2000. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2E+Gradient%2DBased+Optimization+of+Hyperparameters%2E+Neural+Computation%2C+12%288%29%3A1889%E2%80%931900%2C+08+2000%2E)
52. O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for support vector machines. Machine Learning, 46:131–159, 2004. [find](https://scholar.google.com/scholar?q=O%2E+Chapelle%2C+V%2E+Vapnik%2C+O%2E+Bousquet%2C+and+S%2E+Mukherjee%2E+Choosing+multiple+parameters+for+support+vector+machines%2E+Machine+Learning%2C+46%3A131%E2%80%93159%2C+2004%2E)
53. Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Hanxiao+Liu%2C+Karen+Simonyan%2C+and+Yiming+Yang%2E+DARTS%3A+Differentiable+architecture+search%2E+In+ICLR%2C+2019%2E)
54. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In EMNLP, pages 4222–4235, Online, November 2020. ACL. [find](https://scholar.google.com/scholar?q=Taylor+Shin%2C+Yasaman+Razeghi%2C+Robert+L%2E+Logan+IV%2C+Eric+Wallace%2C+and+Sameer+Singh%2E+AutoPrompt%3A+Eliciting+Knowledge+from+Language+Models+with+Automatically+Generated+Prompts%2E+In+EMNLP%2C+pages+4222%E2%80%934235%2C+) [our card](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/)
55. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2021. [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Yanan+Zheng%2C+Zhengxiao+Du%2C+Ming+Ding%2C+Yujie+Qian%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+Gpt+understands%2C+too%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/)
56. Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: learning vs. learning to recall. CoRR, abs/2104.05240, 2021. [find](https://scholar.google.com/scholar?q=Zexuan+Zhong%2C+Dan+Friedman%2C+and+Danqi+Chen%2E+Factual+probing+is+%5BMASK%5D%3A+learning+vs%2E+learning+to+recall%2E+CoRR%2C+abs%2F2104%2E05240%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/)
57. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of EMNLP, pages 803–818, Online, November 2020. ACL. [find](https://scholar.google.com/scholar?q=Nina+Poerner%2C+Ulli+Waltinger%2C+and+Hinrich+Sch%C3%BCtze%2E+E%2DBERT%3A+Efficient%2Dyet%2Deffective+entity+embeddings+for+BERT%2E+In+Findings+of+EMNLP%2C+pages+803%E2%80%93818%2C+Online%2C+November+2020%2E+ACL%2E)
58. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+R%C3%A9mi+Louf%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+Patrick+von+Platen%2C+Clara+M)
59. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+Alban+Desmaison%2C+Andreas+Kopf%2C+Edward+Yang%2C+Zachary+De) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
60. Elena Voita and Ivan Titov. Information-theoretic probing with minimum description length. In EMNLP, pages 183–196, Online, November 2020. ACL. [find](https://scholar.google.com/scholar?q=Elena+Voita+and+Ivan+Titov%2E+Information%2Dtheoretic+probing+with+minimum+description+length%2E+In+EMNLP%2C+pages+183%E2%80%93196%2C+Online%2C+November+2020%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)
61. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+syst) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
62. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+ICLR%2C+2020%2E)
63. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, Minneapolis, Minnesota, June 2019. ACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2C+pages+4171%E2%80%934186%2C+Minneapolis%2C+Minnesota%2C+J) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
64. Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=Eleni+Triantafillou%2C+Tyler+Zhu%2C+Vincent+Dumoulin%2C+Pascal+Lamblin%2C+Utku+Evci%2C+Kelvin+Xu%2C+Ross+Goroshin%2C+Carles+Gelada%2C+Kevin+Swersky%2C+Pierre%2DAntoine+Manzagol%2C+and+Hugo+Larochelle%2E+Meta%2Ddataset%3A+A+datas)
65. Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in NLP. CoRR, abs/2104.08835, 2021. [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Bill+Yuchen+Lin%2C+and+Xiang+Ren%2E+Crossfit%3A+A+few%2Dshot+learning+challenge+for+cross%2Dtask+generalization+in+NLP%2E+CoRR%2C+abs%2F2104%2E08835%2C+2021%2E)
66. Rich Caruana. Learning many related tasks at the same time with backpropagation. In G. Tesauro, D. Touretzky, and T. Leen, editors, NeuRIPS, volume 7. MIT Press, 1995. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+Learning+many+related+tasks+at+the+same+time+with+backpropagation%2E+In+G%2E+Tesauro%2C+D%2E+Touretzky%2C+and+T%2E+Leen%2C+editors%2C+NeuRIPS%2C+volume+7%2E+MIT+Press%2C+1995%2E)
67. Rich Caruana. Multitask learning. Machine Learning, 28(1):41–75, July 1997. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+Multitask+learning%2E+Machine+Learning%2C+28%281%29%3A41%E2%80%9375%2C+July+1997%2E)
68. Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088, 2018. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+F%C3%A9vry%2C+and+Samuel+R%2E+Bowman%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+CoRR%2C+abs%2F1811%2E01088%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
69. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In ACL, pages 4487–4496, Florence, Italy, July 2019. ACL. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+In+ACL%2C+pages+4487%E2%80%934496%2C+Florence%2C+Italy%2C+July+2019%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
70. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for the Winograd schema challenge. In ACL, pages 4837–4842, Florence, Italy, July 2019. ACL. [find](https://scholar.google.com/scholar?q=Vid+Kocijan%2C+Ana%2DMaria+Cretu%2C+Oana%2DMaria+Camburu%2C+Yordan+Yordanov%2C+and+Thomas+Lukasiewicz%2E+A+surprisingly+robust+trick+for+the+Winograd+schema+challenge%2E+In+ACL%2C+pages+4837%E2%80%934842%2C+Florence%2C+Italy%2C+July)
71. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, NeuRIPS, volume 33, pages 6256–6268. Curran Associates, Inc., 2020. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Zihang+Dai%2C+Eduard+Hovy%2C+Thang+Luong%2C+and+Quoc+Le%2E+Unsupervised+data+augmentation+for+consistency+training%2E+In+H%2E+Larochelle%2C+M%2E+Ranzato%2C+R%2E+Hadsell%2C+M%2E+F%2E+Balcan%2C+and+H%2E+Lin%2C+editors%2C+NeuR) [our card](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/)
72. Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL, pages 2147–2157, Online, July 2020. ACL. [find](https://scholar.google.com/scholar?q=Jiaao+Chen%2C+Zichao+Yang%2C+and+Diyi+Yang%2E+MixText%3A+Linguistically%2Dinformed+interpolation+of+hidden+space+for+semi%2Dsupervised+text+classification%2E+In+ACL%2C+pages+2147%E2%80%932157%2C+Online%2C+July+2020%2E+ACL%2E)
73. Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP, pages 1008–1025, Online, November 2020. ACL. [find](https://scholar.google.com/scholar?q=Yiben+Yang%2C+Chaitanya+Malaviya%2C+Jared+Fernandez%2C+Swabha+Swayamdipta%2C+Ronan+Le+Bras%2C+Ji%2DPing+Wang%2C+Chandra+Bhagavatula%2C+Yejin+Choi%2C+and+Doug+Downey%2E+Generative+data+augmentation+for+commonsense+reasoni) [our card](https://md.research.elamur.ai/free/en/papers/yang-2020-generative-data-augmentation-commonsense-reasoning/)
74. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+Eneko+Agirre%2C+and+Kyunghyun+Cho%2E+Unsupervised+neural+machine+translation%2E+In+ICLR%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
75. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Alexis+Conneau%2C+Ludovic+Denoyer%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+Unsupervised+machine+translation+using+monolingual+corpora+only%2E+In+ICLR%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/lample-2017-unsupervised-machine-translation-using-monolingual/)
76. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP, pages 8864–8880, Online, November 2020. ACL. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Patrick+Lewis%2C+Wen%2Dtau+Yih%2C+Kyunghyun+Cho%2C+and+Douwe+Kiela%2E+Unsupervised+question+decomposition+for+question+answering%2E+In+EMNLP%2C+pages+8864%E2%80%938880%2C+Online%2C+November+2020%2E+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2020-unsupervised-question-decomposition-question-answering/)
77. Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. CoRR, abs/1804.09170, 2018. — [arXiv:1804.09170](https://arxiv.org/abs/1804.09170) [find](https://scholar.google.com/scholar?q=Avital+Oliver%2C+Augustus+Odena%2C+Colin+Raffel%2C+Ekin+D%2E+Cubuk%2C+and+Ian+J%2E+Goodfellow%2E+Realistic+evaluation+of+deep+semi%2Dsupervised+learning+algorithms%2E+CoRR%2C+abs%2F1804%2E09170%2C+2018%2E+%E2%80%94+%5BarXiv%3A1804%2E09170%5D%28ht)
78. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, pages 2185–2194, Hong Kong, China, November 2019. ACL. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Suchin+Gururangan%2C+Dallas+Card%2C+Roy+Schwartz%2C+and+Noah+A%2E+Smith%2E+Show+your+work%3A+Improved+reporting+of+experimental+results%2E+In+EMNLP%2C+pages+2185%E2%80%932194%2C+Hong+Kong%2C+China%2C+November+2019%2E+AC)
79. Allyson Ettinger. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. TACL, 8:34–48, 2020. [find](https://scholar.google.com/scholar?q=Allyson+Ettinger%2E+What+BERT+is+not%3A+Lessons+from+a+new+suite+of+psycholinguistic+diagnostics+for+language+models%2E+TACL%2C+8%3A34%E2%80%9348%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/)
80. Aki Vehtari, Andrew Gelman, and Jonah Gabry. Practical bayesian model evaluation using leave-one-out cross-validation and waic. Statistics and Computing, 27(5):1413–1432, September 2017. [find](https://scholar.google.com/scholar?q=Aki+Vehtari%2C+Andrew+Gelman%2C+and+Jonah+Gabry%2E+Practical+bayesian+model+evaluation+using+leave%2Done%2Dout+cross%2Dvalidation+and+waic%2E+Statistics+and+Computing%2C+27%285%29%3A1413%E2%80%931432%2C+September+2017%2E)
81. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06305, 2020. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Gabriel+Ilharco%2C+Roy+Schwartz%2C+Ali+Farhadi%2C+Hannaneh+Hajishirzi%2C+and+Noah+A%2E+Smith%2E+Fine%2Dtuning+pretrained+language+models%3A+Weight+initializations%2C+data+orders%2C+and+early+stopping%2E+CoRR%2C+)
82. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACL, pages 2924–2936, Minneapolis, Minnesota, June 2019. ACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+NAACL%2C+pages+2924%E2%80%9329) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
83. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. Sinn und Bedeutung, 23(2):107–124, July 2019. [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+de+Marneffe%2C+Mandy+Simons%2C+and+Judith+Tonhauser%2E+The+commitmentbank%3A+Investigating+projection+in+naturally+occurring+discourse%2E+Sinn+und+Bedeutung%2C+23%282%29%3A107%E2%80%93124%2C+July+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/enprag-2019-marneffe-commitmentbank-investigating-projection-naturally/)
84. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recogni… [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Joaquin+Qui%C3%B1onero%2DCandela%2C+Ido+Dagan%2C+Bernardo+Magnini%2C+and+Florence+d%E2%80%99Alch%C3%A9+Buc%2C+editors%2C+Machi)
85. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E+In+Second+PASCAL+Challenges+Worksh)
86. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, RTE ’07, page 1–9, USA, 2007. ACL. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+The+third+pascal+recognizing+textual+entailment+challenge%2E+In+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Paraphrasing%2C+RTE+%E2%80%9907%2C+page)
87. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In TAC, 2009. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E+In+TAC%2C+2009%2E)
88. Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In NAACL. ACL, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar+and+Jose+Camacho%2DCollados%2E+WiC%3A+The+word%2Din%2Dcontext+dataset+for+evaluating+context%2Dsensitive+meaning+representations%2E+In+NAACL%2E+ACL%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
89. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In KR, KR’12, page 552–561. AAAI Press, 2012. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+winograd+schema+challenge%2E+In+KR%2C+KR%E2%80%9912%2C+page+552%E2%80%93561%2E+AAAI+Press%2C+2012%2E)
90. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL. ACL, 2018. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+NAACL%2E+ACL%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
91. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810.12885, 2018. [find](https://scholar.google.com/scholar?q=Sheng+Zhang%2C+Xiaodong+Liu%2C+Jingjing+Liu%2C+Jianfeng+Gao%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+ReCoRD%3A+Bridging+the+gap+between+human+and+machine+commonsense+reading+comprehension%2E+arXiv+preprint+1810%2E1288) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)