---
title: "BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language — Huebner 2021"
description: "The authors test whether a transformer language model can acquire English grammar when it receives the same amount… 46 references, cited by 91 papers here."
authors: ["Philip A. Huebner", "Elior Sulem", "Fisher Cynthia", "Dan Roth"]
year: 2021
venue: "Proceedings of the 25th Conference on Computational Natural Language Learning"
source: "https://aclanthology.org/2021.conll-1.49/"
doi: "10.18653/v1/2021.conll-1.49"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huebner-2021-babyberta-learning-more-grammar-small/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huebner-2021-babyberta-learning-more-grammar-small/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

# BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

**Authors:** Philip A. Huebner, [Elior Sulem](https://md.research.elamur.ai/free/en/authors/elior-sulem/), Fisher Cynthia and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2021 · **Venue:** Proceedings of the 25th Conference on Computational Natural Language Learning · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.conll-1.49/) · [PDF](https://aclanthology.org/2021.conll-1.49.pdf) · [DOI 10.18653/v1/2021.conll-1.49](https://doi.org/10.18653/v1/2021.conll-1.49)

**Citations:** [91](https://md.research.elamur.ai/free/en/papers/huebner-2021-babyberta-learning-more-grammar-small/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors test whether a transformer language model can acquire English grammar when it receives the same amount and the same type of speech as a child before the age of six. The answer bears on a long-standing debate in cognitive science: whether linguistic experience alone suffices or whether innate linguistic biases are required. Standard models such as RoBERTa are trained on far larger amounts of text than a child hears, so they say little about child learning. The team trained a scaled-down version of RoBERTa, BabyBERTa, from scratch on the AO-CHILDES corpus, that is, transcripts of adult speech directed at children. Hyperparameters were tuned for small data, and prediction of unmasked tokens was disabled entirely. For comparison, the model was also trained on Wikipedia corpora and on simplified news, AO-Newsela. For evaluation, a new BLiMP-style set of minimal pairs with child-accessible vocabulary was created. In average accuracy, BabyBERTa nearly matched RoBERTa-base, although it has about 15 times fewer parameters and saw about 6,000 times fewer words. The model and the test set have been released for psycholinguists, as well as for those developing more economical models.

## Key points
- **Data:** AO-CHILDES contains about 5M words of American English from the CHILDES database. As controls, three Wikipedia corpora (Wikipedia-1/2/3) and AO-Newsela, news articles simplified for children, were used. All corpora are of roughly the same size.
- **Model:** about 8M parameters, 8 layers, 8 attention heads, 256 hidden units. A BPE vocabulary of 8192 units was built from lowercased corpora. The unmasking probability was reduced from 0.1 to 0.0, and each sequence was repeated 10 times with different masks.
- **Comparison with RoBERTa-base:** 125M parameters and 30B words versus 5M words; 1024 V100 GPUs for a day versus a single GTX1080 for two hours. Average accuracy on the test set is 81.0 for RoBERTa-base and 80.5 for BabyBERTa.
- **Limitations:** of 13 grammatical phenomena, the model performs noticeably above chance on most. However, its performance stays at chance on adjunct islands, ellipsis, gender agreement, NPI licensing, and superlative quantifiers.
- **Resource:** a new test set of grammatical minimal pairs, compatible with the restricted vocabulary of child-directed speech, has been released.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/)

## Cite

Cite as: Philip A. Huebner, Elior Sulem, Fisher Cynthia and Dan Roth. 2021. BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language. Proceedings of the 25th Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{huebner2021babyberta,
  title = {{BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language}},
  author = {Philip A. Huebner, Elior Sulem, Fisher Cynthia and Dan Roth},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.49},
  url = {https://doi.org/10.18653/v1/2021.conll-1.49}
}
```

- [← previous SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/)
- [next → Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://md.research.elamur.ai/free/en/papers/kim-2021-perspective-taking-pragmatics-generating-empathetic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Geoff Bacon and Terry Regier. 2019. Does BERT agree? Evaluating knowledge of structure dependence through agreement relations. arXiv preprint arXiv:1908.09892. — [arXiv:1908.09892](https://arxiv.org/abs/1908.09892) [find](https://scholar.google.com/scholar?q=Geoff+Bacon+and+Terry+Regier%2E+2019%2E+Does+BERT+agree%3F+Evaluating+knowledge+of+structure+dependence+through+agreement+relations%2E+arXiv+preprint+arXiv%3A1908%2E09892%2E+%E2%80%94+%5BarXiv%3A1908%2E09892%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
2. Ellen Gurman Bard and Anne H Anderson. 1994. The unintelligibility of speech to children: effects of referent availability. J. Child Lang., 21(3):623–648. [find](https://scholar.google.com/scholar?q=Ellen+Gurman+Bard+and+Anne+H+Anderson%2E+1994%2E+The+unintelligibility+of+speech+to+children%3A+effects+of+referent+availability%2E+J%2E+Child+Lang%2E%2C+21%283%29%3A623%E2%80%93648%2E)
3. Andrew Biemiller. 2003. Vocabulary: Needed if more children are to read well. Reading psychology, 24(3-4):323–335. [find](https://scholar.google.com/scholar?q=Andrew+Biemiller%2E+2003%2E+Vocabulary%3A+Needed+if+more+children+are+to+read+well%2E+Reading+psychology%2C+24%283%2D4%29%3A323%E2%80%93335%2E)
4. Rebecca L Gómez and LouAnn Gerken. 2000. Infant artificial language learning and language acquisition. Trends in cognitive sciences, 4(5):178–186. [find](https://scholar.google.com/scholar?q=Rebecca+L+G%C3%B3mez+and+LouAnn+Gerken%2E+2000%2E+Infant+artificial+language+learning+and+language+acquisition%2E+Trends+in+cognitive+sciences%2C+4%285%29%3A178%E2%80%93186%2E)
5. Noah Daniel Goodman, Joshua B Tenenbaum, and Michael C Frank. 2007. A bayesian framework for cross-situational word-learning. In Proc. of Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Noah+Daniel+Goodman%2C+Joshua+B+Tenenbaum%2C+and+Michael+C+Frank%2E+2007%2E+A+bayesian+framework+for+cross%2Dsituational+word%2Dlearning%2E+In+Proc%2E+of+Advances+in+Neural+Information+Processing+Systems%2E)
6. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… [our card](https://md.research.elamur.ai/free/en/papers/gulordava-2018-colorless-green-recurrent-networks-dream/)
7. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. [find](https://scholar.google.com/scholar?q=Zellig+S+Harris%2E+1954%2E+Distributional+structure%2E+Word%2C+10%282%2D3%29%3A146%E2%80%93162%2E)
8. Betty Hart and Todd R Risley. 1995. Meaningful differences in the everyday experience of young American children. Paul H Brookes Publishing. [find](https://scholar.google.com/scholar?q=Betty+Hart+and+Todd+R+Risley%2E+1995%2E+Meaningful+differences+in+the+everyday+experience+of+young+American+children%2E+Paul+H+Brookes+Publishing%2E)
9. Donald P Hayes and Margaret G Ahrens. 1988. Vocabulary simplification for children: A special case of ‘motherese’? Journal of child language, 15(2):395– 410. [find](https://scholar.google.com/scholar?q=Donald+P+Hayes+and+Margaret+G+Ahrens%2E+1988%2E+Vocabulary+simplification+for+children%3A+A+special+case+of+%E2%80%98motherese%E2%80%99%3F+Journal+of+child+language%2C+15%282%29%3A395%E2%80%93+410%2E)
10. Thea Cameron-Faulkner, Elena Lieven, and Michael Tomasello. 2003. A construction based analysis of child directed speech. Cogn. Sci., 27(6):843–873. [find](https://scholar.google.com/scholar?q=Thea+Cameron%2DFaulkner%2C+Elena+Lieven%2C+and+Michael+Tomasello%2E+2003%2E+A+construction+based+analysis+of+child+directed+speech%2E+Cogn%2E+Sci%2E%2C+27%286%29%3A843%E2%80%93873%2E)
11. Alex De Carvalho, Isabelle Dautriche, Isabelle Lin, and Anne Christophe. 2017. Phrasal prosody constrains syntactic analysis in toddlers. Cognition, 163:67– 79. [find](https://scholar.google.com/scholar?q=Alex+De+Carvalho%2C+Isabelle+Dautriche%2C+Isabelle+Lin%2C+and+Anne+Christophe%2E+2017%2E+Phrasal+prosody+constrains+syntactic+analysis+in+toddlers%2E+Cognition%2C+163%3A67%E2%80%93+79%2E)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
13. Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. [our card](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/)
14. Ruthe Foushee, Tom Griffiths, and Mahesh Srinivasan. 2016. Lexical complexity of child-directed and overheard speech: Implications for learning. In CogSci. [find](https://scholar.google.com/scholar?q=Ruthe+Foushee%2C+Tom+Griffiths%2C+and+Mahesh+Srinivasan%2E+2016%2E+Lexical+complexity+of+child%2Ddirected+and+overheard+speech%3A+Implications+for+learning%2E+In+CogSci%2E)
15. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), page 650–655. [our card](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/)
16. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Jon+Gauthier%2C+Peng+Qian%2C+Ethan+Wilcox%2C+and+Roger+Levy%2E+2020%2E+A+systematic+assessment+of+syntactic+generalization+in+neural+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+th)
17. Philip A Huebner and Jon A Willits. 2021. Using lexical context to discover the noun category: Younger children have it easier. In Psychology of learning and motivation, volume 75. Elsevier. [find](https://scholar.google.com/scholar?q=Philip+A+Huebner+and+Jon+A+Willits%2E+2021%2E+Using+lexical+context+to+discover+the+noun+category%3A+Younger+children+have+it+easier%2E+In+Psychology+of+learning+and+motivation%2C+volume+75%2E+Elsevier%2E)
18. Nenagh Kemp, Elena Lieven, and Michael Tomasello. 2005. Young children’s knowledge of the “determiner” and “adjective” categories. J. Speech Lang. Hear. Res., 48(3):592–609. [find](https://scholar.google.com/scholar?q=Nenagh+Kemp%2C+Elena+Lieven%2C+and+Michael+Tomasello%2E+2005%2E+Young+children%E2%80%99s+knowledge+of+the+%E2%80%9Cdeterminer%E2%80%9D+and+%E2%80%9Cadjective%E2%80%9D+categories%2E+J%2E+Speech+Lang%2E+Hear%2E+Res%2E%2C+48%283%29%3A592%E2%80%93609%2E)
19. Katrin Kirchhoff and Steven Schimmel. 2005. Statistical properties of infant-directed versus adult-directed speech: Insights from speech recognition. The Journal of the Acoustical Society of America, 117(4):2238–2246. [find](https://scholar.google.com/scholar?q=Katrin+Kirchhoff+and+Steven+Schimmel%2E+2005%2E+Statistical+properties+of+infant%2Ddirected+versus+adult%2Ddirected+speech%3A+Insights+from+speech+recognition%2E+The+Journal+of+the+Acoustical+Society+of+America%2C+)
20. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+J%2E+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+schema+challenge%2E+In+Proceedings+of+the+Thirteenth+International+Conference+on+Principles+of+Knowledge+Representation+and+Re)
21. Jeffrey Lidz and Laurel Perkins. 2018. Language acquisition. In Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience, pages 1– 49. Wiley. [find](https://scholar.google.com/scholar?q=Jeffrey+Lidz+and+Laurel+Perkins%2E+2018%2E+Language+acquisition%2E+In+Stevens%E2%80%99+Handbook+of+Experimental+Psychology+and+Cognitive+Neuroscience%2C+pages+1%E2%80%93+49%2E+Wiley%2E)
22. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020a. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
23. Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210– 5217, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/)
24. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020b. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)
25. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annu. Rev. Linguist., 7(1):195–212. [find](https://scholar.google.com/scholar?q=Tal+Linzen+and+Marco+Baroni%2E+2021%2E+Syntactic+structure+from+deep+learning%2E+Annu%2E+Rev%2E+Linguist%2E%2C+7%281%29%3A195%E2%80%93212%2E)
26. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
27. Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. 2021. Probing across time: What does roberta know and when? arXiv preprint arXiv:2104.07885. [find](https://scholar.google.com/scholar?q=Leo+Z+Liu%2C+Yizhong+Wang%2C+Jungo+Kasai%2C+Hannaneh+Hajishirzi%2C+and+Noah+A+Smith%2E+2021%2E+Probing+across+time%3A+What+does+roberta+know+and+when%3F+arXiv+preprint+arXiv%3A2104%2E07885%2E)
28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
29. Brian MacWhinney. 2000. The CHILDES Project: Tools for analyzing talk. transcription format and programs, volume 1. Psychology Press. [find](https://scholar.google.com/scholar?q=Brian+MacWhinney%2E+2000%2E+The+CHILDES+Project%3A+Tools+for+analyzing+talk%2E+transcription+format+and+programs%2C+volume+1%2E+Psychology+Press%2E)
30. Danielle Matthews and Colin Bannard. 2010. Children’s production of unfamiliar word sequences is predicted by positional variability and latent classes in a large sample of child-directed speech. Cognitive science, 34(3):465–488. [find](https://scholar.google.com/scholar?q=Danielle+Matthews+and+Colin+Bannard%2E+2010%2E+Children%E2%80%99s+production+of+unfamiliar+word+sequences+is+predicted+by+positional+variability+and+latent+classes+in+a+large+sample+of+child%2Ddirected+speech%2E+Cogn)
31. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140. [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2020-does-syntax-need-grow-trees/)
32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
33. Patricia A Reeder, Elissa L Newport, and Richard N Aslin. 2017. Distributional learning of subcategories in an artificial grammar: Category generalization and subcategory restrictions. Journal of memory and language, 97:17–29. [find](https://scholar.google.com/scholar?q=Patricia+A+Reeder%2C+Elissa+L+Newport%2C+and+Richard+N+Aslin%2E+2017%2E+Distributional+learning+of+subcategories+in+an+artificial+grammar%3A+Category+generalization+and+subcategory+restrictions%2E+Journal+of+memo)
34. Douglas Roland, Frederic Dick, and Jeffrey L Elman. 2007. Frequency of basic english grammatical structures: A corpus analysis. J. Mem. Lang., 57(3):348– 379. [find](https://scholar.google.com/scholar?q=Douglas+Roland%2C+Frederic+Dick%2C+and+Jeffrey+L+Elman%2E+2007%2E+Frequency+of+basic+english+grammatical+structures%3A+A+corpus+analysis%2E+J%2E+Mem%2E+Lang%2E%2C+57%283%29%3A348%E2%80%93+379%2E)
35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 371–376, Berlin, Germany. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/)
36. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736. [find](https://scholar.google.com/scholar?q=Ke+Tran%2C+Arianna+Bisazza%2C+and+Christof+Monz%2E+2018%2E+The+importance+of+being+recurrent+for+modeling+hierarchical+structure%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language)
37. Virginia Valian. 1986. Syntactic categories in the speech of young children. Dev. Psychol., 22(4):562– 579. [find](https://scholar.google.com/scholar?q=Virginia+Valian%2E+1986%2E+Syntactic+categories+in+the+speech+of+young+children%2E+Dev%2E+Psychol%2E%2C+22%284%29%3A562%E2%80%93+579%2E)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
39. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Proceedings of NeurIPS. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
40. Alex Warstadt and Samuel R Bowman. 2020. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society. [find](https://scholar.google.com/scholar?q=Alex+Warstadt+and+Samuel+R+Bowman%2E+2020%2E+Can+neural+networks+acquire+a+structural+bias+from+raw+linguistic+data%3F+In+Proceedings+of+the+42nd+Annual+Meeting+of+the+Cognitive+Science+Society%2E)
41. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo-nananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020a. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8(0):377–392. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mo%2Dnananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R+Bowman%2E+2020a%2E+BLiMP%3A+The+benchmark+of+linguistic+minimal+pairs+for+English%2E+Transactions+of+the+Asso)
42. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217–235… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Yian+Zhang%2C+Xiaocheng+Li%2C+Haokun+Liu%2C+and+Samuel+R+Bowman%2E+2020b%2E+Learning+which+features+matter%3A+RoBERTa+acquires+a+preference+for+linguistic+generalizations+%28eventually%29%2E+In+Proceedin)
43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
44. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-problems-current-text-simplification-research/)
45. Karolina Zaczynska, Nils Feldhus, Robert Schwarzen-berg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating German transformer language models with syntactic agreement tests. In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing, S… [find](https://scholar.google.com/scholar?q=Karolina+Zaczynska%2C+Nils+Feldhus%2C+Robert+Schwarzen%2Dberg%2C+Aleksandra+Gabryszak%2C+and+Sebastian+M%C3%B6ller%2E+2020%2E+Evaluating+German+transformer+language+models+with+syntactic+agreement+tests%2E+In+Proceedings+)
46. Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R Bowman. 2021. When do you need billions of words of pretraining data? In "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Haau%2DSing+Li%2C+and+Samuel+R+Bowman%2E+2021%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F+In+%22Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computatio)