--- title: "Using Curriculum Masking Based on Child Language Development to Train a Large Language — Lucas 2024" description: "The work was produced as part of the BabyLM Challenge 2024. 4 references, cited by 2 papers here." authors: ["Evan Lucas", "Dylan Gaines", "Tagore Rao Kosireddy", "Kevin Li", "Timothy C. Havens"] year: 2024 venue: "The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning" source: "https://aclanthology.org/2024.conll-babylm.19/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lucas-2024-curriculum-masking-based-child-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lucas-2024-curriculum-masking-based-child-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data # Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data **Authors:** Evan Lucas, Dylan Gaines, Tagore Rao Kosireddy, Kevin Li and Timothy C. Havens · **Year:** 2024 · **Venue:** The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2024.conll-babylm.19/) · [PDF](https://aclanthology.org/2024.conll-babylm.19.pdf) **Citations:** 2 in our library, of the 140,884 papers whose bibliographies we read ## Summary The work was produced as part of the BabyLM Challenge 2024. The goal of the challenge is to train a language model on the amount of text a child hears during the first years of life rather than on trillions of words. The authors from Michigan Tech submitted models to two tracks: STRICT (100M words) and STRICT-SMALL (10M words). The first contribution is a new pretraining corpus drawn from the "Explain Like I'm Five" (ELI5) subreddit, where users explain complex topics in simple language. The second contribution is "curriculum masking", a modification of MLM in which the model learns to predict not random words but first nouns and then progressively more complex parts of speech. The order of the parts of speech is taken from the typical sequence of a child's language development. Among the 10M models, the combination of ELI5 and the new masking gave the best result, and in the STRICT-SMALL track it outperformed the organizers' baseline. In the 100M track, the authors did not manage to beat the baseline. For small corpora, the method is of interest because it speeds up training and potentially makes it possible to use fewer epochs. ## Key points - **ELI5 corpus:** only top-level comments from June 2005 – December 2022 were taken from the The-Eye.eu archive. They were sorted by vote score, and posts containing links or words from a short list of 28 profane words were discarded. From this corpus, 10M-word and 100M-word sets were built, with the smaller one fully contained in the larger. - **Curriculum:** 7 cumulative stages. First come interjections, nouns and personal pronouns, then conjunctions, simple verb constructions, adjectives, complex verbs, adverbs, and finally all remaining parts of speech. Part-of-speech (POS) tags are assigned by NLTK. Up to 15% of words are masked, selected only from the categories of the current stage. - **Model:** BERT with 6 layers, a hidden size of 768 and 12 attention heads, 51.2M parameters in total. The BPE tokenizer has a vocabulary of 10K tokens for the 10M corpus and 50K for the 100M corpus. Total compute is estimated at roughly 30 GPU-days. - **BLiMP:** among the 10M models, the BERT baseline scored 54.6, ELI5 54.7, ELI5 with curriculum masking on updated hyperparameters 55.6, and the organizers' LTG-BERT 60.6. Among the 100M models, ELI5 scored 55.4, ELI5 with curriculum masking 60.2, and LTG-BERT 69.2. Thus, on BLiMP all of the authors' models fell short of the baseline, and their advantage came from other benchmarks, notably EWoK. - **Limitations:** the authors tried only two sets of hyperparameters, did not investigate the masking rate, and did not control for batch size. The quality of the POS tagger on their data was not evaluated. The code is published on GitHub: `evan-person/curriculumMaskingBabyLM2024`. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🧒 Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) ## Cite Cite as: Evan Lucas, Dylan Gaines, Tagore Rao Kosireddy, Kevin Li and Timothy C. Havens. 2024. Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data. The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lucas2024using, title = {{Using Curriculum Masking Based on Child Language Development to Train a Large Language Model with Limited Training Data}}, author = {Evan Lucas, Dylan Gaines, Tagore Rao Kosireddy, Kevin Li and Timothy C. Havens}, booktitle = {The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning}, year = {2024}, url = {https://aclanthology.org/2024.conll-babylm.19/} } ``` - [← previous UOM-Constrained IWSLT 2024 Shared Task Submission - Maltese Speech Translation](https://md.research.elamur.ai/free/en/papers/abela-2024-uom-constrained-iwslt-2024-shared/) - [next → Using the Book "Al-Miftah lil-‘Ulum" in Teaching Arabic Grammar at the Islamic Boarding School in Indonesia: Benefits and Obstacles](https://md.research.elamur.ai/free/en/papers/zakiya-2024-using-book-al-miftah-lil/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) ## References 4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. 2024. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834. [find](https://scholar.google.com/scholar?q=Marco+Bellagente%2C+Jonathan+Tow%2C+Dakota+Mahan%2C+Duy+Phung%2C+Maksym+Zhuravinskyi%2C+Reshinth+Adithyan%2C+James+Baicoianu%2C+Ben+Brooks%2C+Nathan+Cooper%2C+Ashish+Datta%2C+et+al%2E+2024%2E+Stable+lm+2+1%2E6+b+technical+repo) 2. Celeste Roseberry-McKibbin and Mahabalagiri N Hegde. 2006. An advanced review of speech-language pathology: preparation for PRAXIS and comprehensive examination. ERIC. [find](https://scholar.google.com/scholar?q=Celeste+Roseberry%2DMcKibbin+and+Mahabalagiri+N+Hegde%2E+2006%2E+An+advanced+review+of+speech%2Dlanguage+pathology%3A+preparation+for+PRAXIS+and+comprehensive+examination%2E+ERIC%2E) 3. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Chal… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mos%2Dquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+et+al%2E+2023%2E+Findings+of+the+babylm+challenge%3A+) 4. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+2024%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretrai)