---
title: "Batch-wise Convergent Pre-training — Yoshida 2025"
description: "The authors attempt to narrow the gap between large language models, which require enormous corpora, and children, who acquire language from a far…"
authors: ["Ko Yoshida", "Daiki Shiono", "Kai Sato", "Toko Miura", "Momoka Furuhashi", "Jun Suzuki"]
year: 2025
venue: "Proceedings of the First BabyLM Workshop"
source: "https://aclanthology.org/2025.babylm-main.36/"
doi: "10.18653/v1/2025.babylm-main.36"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yoshida-2025-batch-wise-convergent-pre-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yoshida-2025-batch-wise-convergent-pre-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Batch-wise Convergent Pre-training: Step-by-Step Learning Inspired by Child Language Development

# Batch-wise Convergent Pre-training: Step-by-Step Learning Inspired by Child Language Development

**Authors:** Ko Yoshida, Daiki Shiono, Kai Sato, Toko Miura, Momoka Furuhashi, [Jun Suzuki](https://md.research.elamur.ai/free/en/authors/jun-suzuki/) · **Year:** 2025 · **Venue:** Proceedings of the First BabyLM Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.babylm-main.36/) · [PDF](https://aclanthology.org/2025.babylm-main.36.pdf) · [DOI 10.18653/v1/2025.babylm-main.36](https://doi.org/10.18653/v1/2025.babylm-main.36)

**Citations:** 1 in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors attempt to narrow the gap between large language models, which require enormous corpora, and children, who acquire language from a far smaller number of words. The work was carried out within the BabyLM Challenge 2025. Two combined approaches are proposed. The first is a curriculum that reproduces the stages of a child's language development: first two- and three-word utterances from CHILDES, then longer utterances, then short LLM-generated sentences, and finally narratives and dialogues. The second is a constrained optimization algorithm. The model is trained on the same batch until the loss falls below a set threshold. At the same time, a separate term restrains the deviation of the weights from their previous state so that the model does not forget what it has already learned. Checkpoint averaging is added on top of this. The method did not outperform the official baseline models, especially on grammatical benchmarks such as BLiMP. However, on WUG-ADJ and Entity Tracking it yields a small but consistent gain. Thus, this approach affects different aspects of language in different ways. The authors themselves believe that isolated training on individual batches narrows the diversity of contexts needed for syntactic generalization.

## Key points
- The training data were collected from four sources: short and medium-length (4–10 words) utterances from CHILDES, synthetic sentences from Qwen3-14B based on words from the AoA list (Kuperman et al.), and processed texts from KidLM, TinyStories and TinyDialogues. The last of these make up approximately 80 % of the words in the corpus.
- The curriculum has two dimensions. Within a batch, the proportions of the sources change smoothly over the course of training. Across batches, examples are ordered by a difficulty score.
- Models of two sizes were tested: 115M and 372M. For 115M, the average score of the proposed method is 32.5–32.8 versus 30–31 for the control variants, which differ only in data order and repetitions. For 372M, the average score is about 33.0, which does not go beyond the spread of the baseline models.
- In the ablation, the curriculum was compared with fully shuffled data. The difference in the average score did not exceed one point. The order of data presentation helps slightly on WUG-ADJ and COMPS but worsens the result on BLiMP.
- As a direction for future work, the authors propose architectures with memory or with context aggregation across batches.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yoshida2025batch,
  title = {{Batch-wise Convergent Pre-training: Step-by-Step Learning Inspired by Child Language Development}},
  author = {Yoshida, Ko and Shiono, Daiki and Sato, Kai and Miura, Toko and Furuhashi, Momoka and Suzuki, Jun},
  booktitle = {Proceedings of the First BabyLM Workshop},
  year = {2025},
  doi = {10.18653/v1/2025.babylm-main.36},
  url = {https://doi.org/10.18653/v1/2025.babylm-main.36}
}
```

- [← previous Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs](https://md.research.elamur.ai/free/en/papers/aynetdinov-2025-babies-learn-look-ahead-multi/)
- [next → Benchmarking the Medical Understanding and Reasoning of Large Language Models in Arabic Healthcare Tasks](https://md.research.elamur.ai/free/en/papers/aldahoul-2025-benchmarking-medical-understanding-reasoning-large/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rufus Behr. 2024. ELC-ParserBERT: Low-resource language modeling utilizing a parser network with ELC-BERT. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 140–146, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rufus+Behr%2E+2024%2E+ELC%2DParserBERT%3A+Low%2Dresource+language+modeling+utilizing+a+parser+network+with+ELC%2DBERT%2E+In+The+2nd+BabyLM+Challenge+at+the+28th+Conference+on+Computational+Natural+Language+Learning)
2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+J%C3%A9r%C3%B4me+Louradour%2C+Ronan+Collobert%2C+and+Jason+Weston%2E+2009%2E+Curriculum+learning%2E+In+Proceedings+of+the+26th+Annual+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9909%2C+page+41%E2%80%9348%2C+New) [our card](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/)
3. Arielle Borovsky, Jeffrey L Elman, and Anne Fernald. 2012. Knowing a lot for one’s age: Vocabulary skill and not age is associated with anticipatory incremental sentence interpretation in children and adults. Journal of experimental child psychology, 112(4):417–436. [find](https://scholar.google.com/scholar?q=Arielle+Borovsky%2C+Jeffrey+L+Elman%2C+and+Anne+Fernald%2E+2012%2E+Knowing+a+lot+for+one%E2%80%99s+age%3A+Vocabulary+skill+and+not+age+is+associated+with+anticipatory+incremental+sentence+interpretation+in+children+and)
4. Luca Capone, Alessandro Bondielli, and Alessandro Lenci. 2024. ConcreteGPT: A baby GPT-2 based on lexical concreteness and curriculum learning. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 189–196, Miami, FL, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Luca+Capone%2C+Alessandro+Bondielli%2C+and+Alessandro+Lenci%2E+2024%2E+ConcreteGPT%3A+A+baby+GPT%2D2+based+on+lexical+concreteness+and+curriculum+learning%2E+In+The+2nd+BabyLM+Challenge+at+the+28th+Conference+on+Co)
5. Tyler A. Chang and Benjamin K. Bergen. 2022. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics, 10:1–16. [find](https://scholar.google.com/scholar?q=Tyler+A%2E+Chang+and+Benjamin+K%2E+Bergen%2E+2022%2E+Word+acquisition+in+neural+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+10%3A1%E2%80%9316%2E)
6. Lucas Georges Gabriel Charpentier and David Samuel. 2024. GPT or BERT: why not both? In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 262– 283, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lucas+Georges+Gabriel+Charpentier+and+David+Samuel%2E+2024%2E+GPT+or+BERT%3A+why+not+both%3F+In+The+2nd+BabyLM+Challenge+at+the+28th+Conference+on+Computational+Natural+Language+Learning%2C+pages+262%E2%80%93+283%2C+Miam)
7. Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870–7881, Online. Associa… [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yutai+Hou%2C+Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+and+Xiangzhan+Yu%2E+2020%2E+Recall+and+learn%3A+Fine%2Dtuning+deep+pretrained+language+models+with+less+forgetting%2E+In+Proceedings+of+the+2020+Conf)
8. Leshem Choshen, Ryan Cotterell, Michael Y. Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. [Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. arXiv preprint arXiv:2404.062… [find](https://scholar.google.com/scholar?q=Leshem+Choshen%2C+Ryan+Cotterell%2C+Michael+Y%2E+Hu%2C+Tal+Linzen%2C+Aaron+Mueller%2C+Candace+Ross%2C+Alex+Warstadt%2C+Ethan+Wilcox%2C+Adina+Williams%2C+and+Chengxu+Zhuang%2E+2024%2E+%5BCall+for+Papers%5D+The+2nd+BabyLM+Challeng)
9. Gina Conti-Ramsden and Kevin Durkin. 2012. Language development and assessment in the preschool period. Neuropsychology Review, 22(4):384–401. [find](https://scholar.google.com/scholar?q=Gina+Conti%2DRamsden+and+Kevin+Durkin%2E+2012%2E+Language+development+and+assessment+in+the+preschool+period%2E+Neuropsychology+Review%2C+22%284%29%3A384%E2%80%93401%2E)
10. Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. 2023. Cloze probability, predictability ratings, and computational estimates for 205 english sentences, aligned with existing eeg and reading time data. Behavior Research Methods, 56:5190 – 5213. [find](https://scholar.google.com/scholar?q=Andrea+Gregor+de+Varda%2C+Marco+Marelli%2C+and+Simona+Amenta%2E+2023%2E+Cloze+probability%2C+predictability+ratings%2C+and+computational+estimates+for+205+english+sentences%2C+aligned+with+existing+eeg+and+reading+)
11. Emmanuel Dupoux. 2018. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43–59. [find](https://scholar.google.com/scholar?q=Emmanuel+Dupoux%2E+2018%2E+Cognitive+science+in+the+era+of+artificial+intelligence%3A+A+roadmap+for+reverse%2Dengineering+the+infant+language%2Dlearner%2E+Cognition%2C+173%3A43%E2%80%9359%2E)
12. Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? Preprint, arXiv:2305.07759. [find](https://scholar.google.com/scholar?q=Ronen+Eldan+and+Yuanzhi+Li%2E+2023%2E+Tinystories%3A+How+small+can+language+models+be+and+still+speak+coherent+english%3F+Preprint%2C+arXiv%3A2305%2E07759%2E) [our card](https://md.research.elamur.ai/free/en/papers/eldan-2023-tinystories-small-language-models-still/)
13. Steven Y. Feng, Noah D. Goodman, and Michael C. Frank. 2024. Is child-directed speech effective training data for language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 22055–22071, Miami, Florida, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Steven+Y%2E+Feng%2C+Noah+D%2E+Goodman%2C+and+Michael+C%2E+Frank%2E+2024%2E+Is+child%2Ddirected+speech+effective+training+data+for+language+models%3F+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural) [our card](https://md.research.elamur.ai/free/en/papers/feng-2024-is-child-directed-speech-effective/)
14. Robert M. French. 1999. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135. [find](https://scholar.google.com/scholar?q=Robert+M%2E+French%2E+1999%2E+Catastrophic+forgetting+in+connectionist+networks%2E+Trends+in+Cognitive+Sciences%2C+3%284%29%3A128%E2%80%93135%2E)
15. Mohammad Amin Ghanizadeh and Mohammad Javad Dousti. 2024. Towards data-efficient language models: A child-inspired approach to language learning. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 22–27, Miami, FL, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Mohammad+Amin+Ghanizadeh+and+Mohammad+Javad+Dousti%2E+2024%2E+Towards+data%2Defficient+language+models%3A+A+child%2Dinspired+approach+to+language+learning%2E+In+The+2nd+BabyLM+Challenge+at+the+28th+Conference+on+)
16. Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Judith K. Montgomery, Charles R. Greenwood, D. Kim-brough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American Journal of Speech-Language Pathology… [find](https://scholar.google.com/scholar?q=Jill+Gilkerson%2C+Jeffrey+A%2E+Richards%2C+Steven+F%2E+Warren%2C+Judith+K%2E+Montgomery%2C+Charles+R%2E+Greenwood%2C+D%2E+Kim%2Dbrough+Oller%2C+John+H%2E+L%2E+Hansen%2C+and+Terrance+D%2E+Paul%2E+2017%2E+Mapping+the+early+language+enviro)
17. Akari Haga, Akiyo Fukatsu, Miyu Oba, Arianna Bisazza, and Yohei Oseki. 2024. BabyLM challenge: Exploring the effect of variation sets on language model training efficiency. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 252–261, Miami, FL, USA. A… [find](https://scholar.google.com/scholar?q=Akari+Haga%2C+Akiyo+Fukatsu%2C+Miyu+Oba%2C+Arianna+Bisazza%2C+and+Yohei+Oseki%2E+2024%2E+BabyLM+challenge%3A+Exploring+the+effect+of+variation+sets+on+language+model+training+efficiency%2E+In+The+2nd+BabyLM+Challenge)
18. Valentin Hofmann, Leonie Weissweiler, David R. Mortensen, Hinrich Schütze, and Janet B. Pierrehumbert. 2025. Derivational morphology reveals analogical generalization in large language models. Proceedings of the National Academy of Sciences, 122(19):e2423232122. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Leonie+Weissweiler%2C+David+R%2E+Mortensen%2C+Hinrich+Sch%C3%BCtze%2C+and+Janet+B%2E+Pierrehumbert%2E+2025%2E+Derivational+morphology+reveals+analogical+generalization+in+large+language+models%2E+Proceed)
19. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In The 2nd BabyLM Challen… [find](https://scholar.google.com/scholar?q=Michael+Y%2E+Hu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Adina+Williams%2C+Tal+Linzen%2C+Chengxu+Zhuang%2C+Ryan+Cotterell%2C+Leshem+Choshen%2C+Alex+Warstadt%2C+and+Ethan+Gotlieb+Wilcox%2E+2024%2E+Findings+of+the+second+BabyLM+cha)
20. Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Un-nathi U. Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian C. Paulun, Maria Ryskina, Ekin Akyürek, Ethan G. Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum,… [find](https://scholar.google.com/scholar?q=Anna+A%2E+Ivanova%2C+Aalok+Sathe%2C+Benjamin+Lipkin%2C+Un%2Dnathi+U%2E+Kumar%2C+Setayesh+Radkani%2C+Thomas+H%2E+Clark%2C+Carina+Kauf%2C+Jennifer+Hu%2C+R%2E+T%2E+Pramod%2C+Gabriel+Grand%2C+Vivian+C%2E+Paulun%2C+Maria+Ryskina%2C+Ekin+Aky%C3%BCre)
21. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+2020%2E+Scaling+laws+for+neural+language+models%2E+Preprint%2C+) [our card](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)
22. Najoung Kim and Sebastian Schuster. 2023. Entity tracking in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3835–3855, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Sebastian+Schuster%2E+2023%2E+Entity+tracking+in+language+models%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+)
23. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
24. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
25. Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 english words. Behavior Research Methods, 44(4):978–990. [find](https://scholar.google.com/scholar?q=Victor+Kuperman%2C+Hans+Stadthagen%2DGonzalez%2C+and+Marc+Brysbaert%2E+2012%2E+Age%2Dof%2Dacquisition+ratings+for+30%2C000+english+words%2E+Behavior+Research+Methods%2C+44%284%29%3A978%E2%80%93990%2E)
26. David Lopez-Paz and Marc' Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=David+Lopez%2DPaz+and+Marc%27+Aurelio+Ranzato%2E+2017%2E+Gradient+episodic+memory+for+continual+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+30%2E+Curran+Associates%2C+Inc%2E)
27. Brian Macwhinney. 2000. The childes project: tools for analyzing talk. Child Language Teaching and Therapy, 8. [find](https://scholar.google.com/scholar?q=Brian+Macwhinney%2E+2000%2E+The+childes+project%3A+tools+for+analyzing+talk%2E+Child+Language+Teaching+and+Therapy%2C+8%2E)
28. Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual minimal pair sentences for testing robust property knowledge and its inheritance in pre-trained language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, p… [find](https://scholar.google.com/scholar?q=Kanishka+Misra%2C+Julia+Rayz%2C+and+Allyson+Ettinger%2E+2023%2E+COMPS%3A+Conceptual+minimal+pair+sentences+for+testing+robust+property+knowledge+and+its+inheritance+in+pre%2Dtrained+language+models%2E+In+Proceeding)
29. Mir Tafseer Nayeem and Davood Rafiei. 2024. KidLM: Advancing language models for children – early insights and future directions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4813–4836, Miami, Florida, USA. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Mir+Tafseer+Nayeem+and+Davood+Rafiei%2E+2024%2E+KidLM%3A+Advancing+language+models+for+children+%E2%80%93+early+insights+and+future+directions%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Methods+in+Natural+)
30. German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71. [find](https://scholar.google.com/scholar?q=German+I%2E+Parisi%2C+Ronald+Kemker%2C+Jose+L%2E+Part%2C+Christopher+Kanan%2C+and+Stefan+Wermter%2E+2019%2E+Continual+lifelong+learning+with+neural+networks%3A+A+review%2E+Neural+Networks%2C+113%3A54%E2%80%9371%2E) [our card](https://md.research.elamur.ai/free/en/papers/parisi-2018-continual-lifelong-learning-neural-networks/)
31. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
32. Tracy Reuter, Arielle Borovsky, and Casey Lew-Williams. 2019. Predict and redirect: Prediction errors support children’s word learning. Developmental psychology, 55(8):1656. [find](https://scholar.google.com/scholar?q=Tracy+Reuter%2C+Arielle+Borovsky%2C+and+Casey+Lew%2DWilliams%2E+2019%2E+Predict+and+redirect%3A+Prediction+errors+support+children%E2%80%99s+word+learning%2E+Developmental+psychology%2C+55%288%29%3A1656%2E)
33. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2022. Progressive neural networks. Preprint, arXiv:1606.04671. [find](https://scholar.google.com/scholar?q=Andrei+A%2E+Rusu%2C+Neil+C%2E+Rabinowitz%2C+Guillaume+Desjardins%2C+Hubert+Soyer%2C+James+Kirkpatrick%2C+Koray+Kavukcuoglu%2C+Razvan+Pascanu%2C+and+Raia+Hadsell%2E+2022%2E+Progressive+neural+networks%2E+Preprint%2C+arXiv%3A1606%2E)
34. Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, and Paula Buttery. 2024. Less is more: Pre-training cross-lingual small-scale language models with cognitively-plausible curriculum learning strategies. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learni… [find](https://scholar.google.com/scholar?q=Suchir+Salhan%2C+Richard+Diehl+Martinez%2C+Z%C3%A9bulon+Goriely%2C+and+Paula+Buttery%2E+2024%2E+Less+is+more%3A+Pre%2Dtraining+cross%2Dlingual+small%2Dscale+language+models+with+cognitively%2Dplausible+curriculum+learning+str)
35. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Superglue: Learning feature matching with graph neural networks. Preprint, arXiv:1911.11763. [find](https://scholar.google.com/scholar?q=Paul%2DEdouard+Sarlin%2C+Daniel+DeTone%2C+Tomasz+Malisiewicz%2C+and+Andrew+Rabinovich%2E+2020%2E+Superglue%3A+Learning+feature+matching+with+graph+neural+networks%2E+Preprint%2C+arXiv%3A1911%2E11763%2E)
36. Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020. {LAMAL}: {LA}nguage modeling is all you need for lifelong language learning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Fan%2DKeng+Sun%2C+Cheng%2DHao+Ho%2C+and+Hung%2DYi+Lee%2E+2020%2E+%7BLAMAL%7D%3A+%7BLA%7Dnguage+modeling+is+all+you+need+for+lifelong+language+learning%2E+In+International+Conference+on+Learning+Representations%2E)
37. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
38. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mos%2Dquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+Ryan+Cotterell%2E+2023%2E+Findings+of+the+BabyL)
39. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+BLiMP%3A+The+benchmark+of+linguistic+minimal+pairs+for+English%2E+Transactions+of+the+Assoc)
40. Leonie Weissweiler, Valentin Hofmann, Anjali Kantharuban, Anna Cai, Ritam Dutt, Amey Hengle, Anubha Kabra, Atharva Kulkarni, Abhishek Vijayakumar, Haofei Yu, Hinrich Schuetze, Kemal Oflazer, and David Mortensen. 2023. Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphol… [find](https://scholar.google.com/scholar?q=Leonie+Weissweiler%2C+Valentin+Hofmann%2C+Anjali+Kantharuban%2C+Anna+Cai%2C+Ritam+Dutt%2C+Amey+Hengle%2C+Anubha+Kabra%2C+Atharva+Kulkarni%2C+Abhishek+Vijayakumar%2C+Haofei+Yu%2C+Hinrich+Schuetze%2C+Kemal+Oflazer%2C+and+David) [our card](https://md.research.elamur.ai/free/en/papers/lex-2023-weissweiler-chatgpt-wugs-morphology/)
41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Anfeng+Li%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Gao%2C+Chengen+Huang%2C+Chenxu+Lv%2C+Chujie+Zheng%2C+Dayiheng+Liu%2C+Fan+Zhou%2C+Fei+Huang%2C+Feng+Hu%2C+Hao+Ge%2C+Haoran+Wei%2C+Hua) [our card](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)
42. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXi… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
43. Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3987– 3995. PMLR. [find](https://scholar.google.com/scholar?q=Friedemann+Zenke%2C+Ben+Poole%2C+and+Surya+Ganguli%2E+2017%2E+Continual+learning+through+synaptic+intelligence%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+volume+70+of+Proceeding)
44. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Xiaocheng+Li%2C+and+Samuel+R%2E+Bowman%2E+2021%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computatio)