---
title: "Babies Learn to Look Ahead — Aynetdinov 2025"
description: "The paper tests whether multi-token prediction (MTP) helps very small language models."
authors: ["Ansar Aynetdinov", "Alan Akbik"]
year: 2025
venue: "Proceedings of the First BabyLM Workshop"
source: "https://aclanthology.org/2025.babylm-main.41/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/aynetdinov-2025-babies-learn-look-ahead-multi/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/aynetdinov-2025-babies-learn-look-ahead-multi/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/) / Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs

# Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs

**Authors:** Ansar Aynetdinov, [Alan Akbik](https://md.research.elamur.ai/free/authors/alan-akbik/) · **Year:** 2025 · **Venue:** Proceedings of the First BabyLM Workshop · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.babylm-main.41/) · [PDF](https://aclanthology.org/2025.babylm-main.41.pdf)

**Citations:** 1 in our library, of the 140673 papers whose bibliographies we read

## Summary

The paper tests whether multi-token prediction (MTP) helps very small language models. In MTP, the model guesses several next tokens at each step rather than just one. It was previously believed that mainly large models benefit from such an objective, while it may harm small ones. The authors took a 130M-parameter GPT-2 and trained it for 10 epochs on the BabyLM strict-small corpus of 10M words. They compared four variants: standard NTP, static MTP with two tokens, and a "forward" and a "reverse" curriculum. The forward curriculum first trains with NTP and adds the second token later; the reverse one does the opposite. At evaluation time, all models predict only one token. The reverse curriculum performed best. It outperformed NTP on all zero-shot tests except BLiMP, and pulled furthest ahead in entity tracking. The forward curriculum gave the weakest result. The authors explain this by MTP acting as a regularizer early in training: on a small corpus that is passed over many times, the model more slowly loses the non-local regularities it has learned. Thus, even tiny models in data-scarce settings can benefit from MTP, and inference speed does not drop.

## Key points

- **Setup:** a 130M-parameter decoder-only GPT-2 and the BabyLM strict-small corpus (10M words). The model has at most two prediction heads. The additional linear heads add 8M or 16M parameters, depending on the vocabulary size (8K or 16K).
- **Average zero-shot score:** reverse curriculum — 38.54, static MTP — 37.99, NTP — 37.06, forward curriculum — 36.73.
- **Entity Tracking:** NTP scores 13.74, static MTP — 17.88, reverse curriculum — 18.60. On BLiMP, however, NTP has the highest score (62.17), so it is this model that learns local grammar better.
- **Other tests:** on EWoK, all models stay close to chance level (~50). The authors believe the BabyLM data contains too little factual knowledge. The Eye Tracking and Self-paced Reading metrics are slightly better for static MTP and the reverse curriculum.
- **SuperGLUE after fine-tuning:** the difference between the variants is barely noticeable, with average scores in the range 63.48–63.85. A smaller vocabulary gave no advantage to either objective.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aynetdinov2025babies,
  title = {{Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs}},
  author = {Aynetdinov, Ansar and Akbik, Alan},
  booktitle = {Proceedings of the First BabyLM Workshop},
  year = {2025},
  url = {https://aclanthology.org/2025.babylm-main.41/}
}
```

- [← previous Automatic Text Segmentation of Ancient and Historic Hebrew](https://md.research.elamur.ai/free/papers/aclbulk-2025-rosensweig-automatic-text-segmentation-ancient-historic/)
- [next → Benchmarking LLMs for Translating Classical Chinese Poetry: Evaluating Adequacy, Fluency, and Elegance](https://md.research.elamur.ai/free/papers/chen-2025-benchmarking-llms-translating-classical-chinese/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/)

## References

37 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ansar Aynetdinov and Alan Akbik. 2025. Pre-training curriculum for multi-token prediction in language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25573–25588, Vienna, Austria. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Ansar+Aynetdinov+and+Alan+Akbik%2E+2025%2E+Pre%2Dtraining+curriculum+for+multi%2Dtoken+prediction+in+language+models%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics)
2. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, page 41–48, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+J%C3%A9r%C3%B4me+Louradour%2C+Ronan+Collobert%2C+and+Jason+Weston%2E+2009%2E+Curriculum+learning%2E+In+Proceedings+of+the+26th+Annual+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9909%2C+page+41%E2%80%9348%2C+New) [our card](https://md.research.elamur.ai/free/papers/bengio-2009-curriculum-learning/)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
4. Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Fer… [find](https://scholar.google.com/scholar?q=Hinsvark%2C+Arun+Rao%2C+Aston+Zhang%2C+Aurelien+Rodriguez%2C+Austen+Gregerson%2C+Ava+Spataru%2C+Baptiste+Roziere%2C+Bethany+Biron%2C+Binh+Tang%2C+Bobbie+Chern%2C+Charlotte+Caucheteux%2C+Chaya+Nayak%2C+Chloe+Bi%2C+Chris+Marra%2C+)
5. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Che… [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+Xiaokang+Zhang%2C+Xingkai+Yu%2C+Yu+Wu%2C+Z%2E+F%2E+Wu%2C+Zhibin+Gou%2C+Zhihong+Shao%2C+Zhuoshu+Li) [our card](https://md.research.elamur.ai/free/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
6. Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. Preprint, arXiv:2401.10774. [find](https://scholar.google.com/scholar?q=Tianle+Cai%2C+Yuhong+Li%2C+Zhengyang+Geng%2C+Hongwu+Peng%2C+Jason+D%2E+Lee%2C+Deming+Chen%2C+and+Tri+Dao%2E+2024%2E+Medusa%3A+Simple+llm+inference+acceleration+framework+with+multiple+decoding+heads%2E+Preprint%2C+arXiv%3A2401) [our card](https://md.research.elamur.ai/free/papers/cai-2024-medusa-simple-llm-inference-acceleration/)
7. Valentin Hofmann, Leonie Weissweiler, David Mortensen, Hinrich Schütze, and Janet Pierrehumbert. 2024. Derivational morphology reveals analogical generalization in large language models. Preprint, arXiv:2411.07990. [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Leonie+Weissweiler%2C+David+Mortensen%2C+Hinrich+Sch%C3%BCtze%2C+and+Janet+Pierrehumbert%2E+2024%2E+Derivational+morphology+reveals+analogical+generalization+in+large+language+models%2E+Preprint%2C+arX)
8. Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Hu, Jaap Jumelet, Tal Linzen, Jing Liu, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Wilcox, and Adina Williams. 2025. Babylm turns 3: Call for papers for the 2025 babylm workshop. Preprint, arXiv:2502… [find](https://scholar.google.com/scholar?q=Lucas+Charpentier%2C+Leshem+Choshen%2C+Ryan+Cotterell%2C+Mustafa+Omer+Gul%2C+Michael+Hu%2C+Jaap+Jumelet%2C+Tal+Linzen%2C+Jing+Liu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Raj+Sanjay+Shah%2C+Alex+Warstadt%2C+Ethan+Wilcox%2C+and+Adin)
9. Xudong Hong, Sharid Loáiciga, and Asad Sayeed. 2024. A surprisal oracle for when every layer counts. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 237–243, Miami, FL, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xudong+Hong%2C+Sharid+Lo%C3%A1iciga%2C+and+Asad+Sayeed%2E+2024%2E+A+surprisal+oracle+for+when+every+layer+counts%2E+In+The+2nd+BabyLM+Challenge+at+the+28th+Conference+on+Computational+Natural+Language+Learning%2C+page)
10. Andrea Gregor de Varda, Marco Marelli, and Simona Amenta. 2024. Cloze probability, predictability ratings, and computational estimates for 205 english sentences, aligned with existing eeg and reading time data. Behavior Research Methods, 56(5):5190–5213. [find](https://scholar.google.com/scholar?q=Andrea+Gregor+de+Varda%2C+Marco+Marelli%2C+and+Simona+Amenta%2E+2024%2E+Cloze+probability%2C+predictability+ratings%2C+and+computational+estimates+for+205+english+sentences%2C+aligned+with+existing+eeg+and+reading+)
11. Steven Feng, Shrimai Prabhumoye, Kezhi Kong, Dan Su, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Maximize your data’s potential: Enhancing llm accuracy with two-phase pre-training. Preprint, arXiv:2412.15285. [find](https://scholar.google.com/scholar?q=Steven+Feng%2C+Shrimai+Prabhumoye%2C+Kezhi+Kong%2C+Dan+Su%2C+Mostofa+Patwary%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+2024%2E+Maximize+your+data%E2%80%99s+potential%3A+Enhancing+llm+accuracy+with+two%2Dphase+pre%2Dtraining%2E+P)
12. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. Preprint, arXiv:2412.0514… [find](https://scholar.google.com/scholar?q=Michael+Y%2E+Hu%2C+Aaron+Mueller%2C+Candace+Ross%2C+Adina+Williams%2C+Tal+Linzen%2C+Chengxu+Zhuang%2C+Ryan+Cotterell%2C+Leshem+Choshen%2C+Alex+Warstadt%2C+and+Ethan+Gotlieb+Wilcox%2E+2024%2E+Findings+of+the+second+babylm+cha)
13. Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. 2024. Better & faster large language models via multi-token prediction. Preprint, arXiv:2404.19737. [find](https://scholar.google.com/scholar?q=Fabian+Gloeckle%2C+Badr+Youbi+Idrissi%2C+Baptiste+Rozi%C3%A8re%2C+David+Lopez%2DPaz%2C+and+Gabriel+Synnaeve%2E+2024%2E+Better+%26+faster+large+language+models+via+multi%2Dtoken+prediction%2E+Preprint%2C+arXiv%3A2404%2E19737%2E)
14. Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Un-nathi Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian Paulun, Maria Ryskina, Ekin Akyurek, Ethan Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum, and Jacob… [find](https://scholar.google.com/scholar?q=Anna+A%2E+Ivanova%2C+Aalok+Sathe%2C+Benjamin+Lipkin%2C+Un%2Dnathi+Kumar%2C+Setayesh+Radkani%2C+Thomas+H%2E+Clark%2C+Carina+Kauf%2C+Jennifer+Hu%2C+R%2E+T%2E+Pramod%2C+Gabriel+Grand%2C+Vivian+Paulun%2C+Maria+Ryskina%2C+Ekin+Akyurek%2C+Eth)
15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Najoung Kim and Seb… [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Alex+Vaughan%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anth)
16. Bruce W Lee, Hyunsoo Cho, and Kang Min Yoo. 2024. Instruction tuning with human curriculum. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1281–1309, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bruce+W+Lee%2C+Hyunsoo+Cho%2C+and+Kang+Min+Yoo%2E+2024%2E+Instruction+tuning+with+human+curriculum%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2024%2C+pages+1281%E2%80%931309%2C+Mexico+City%2C+Mexi)
17. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Hao… [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhang+Li%2C+Fangyun)
18. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
19. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. Preprint, arXiv:2306.02707. [find](https://scholar.google.com/scholar?q=Subhabrata+Mukherjee%2C+Arindam+Mitra%2C+Ganesh+Jawahar%2C+Sahaj+Agarwal%2C+Hamid+Palangi%2C+and+Ahmed+Awadallah%2E+2023%2E+Orca%3A+Progressive+learning+from+complex+explanation+traces+of+gpt%2D4%2E+Preprint%2C+arXiv%3A2306%2E) [our card](https://md.research.elamur.ai/free/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/)
20. Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2021. Pre-training a BERT with curriculum learning by increasing block-size of input text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 989–996, Held Online. INCOMA… [find](https://scholar.google.com/scholar?q=Koichi+Nagatsuka%2C+Clifford+Broni%2DBediako%2C+and+Masayasu+Atsumi%2E+2021%2E+Pre%2Dtraining+a+BERT+with+curriculum+learning+by+increasing+block%2Dsize+of+input+text%2E+In+Proceedings+of+the+International+Conference)
21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+Red+Avila%2C+Igor+Babuschkin%2C+Suchir) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
22. Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. 2023. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 548–560, Singapore. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Koyena+Pal%2C+Jiuding+Sun%2C+Andrew+Yuan%2C+Byron+Wallace%2C+and+David+Bau%2E+2023%2E+Future+lens%3A+Anticipating+subsequent+tokens+from+a+single+hidden+state%2E+In+Proceedings+of+the+27th+Conference+on+Computational)
23. Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. 2025. Diffusion beats autoregressive in data-constrained settings. Preprint, arXiv:2507.15857. — [arXiv:2507.15857](https://arxiv.org/abs/2507.15857) [find](https://scholar.google.com/scholar?q=Mihir+Prabhudesai%2C+Mengning+Wu%2C+Amir+Zadeh%2C+Katerina+Fragkiadaki%2C+and+Deepak+Pathak%2E+2025%2E+Diffusion+beats+autoregressive+in+data%2Dconstrained+settings%2E+Preprint%2C+arXiv%3A2507%2E15857%2E+%E2%80%94+%5BarXiv%3A2507%2E15857%5D)
24. Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-SequencePre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410, Online. Association for Comp… [find](https://scholar.google.com/scholar?q=Weizhen+Qi%2C+Yu+Yan%2C+Yeyun+Gong%2C+Dayiheng+Liu%2C+Nan+Duan%2C+Jiusheng+Chen%2C+Ruofei+Zhang%2C+and+Ming+Zhou%2E+2020%2E+ProphetNet%3A+Predicting+future+n%2Dgram+for+sequence%2Dto%2DSequencePre%2Dtraining%2E+In+Findings+of+the+)
25. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
27. Leonardo Ranaldi, Giulia Pucci, and Fabio Massimo Zanzotto. 2023. Modeling easiness for training transformers with curriculum learning. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 937– 948, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgar… [find](https://scholar.google.com/scholar?q=Leonardo+Ranaldi%2C+Giulia+Pucci%2C+and+Fabio+Massimo+Zanzotto%2E+2023%2E+Modeling+easiness+for+training+transformers+with+curriculum+learning%2E+In+Proceedings+of+the+14th+International+Conference+on+Recent+Ad)
28. Suchir Salhan, Richard Diehl Martinez, Zébulon Goriely, and Paula Buttery. 2024. Less is more: Pre-training cross-lingual small-scale language models with cognitively-plausible curriculum learning strategies. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learni… [find](https://scholar.google.com/scholar?q=Suchir+Salhan%2C+Richard+Diehl+Martinez%2C+Z%C3%A9bulon+Goriely%2C+and+Paula+Buttery%2E+2024%2E+Less+is+more%3A+Pre%2Dtraining+cross%2Dlingual+small%2Dscale+language+models+with+cognitively%2Dplausible+curriculum+learning+str)
29. Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum learning: A survey. Preprint, arXiv:2101.10382. — [arXiv:2101.10382](https://arxiv.org/abs/2101.10382) [find](https://scholar.google.com/scholar?q=Petru+Soviany%2C+Radu+Tudor+Ionescu%2C+Paolo+Rota%2C+and+Nicu+Sebe%2E+2022%2E+Curriculum+learning%3A+A+survey%2E+Preprint%2C+arXiv%3A2101%2E10382%2E+%E2%80%94+%5BarXiv%3A2101%2E10382%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2101%2E10382%29)
30. Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+Noam+Shazeer%2C+and+Jakob+Uszkoreit%2E+2018%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+31%2E+Curran+Associates%2C)
31. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprin… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edoua)
32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
33. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2020. Superglue: A stickier benchmark for general-purpose language understanding systems. Preprint, arXiv:1905.00537. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understan) [our card](https://md.research.elamur.ai/free/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
34. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos-quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of th… [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Aaron+Mueller%2C+Leshem+Choshen%2C+Ethan+Wilcox%2C+Chengxu+Zhuang%2C+Juan+Ciro%2C+Rafael+Mos%2Dquera%2C+Bhargavi+Paranjabe%2C+Adina+Williams%2C+Tal+Linzen%2C+and+Ryan+Cotterell%2E+2023%2E+Findings+of+the+BabyL)
35. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R%2E+Bowman%2E+2020%2E+Blimp%3A+The+benchmark+of+linguistic+minimal+pairs+for+english%2E+Transactions+of+the+Assoc)
36. Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095–6104, Online. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Benfeng+Xu%2C+Licheng+Zhang%2C+Zhendong+Mao%2C+Quan+Wang%2C+Hongtao+Xie%2C+and+Yongdong+Zhang%2E+2020%2E+Curriculum+learning+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso)
37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Anfeng+Li%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Gao%2C+Chengen+Huang%2C+Chenxu+Lv%2C+Chujie+Zheng%2C+Dayiheng+Liu%2C+Fan+Zhou%2C+Fei+Huang%2C+Feng+Hu%2C+Hao+Ge%2C+Haoran+Wei%2C+Hua) [our card](https://md.research.elamur.ai/free/papers/yang-2025-qwen3-technical-report/)