---
title: "Should Code Models Learn Pedagogically? A Preliminary"
description: "The authors test whether curriculum learning (CL) helps pre-trained code models on… 10.44 F1. 28 references, cited by 1 paper here."
authors: ["Kyi Shin Khant", "Hong Yi Lin", "Patanamon Thongtanunam"]
year: 2025
venue: "arXiv:2502.03806"
source: "https://arxiv.org/abs/2502.03806"
doi: "10.48550/arxiv.2502.03806"
arxiv: "2502.03806"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khant-2025-should-code-models-learn-pedagogically/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khant-2025-should-code-models-learn-pedagogically/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/18/) / Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

# Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

**Authors:** Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam · **Year:** 2025 · **Venue:** arXiv:2502.03806 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.03806) · [PDF](https://arxiv.org/pdf/2502.03806) · [DOI 10.48550/arxiv.2502.03806](https://doi.org/10.48550/arxiv.2502.03806) · [arXiv 2502.03806](https://arxiv.org/abs/2502.03806) · [OpenAlex](https://openalex.org/W4411272382)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors test whether curriculum learning (CL) helps pre-trained code models on real-world software engineering tasks. CL means training from simple examples to complex ones. Previously, the benefits of CL had been shown mostly on synthetic or artificially complicated code. Here, code complexity is measured with two classic metrics: length (the number of whitespace-separated tokens) and cyclomatic complexity (computed with the Lizard tool). Data from the CodeXGLUE benchmark are split by quartiles into four difficulty levels and balanced in size. CodeT5 is then progressively fine-tuned on the code clone detection and code summarization tasks. The comparison covers a forward schedule (from easy to hardest), a reverse schedule, and conventional training on shuffled data. No CL variant outperformed conventional training. In clone detection there are signs of catastrophic forgetting, while in summarization the model barely responds to the order of the data. The authors suggest that it relies on meaningful identifier names (shortcut learning). In addition, performance plateaus already after the first quarter of the data. The work calls into question the universality of the "pedagogical" approach to training code models.

## Key points
- Model: CodeT5; data: CodeXGLUE (code from GitHub). Complexity metrics: length (L) and cyclomatic complexity (C). For program pairs in clone detection, the complexities of the two programs are summed.
- Code summarization: conventional training yields BLEU 20.77 (L) and 20.44 (C). The CL variants obtained 20.43, 20.89, 20.28 and 20.24, i.e., the difference is negligible. BERTScore F1 shows the same picture.
- Clone detection: on the hardest subset, the reverse schedule lags behind conventional training by up to 10.44 F1 points. Examples the model saw at the beginning of training are forgotten by the end.
- The model plateaus already after 25% of the training data, regardless of whether training started with the easy or the hardest part. Subsequent difficulty levels add almost nothing.
- All study materials are published on Zenodo (records/14059483). The authors recommend examining other complexity metrics (e.g., clone type), other models and programming languages.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 104 works · 22,674 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam. 2025. Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks. arXiv:2502.03806.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{khant2025should,
  title = {{Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks}},
  author = {Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam},
  year = {2025},
  doi = {10.48550/arxiv.2502.03806},
  eprint = {2502.03806},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.03806}
}
```

- [← previous Shawarma Chats: A Benchmark Exact Dialogue & Evaluation Platter in Egyptian, Maghrebi & Modern Standard Arabic–-A Triple-Dialect Feast for Hungry Language Models](https://md.research.elamur.ai/free/en/papers/zeinalipour-2025-shawarma-chats-benchmark-exact-dialogue/)
- [next → Streamlining Arabic Grammar to Facilitate Mastery of Qirâ’at al-Kutub for University Students](https://md.research.elamur.ai/free/en/papers/ardiansyah-2025-streamlining-arabic-grammar-facilitate-mastery/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/18/)

## References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line-level vulnerability prediction,” in MSR. New York, NY, USA: ACM, 2022, pp. 608–620. [find](https://scholar.google.com/scholar?q=M%2E+Fu+and+C%2E+Tantithamthavorn%2C+%E2%80%9CLinevul%3A+A+transformer%2Dbased+line%2Dlevel+vulnerability+prediction%2C%E2%80%9D+in+MSR%2E+New+York%2C+NY%2C+USA%3A+ACM%2C+2022%2C+pp%2E+608%E2%80%93620%2E)
2. M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural 2 [https://zenodo.org/records/14059483](https://zenodo.org/records/14059483) machine translation,” TOSEM, vol. 28, no. 4, pp. 19:1– 19:29, 2019. [↗](https://zenodo.org/records/14059483)
3. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and g… [find](https://scholar.google.com/scholar?q=S%2E+Lu%2C+D%2E+Guo%2C+S%2E+Ren%2C+J%2E+Huang%2C+A%2E+Svyatkovskiy%2C+A%2E+Blanco%2C+C%2E+B%2E+Clement%2C+D%2E+Drain%2C+D%2E+Jiang%2C+D%2E+Tang%2C+G%2E+Li%2C+L%2E+Zhou%2C+L%2E+Shou%2C+L%2E+Zhou%2C+M%2E+Tufano%2C+M%2E+Gong%2C+M%2E+Zhou%2C+N%2E+Duan%2C+N%2E+Sundaresan%2C+S%2E+K%2E+De)
4. H. Y. Lin and P. Thongtanunam, “Towards automated code reviews: Does learning code structure help?” in SANER. NJ, USA: IEEE, 2023, pp. 703–707. [find](https://scholar.google.com/scholar?q=H%2E+Y%2E+Lin+and+P%2E+Thongtanunam%2C+%E2%80%9CTowards+automated+code+reviews%3A+Does+learning+code+structure+help%3F%E2%80%9D+in+SANER%2E+NJ%2C+USA%3A+IEEE%2C+2023%2C+pp%2E+703%E2%80%93707%2E)
5. J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton, “[Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/),” CoRR, vol. abs/2108.07732, 2021.
6. Z. Zeng, H. Tan, H. Zhang, J. Li, Y. Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in ISSTA. New York, NY, USA: ACM, 2022, pp. 39–51. [find](https://scholar.google.com/scholar?q=Z%2E+Zeng%2C+H%2E+Tan%2C+H%2E+Zhang%2C+J%2E+Li%2C+Y%2E+Zhang%2C+and+L%2E+Zhang%2C+%E2%80%9CAn+extensive+study+on+pre%2Dtrained+models+for+program+understanding+and+generation%2C%E2%80%9D+in+ISSTA%2E+New+York%2C+NY%2C+USA%3A+ACM%2C+2022%2C+pp%2E+39%E2%80%9351%2E)
7. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “[Curriculum learning](https://md.research.elamur.ai/free/en/papers/bengio-2009-curriculum-learning/),” in ICML, vol. 382. New York, NY, USA: ACM, 2009, pp. 41–48.
8. X. Wang, Y. Chen, and W. Zhu, “A survey on curriculum learning,” TPAMI, vol. 44, no. 9, pp. 4555–4576, 2022. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+Y%2E+Chen%2C+and+W%2E+Zhu%2C+%E2%80%9CA+survey+on+curriculum+learning%2C%E2%80%9D+TPAMI%2C+vol%2E+44%2C+no%2E+9%2C+pp%2E+4555%E2%80%934576%2C+2022%2E)
9. M. Naı̈r, K. M. Yamani, L. S. L’Hadj, and R. Baghdadi, “Curriculum learning for small code language models,” in ACL - Student Research Workshop, 2024, pp. 531–542. [find](https://scholar.google.com/scholar?q=M%2E+Na%C4%B1%CC%88r%2C+K%2E+M%2E+Yamani%2C+L%2E+S%2E+L%E2%80%99Hadj%2C+and+R%2E+Baghdadi%2C+%E2%80%9CCurriculum+learning+for+small+code+language+models%2C%E2%80%9D+in+ACL+%2D+Student+Research+Workshop%2C+2024%2C+pp%2E+531%E2%80%93542%2E)
10. C. Liu, S. Lu, W. Chen, D. Jiang, A. Svyatkovskiy, S. Fu, N. Sundaresan, and N. Duan, “[Code execution with pre-trained language models](https://md.research.elamur.ai/free/en/papers/liu-2023-code-execution-pre-trained-language/),” in ACL, 2023, pp. 4984–4999.
11. D. Wang, Z. Jia, S. Li, Y. Yu, Y. Xiong, W. Dong, and X. Liao, “Bridging pre-trained models and downstream tasks for source code understanding,” in ICSE. New York, NY, USA: ACM, 2022, pp. 287–298. [find](https://scholar.google.com/scholar?q=D%2E+Wang%2C+Z%2E+Jia%2C+S%2E+Li%2C+Y%2E+Yu%2C+Y%2E+Xiong%2C+W%2E+Dong%2C+and+X%2E+Liao%2C+%E2%80%9CBridging+pre%2Dtrained+models+and+downstream+tasks+for+source+code+understanding%2C%E2%80%9D+in+ICSE%2E+New+York%2C+NY%2C+USA%3A+ACM%2C+2022%2C+pp%2E+287%E2%80%93298%2E)
12. S. Hao, X. Shi, H. Liu, and Y. Shu, “Enhancing code language models for program repair by curricular fine-tuning framework,” in ICSME. NJ, USA: IEEE, 2023, pp. 136–146. [find](https://scholar.google.com/scholar?q=S%2E+Hao%2C+X%2E+Shi%2C+H%2E+Liu%2C+and+Y%2E+Shu%2C+%E2%80%9CEnhancing+code+language+models+for+program+repair+by+curricular+fine%2Dtuning+framework%2C%E2%80%9D+in+ICSME%2E+NJ%2C+USA%3A+IEEE%2C+2023%2C+pp%2E+136%E2%80%93146%2E)
13. Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in EMNLP. ACL, 2021, pp. 8696–8708. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+W%2E+Wang%2C+S%2E+R%2E+Joty%2C+and+S%2E+C%2E+H%2E+Hoi%2C+%E2%80%9CCodet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2C%E2%80%9D+in+EMNLP%2E+ACL%2C+2021%2C+pp%2E+8696%E2%80%938708%2E)
14. L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in AAAI. AAAI Press, 2016, pp. 1287–1293. [find](https://scholar.google.com/scholar?q=L%2E+Mou%2C+G%2E+Li%2C+L%2E+Zhang%2C+T%2E+Wang%2C+and+Z%2E+Jin%2C+%E2%80%9CConvolutional+neural+networks+over+tree+structures+for+programming+language+processing%2C%E2%80%9D+in+AAAI%2E+AAAI+Press%2C+2016%2C+pp%2E+1287%E2%80%931293%2E)
15. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in ACL, 2016. [find](https://scholar.google.com/scholar?q=S%2E+Iyer%2C+I%2E+Konstas%2C+A%2E+Cheung%2C+and+L%2E+Zettlemoyer%2C+%E2%80%9CSummarizing+source+code+using+a+neural+attention+model%2C%E2%80%9D+in+ACL%2C+2016%2E)
16. A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot AI pair programmer: Asset or liability?” J. Syst. Softw., vol. 203, p. 111734, 2023. [find](https://scholar.google.com/scholar?q=A%2E+M%2E+Dakhel%2C+V%2E+Majdinasab%2C+A%2E+Nikanjam%2C+F%2E+Khomh%2C+M%2E+C%2E+Desmarais%2C+and+Z%2E+M%2E+J%2E+Jiang%2C+%E2%80%9CGithub+copilot+AI+pair+programmer%3A+Asset+or+liability%3F%E2%80%9D+J%2E+Syst%2E+Softw%2E%2C+vol%2E+203%2C+p%2E+111734%2C+2023%2E)
17. A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, “Automated curriculum learning for neural networks,” in ICML, vol. 70. PMLR, 2017, pp. 1311–1320. [find](https://scholar.google.com/scholar?q=A%2E+Graves%2C+M%2E+G%2E+Bellemare%2C+J%2E+Menick%2C+R%2E+Munos%2C+and+K%2E+Kavukcuoglu%2C+%E2%80%9CAutomated+curriculum+learning+for+neural+networks%2C%E2%80%9D+in+ICML%2C+vol%2E+70%2E+PMLR%2C+2017%2C+pp%2E+1311%E2%80%931320%2E)
18. E. A. Platanios, O. Stretcu, G. Neubig, B. Póczos, and T. M. Mitchell, “Competence-based curriculum learning for neural machine translation,” in NAACL-HLT. ACL, 2019, pp. 1162–1172. [find](https://scholar.google.com/scholar?q=E%2E+A%2E+Platanios%2C+O%2E+Stretcu%2C+G%2E+Neubig%2C+B%2E+Po%CC%81czos%2C+and+T%2E+M%2E+Mitchell%2C+%E2%80%9CCompetence%2Dbased+curriculum+learning+for+neural+machine+translation%2C%E2%80%9D+in+NAACL%2DHLT%2E+ACL%2C+2019%2C+pp%2E+1162%E2%80%931172%2E)
19. T. J. McCabe, “A complexity measure,” TSE., vol. 2, no. 4, pp. 308–320, 1976. [find](https://scholar.google.com/scholar?q=T%2E+J%2E+McCabe%2C+%E2%80%9CA+complexity+measure%2C%E2%80%9D+TSE%2E%2C+vol%2E+2%2C+no%2E+4%2C+pp%2E+308%E2%80%93320%2C+1976%2E)
20. M. Allamanis, E. T. Barr, P. T. Devanbu, and C. Sutton, “[A survey of machine learning for big code and naturalness](https://md.research.elamur.ai/free/en/papers/allamanis-2017-survey-machine-learning-big-code/),” CSUR, vol. 51, no. 4, pp. 81:1–81:37, 2018.
21. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “[Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/),” TACL, vol. 12, pp. 157–173, 2024.
22. M. J. Shepperd, “A critique of cyclomatic complexity as a software metric,” Softw. Eng. J., vol. 3, no. 2, pp. 30–36, 1988. [find](https://scholar.google.com/scholar?q=M%2E+J%2E+Shepperd%2C+%E2%80%9CA+critique+of+cyclomatic+complexity+as+a+software+metric%2C%E2%80%9D+Softw%2E+Eng%2E+J%2E%2C+vol%2E+3%2C+no%2E+2%2C+pp%2E+30%E2%80%9336%2C+1988%2E)
23. T. L. Alves, C. Ypma, and J. Visser, “Deriving metric thresholds from benchmark data,” in ICSM. NJ, USA: IEEE, 2010, pp. 1–10. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Alves%2C+C%2E+Ypma%2C+and+J%2E+Visser%2C+%E2%80%9CDeriving+metric+thresholds+from+benchmark+data%2C%E2%80%9D+in+ICSM%2E+NJ%2C+USA%3A+IEEE%2C+2010%2C+pp%2E+1%E2%80%9310%2E)
24. M. Khajezade, J. J. Wu, F. H. Fard, G. Rodrı́guez-Pérez, and M. S. Shehata, “Investigating the efficacy of large language models for code clone detection,” in ICPC. New York, NY, USA: ACM, 2024, pp. 161–165. [find](https://scholar.google.com/scholar?q=M%2E+Khajezade%2C+J%2E+J%2E+Wu%2C+F%2E+H%2E+Fard%2C+G%2E+Rodr%C4%B1%CC%81guez%2DPe%CC%81rez%2C+and+M%2E+S%2E+Shehata%2C+%E2%80%9CInvestigating+the+efficacy+of+large+language+models+for+code+clone+detection%2C%E2%80%9D+in+ICPC%2E+New+York%2C+NY%2C+USA%3A+ACM%2C+2024%2C+pp%2E+)
25. K. Papineni, S. Roukos, T. Ward, and W. Zhu, “[Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/),” in ACL, 2002, pp. 311–318.
26. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with BERT,” in ICLR, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+V%2E+Kishore%2C+F%2E+Wu%2C+K%2E+Q%2E+Weinberger%2C+and+Y%2E+Artzi%2C+%E2%80%9CBertscore%3A+Evaluating+text+generation+with+BERT%2C%E2%80%9D+in+ICLR%2C+2020%2E)
27. C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto, “Toward understanding catastrophic forgetting in continual learning,” CoRR, vol. abs/1908.01091, 2019. — [arXiv:1908.01091](https://arxiv.org/abs/1908.01091) [find](https://scholar.google.com/scholar?q=C%2E+V%2E+Nguyen%2C+A%2E+Achille%2C+M%2E+Lam%2C+T%2E+Hassner%2C+V%2E+Mahadevan%2C+and+S%2E+Soatto%2C+%E2%80%9CToward+understanding+catastrophic+forgetting+in+continual+learning%2C%E2%80%9D+CoRR%2C+vol%2E+abs%2F1908%2E01091%2C+2019%2E+%E2%80%94+%5BarXiv%3A1908%2E01091%5D%28h)
28. M. Du, F. He, N. Zou, D. Tao, and X. Hu, “Shortcut learning of large language models in natural language understanding,” Commun., vol. 67, no. 1, pp. 110–120, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Du%2C+F%2E+He%2C+N%2E+Zou%2C+D%2E+Tao%2C+and+X%2E+Hu%2C+%E2%80%9CShortcut+learning+of+large+language+models+in+natural+language+understanding%2C%E2%80%9D+Commun%2E%2C+vol%2E+67%2C+no%2E+1%2C+pp%2E+110%E2%80%93120%2C+2024%2E)

*Updated 3 November 2026*