---
title: "Balancing Knowledge Breadth and Task Depth for Effective"
description: "They propose a curriculum that orders training data along two independent dimensions. 54 references."
authors: ["Mu Zhang", "Yuxiang Chu", "Guangya Yu", "Yongqi Fan", "Weiyan Zhang", "Hang Hu", "Tong Ruan", "Jingping Liu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.405/"
doi: "10.18653/v1/2026.findings-acl.405"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-balancing-knowledge-breadth-task-depth/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-balancing-knowledge-breadth-task-depth/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Balancing Knowledge Breadth and Task Depth for Effective Domain Adaptation Fine-Tuning

# Balancing Knowledge Breadth and Task Depth for Effective Domain Adaptation Fine-Tuning

**Authors:** Mu Zhang, Yuxiang Chu, [Guangya Yu](https://md.research.elamur.ai/free/en/authors/guangya-yu/), [Yongqi Fan](https://md.research.elamur.ai/free/en/authors/yongqi-fan/), [Weiyan Zhang](https://md.research.elamur.ai/free/en/authors/weiyan-zhang/), Hang Hu, [Tong Ruan](https://md.research.elamur.ai/free/en/authors/tong-ruan/) and [Jingping Liu](https://md.research.elamur.ai/free/en/authors/jingping-liu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.405/) · [PDF](https://aclanthology.org/2026.findings-acl.405.pdf) · [DOI 10.18653/v1/2026.findings-acl.405](https://doi.org/10.18653/v1/2026.findings-acl.405) · [OpenAlex](https://openalex.org/W7166847191)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Fine-tuning large language models on specialist corpora tends to improve in-domain skill while eroding general abilities, a problem known as catastrophic forgetting. The authors argue that curriculum learning based on a single difficulty axis cannot handle this trade-off. They propose a curriculum that orders training data along two independent dimensions. For the knowledge dimension, general and domain instances are embedded together with bge-m3 and grouped with K-means. Clusters are then ranked by cosine distance from the general-data centroid. For the task dimension, Qwen3-8B labels each domain instance with a Bloom's taxonomy level, and these labels are merged into three complexity tiers. Training proceeds from general-like clusters to domain-specific ones. Inside each cluster, it moves from simpler to more demanding tasks. An exponential schedule gradually increases the share of domain data during SFT. On Qwen3-8B, the method beats the strongest baseline by 2.49% on medical benchmarks and 1.2% on financial ones, while keeping general performance intact. This makes it a practical recipe for domain adaptation that does not sacrifice general competence.

## Key points
- The pipeline has three stages: data reconstruction (bge-m3 embeddings, K-means clustering, Bloom's-taxonomy labels from Qwen3-8B), curriculum design, and curriculum training with exponential data mixing. K = 100 clusters was chosen, and performance remained stable for K roughly between 50 and 200.
- An IDF-based analysis showed that a cluster's distance from the general centroid correlates strongly with term rarity. This supports using that distance as a proxy for domain specificity.
- In the financial domain, the method scored 0.4889, which is 1.2% above the runner-up PDPC, with a general-benchmark average of 0.5402. Compared with a fixed 50% Uniform ratio, exponential mixing raised medical results by +2.88% and general results by +1.03%.
- Ablations with Qwen3-8B and LLaMA3.1-8B separated the roles of the two dimensions:
  - Removing the knowledge ordering mainly hurt general benchmarks, for example MMLU on Qwen3-8B fell from 0.7291 to 0.7081.
  - Removing the task ordering mainly hurt domain benchmarks, for example MedMCQA on Qwen3-8B fell from 0.6014 to 0.5656, and MedQA on LLaMA3.1-8B fell from 0.6311 to 0.6025.
  - Reversing the knowledge order also lowered results.
- Limitations include testing only on models up to 8B parameters and only in the medical and financial domains. Gains held when the SFT set was enlarged to 60k instances. Code is available on GitHub (Melo-1017/Balancing-Knowledge-Breadth-and-Task-Depth).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Sun Yat-sen University](https://md.research.elamur.ai/free/en/universities/sun-yat-sen-university/) — University, Guangzhou, China · 72 works · 9,502 citations
- [East China University of Science and Technology](https://md.research.elamur.ai/free/en/universities/east-china-university-of-science-and-technology/) — University, Shanghai, China · 14 works · 87 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Mu Zhang, Yuxiang Chu, Guangya Yu, Yongqi Fan, Weiyan Zhang, Hang Hu, Tong Ruan and Jingping Liu. 2026. Balancing Knowledge Breadth and Task Depth for Effective Domain Adaptation Fine-Tuning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026balancing,
  title = {{Balancing Knowledge Breadth and Task Depth for Effective Domain Adaptation Fine-Tuning}},
  author = {Mu Zhang, Yuxiang Chu, Guangya Yu, Yongqi Fan, Weiyan Zhang, Hang Hu, Tong Ruan and Jingping Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.405},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.405}
}
```

- [← previous Balancing Fluency and Adherence: Hybrid Fallback Term Injection in Low-Resource Terminology Translation](https://md.research.elamur.ai/free/en/papers/abela-2026-balancing-fluency-adherence-hybrid-fallback/)
- [next → Bangla Key2Text: Text Generation from Keywords for a Low Resource Language](https://md.research.elamur.ai/free/en/papers/talukder-2026-bangla-key2text-text-generation-keywords/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

54 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Benjamin S Bloom, Max D Engelhart, Edward J Furst, Walker H Hill, David R Krathwohl, and 1 others. 1956. Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. Longman New York. [find](https://scholar.google.com/scholar?q=Benjamin+S+Bloom%2C+Max+D+Engelhart%2C+Edward+J+Furst%2C+Walker+H+Hill%2C+David+R+Krathwohl%2C+and+1+others%2E+1956%2E+Taxonomy+of+educational+objectives%3A+The+classification+of+educational+goals%2E+Handbook+1%3A+Cognit)
3. Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930. [find](https://scholar.google.com/scholar?q=Pietro+Buzzega%2C+Matteo+Boschini%2C+Angelo+Porrello%2C+Davide+Abati%2C+and+Simone+Calderara%2E+2020%2E+Dark+experience+for+general+continual+learning%3A+a+strong%2C+simple+baseline%2E+Advances+in+neural+information+pr)
4. Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. 2023. [Instruction mining: Instruction data selection for tuning large language models](https://md.research.elamur.ai/free/en/papers/cao-2023-instruction-mining-instruction-data-selection/). arXiv preprint arXiv:2307.06290.
5. Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato. 2019. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486. [find](https://scholar.google.com/scholar?q=Arslan+Chaudhry%2C+Marcus+Rohrbach%2C+Mohamed+Elhoseiny%2C+Thalaiyasingam+Ajanthan%2C+Puneet+K+Dokania%2C+Philip+HS+Torr%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+On+tiny+episodic+memories+in+continual+learning%2E+arXiv+p)
6. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. [Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). Preprint, arXiv:2402.03216.
7. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024b. [Huatuogpt-o1, towards medical complex reasoning with llms](https://md.research.elamur.ai/free/en/papers/chen-2024-huatuogpt-o1-towards-medical-complex/). arXiv preprint arXiv:2412.18925.
8. Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. 2025. Self-evolving curriculum for llm reasoning. arXiv preprint arXiv:2505.14970. [find](https://scholar.google.com/scholar?q=Xiaoyin+Chen%2C+Jiarui+Lu%2C+Minsu+Kim%2C+Dinghuai+Zhang%2C+Jian+Tang%2C+Alexandre+Pich%C3%A9%2C+Nicolas+Gontier%2C+Yoshua+Bengio%2C+and+Ehsan+Kamalloo%2E+2025%2E+Self%2Devolving+curriculum+for+llm+reasoning%2E+arXiv+preprint+arX)
9. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. [FinQA: A dataset of numerical reasoning over financial data](https://md.research.elamur.ai/free/en/papers/chen-2021-finqa-dataset-numerical-reasoning-over/). In Proceedings of the 2021 Conference on Empirical Methods in Natural…
10. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
11. Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385. [find](https://scholar.google.com/scholar?q=Matthias+De+Lange%2C+Rahaf+Aljundi%2C+Marc+Masana%2C+Sarah+Parisot%2C+Xu+Jia%2C+Ale%C5%A1+Leonardis%2C+Gregory+Slabaugh%2C+and+Tinne+Tuytelaars%2E+2021%2E+A+continual+learning+survey%3A+Defying+forgetting+in+classification+ta)
12. Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2024. [How abilities in large language models are affected by supervised fine-tuning data composition](https://md.research.elamur.ai/free/en/papers/dong-2024-how-abilities-large-language-models/). In Proceedings of the 62nd Annual Meeting of the Association for…
13. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, pages 3762–3773. PMLR. [find](https://scholar.google.com/scholar?q=Mehrdad+Farajtabar%2C+Navid+Azizan%2C+Alex+Mott%2C+and+Ang+Li%2E+2020%2E+Orthogonal+gradient+descent+for+continual+learning%2E+In+International+conference+on+artificial+intelligence+and+statistics%2C+pages+3762%E2%80%93377)
14. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135. [find](https://scholar.google.com/scholar?q=Robert+M+French%2E+1999%2E+Catastrophic+forgetting+in+connectionist+networks%2E+Trends+in+cognitive+sciences%2C+3%284%29%3A128%E2%80%93135%2E)
15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
18. Ruihui Hou, Shencheng Chen, Yongqi Fan, Guangya Yu, Lifeng Zhu, Jing Sun, Jingping Liu, and Tong Ruan. 2025a. Msdiagnosis: A benchmark and framework for evaluating large language models in multi-step clinical diagnosis. Knowledge-Based Systems, 330:114524. [find](https://scholar.google.com/scholar?q=Ruihui+Hou%2C+Shencheng+Chen%2C+Yongqi+Fan%2C+Guangya+Yu%2C+Lifeng+Zhu%2C+Jing+Sun%2C+Jingping+Liu%2C+and+Tong+Ruan%2E+2025a%2E+Msdiagnosis%3A+A+benchmark+and+framework+for+evaluating+large+language+models+in+multi%2Dstep+)
19. Zhitian Hou, Zihan Ye, Nanli Zeng, Tianyong Hao, and Kun Zeng. 2025b. Large language models meet legal artificial intelligence: A survey. arXiv preprint arXiv:2509.09969. [find](https://scholar.google.com/scholar?q=Zhitian+Hou%2C+Zihan+Ye%2C+Nanli+Zeng%2C+Tianyong+Hao%2C+and+Kun+Zeng%2E+2025b%2E+Large+language+models+meet+legal+artificial+intelligence%3A+A+survey%2E+arXiv+preprint+arXiv%3A2509%2E09969%2E)
20. Thomas Huber and Christina Niklaus. 2025. LLMs meet bloom’s taxonomy: A cognitive view on large language model evaluations. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5211–5246, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Thomas+Huber+and+Christina+Niklaus%2E+2025%2E+LLMs+meet+bloom%E2%80%99s+taxonomy%3A+A+cognitive+view+on+large+language+model+evaluations%2E+In+Proceedings+of+the+31st+International+Conference+on+Computational+Linguis)
21. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
22. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. [PubMedQA: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
23. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(1… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+and+1+others%2E+2017%2E+Overcomin)
24. M Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. Advances in neural information processing systems, 23. [find](https://scholar.google.com/scholar?q=M+Kumar%2C+Benjamin+Packer%2C+and+Daphne+Koller%2E+2010%2E+Self%2Dpaced+learning+for+latent+variable+models%2E+Advances+in+neural+information+processing+systems%2C+23%2E)
25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
26. Bruce W Lee, Hyunsoo Cho, and Kang Min Yoo. 2024. Instruction tuning with human curriculum. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1281–1309, Mexico City, Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bruce+W+Lee%2C+Hyunsoo+Cho%2C+and+Kang+Min+Yoo%2E+2024%2E+Instruction+tuning+with+human+curriculum%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2024%2C+pages+1281%E2%80%931309%2C+Mexico+City%2C+Mexi)
27. Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao. 2024. Revisiting catastrophic forgetting in large language model tuning. arXiv preprint arXiv:2406.04836. [find](https://scholar.google.com/scholar?q=Hongyu+Li%2C+Liang+Ding%2C+Meng+Fang%2C+and+Dacheng+Tao%2E+2024%2E+Revisiting+catastrophic+forgetting+in+large+language+model+tuning%2E+arXiv+preprint+arXiv%3A2406%2E04836%2E)
28. Jijie Li, Li Du, Hanyu Zhao, Bo-wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. 2025. Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models. arXiv preprint. ArXiv:2506.11116 [cs]. — [arXiv:2506.11116](https://arxiv.org/abs/2506.11116) [find](https://scholar.google.com/scholar?q=Jijie+Li%2C+Li+Du%2C+Hanyu+Zhao%2C+Bo%2Dwen+Zhang%2C+Liangdong+Wang%2C+Boyan+Gao%2C+Guang+Liu%2C+and+Yonghua+Lin%2E+2025%2E+Infinity+Instruct%3A+Scaling+Instruction+Selection+and+Synthesis+to+Enhance+Language+Models%2E+arXiv)
29. Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947. [find](https://scholar.google.com/scholar?q=Zhizhong+Li+and+Derek+Hoiem%2E+2017%2E+Learning+without+forgetting%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+40%2812%29%3A2935%E2%80%932947%2E)
30. Yupian Lin, Tong Ruan, Ming Liang, Tingting Cai, Wen Du, and Yi Wang. 2022. Dotat: A domain-oriented text annotation tool. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 1–8. [find](https://scholar.google.com/scholar?q=Yupian+Lin%2C+Tong+Ruan%2C+Ming+Liang%2C+Tingting+Cai%2C+Wen+Du%2C+and+Yi+Wang%2E+2022%2E+Dotat%3A+A+domain%2Doriented+text+annotation+tool%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
31. Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and 1 others. 2024. [Rho-1: Not all tokens are what you need](https://md.research.elamur.ai/free/en/papers/lin-2024-rho-1-all-tokens-need/). arXiv preprint arXiv:2404.07965.
32. Stuart Lloyd. 1982. Least squares quantization in pcm. IEEE transactions on information theory, 28(2):129– 137. [find](https://scholar.google.com/scholar?q=Stuart+Lloyd%2E+1982%2E+Least+squares+quantization+in+pcm%2E+IEEE+transactions+on+information+theory%2C+28%282%29%3A129%E2%80%93+137%2E)
33. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2025. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. IEEE Transactions on Audio, Speech and Language Processing. [find](https://scholar.google.com/scholar?q=Yun+Luo%2C+Zhen+Yang%2C+Fandong+Meng%2C+Yafu+Li%2C+Jie+Zhou%2C+and+Yue+Zhang%2E+2025%2E+An+empirical+study+of+catastrophic+forgetting+in+large+language+models+during+continual+fine%2Dtuning%2E+IEEE+Transactions+on+Audi)
34. Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782–796. [find](https://scholar.google.com/scholar?q=Pekka+Malo%2C+Ankur+Sinha%2C+Pekka+Korhonen%2C+Jyrki+Wallenius%2C+and+Pyry+Takala%2E+2014%2E+Good+debt+or+bad+debt%3A+Detecting+semantic+orientations+in+economic+texts%2E+Journal+of+the+Association+for+Information+Sc)
35. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35:27730–27744.
37. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on health, inference, and learning, pages 248–260. PMLR.
38. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
39. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv preprint arXiv:1606.04671. [find](https://scholar.google.com/scholar?q=Andrei+A+Rusu%2C+Neil+C+Rabinowitz%2C+Guillaume+Desjardins%2C+Hubert+Soyer%2C+James+Kirkpatrick%2C+Koray+Kavukcuoglu%2C+Razvan+Pascanu%2C+and+Raia+Hadsell%2E+2016%2E+Progressive+neural+networks%2E+arXiv+preprint+arXiv%3A16)
40. Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pages 4548–4557. PMLR. [find](https://scholar.google.com/scholar?q=Joan+Serra%2C+Didac+Suris%2C+Marius+Miron%2C+and+Alexandros+Karatzoglou%2E+2018%2E+Overcoming+catastrophic+forgetting+with+hard+attention+to+the+task%2E+In+International+conference+on+machine+learning%2C+pages+4548)
41. Ankur Sinha and Tanmay Khandait. 2021. Impact of news on the commodity market: Dataset and results. In Future of Information and Communication Conference, pages 589–601. Springer. [find](https://scholar.google.com/scholar?q=Ankur+Sinha+and+Tanmay+Khandait%2E+2021%2E+Impact+of+news+on+the+commodity+market%3A+Dataset+and+results%2E+In+Future+of+Information+and+Communication+Conference%2C+pages+589%E2%80%93601%2E+Springer%2E)
42. Shezheng Song, Hao Xu, Jun Ma, Shasha Li, Long Peng, Qian Wan, Xiaodong Liu, and Jie Yu. 2025. How to alleviate catastrophic forgetting in llms fine-tuning? hierarchical layer-wise and element-wise regularization. arXiv preprint arXiv:2501.13669. — [arXiv:2501.13669](https://arxiv.org/abs/2501.13669) [find](https://scholar.google.com/scholar?q=Shezheng+Song%2C+Hao+Xu%2C+Jun+Ma%2C+Shasha+Li%2C+Long+Peng%2C+Qian+Wan%2C+Xiaodong+Liu%2C+and+Jie+Yu%2E+2025%2E+How+to+alleviate+catastrophic+forgetting+in+llms+fine%2Dtuning%3F+hierarchical+layer%2Dwise+and+element%2Dwise+re)
43. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). Advances in Neural Information Processing Systems, 37:95266–95290…
44. Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149. [find](https://scholar.google.com/scholar?q=Zifeng+Wang%2C+Zizhao+Zhang%2C+Chen%2DYu+Lee%2C+Han+Zhang%2C+Ruoxi+Sun%2C+Xiaoqi+Ren%2C+Guolong+Su%2C+Vincent+Perot%2C+Jennifer+Dy%2C+and+Tomas+Pfister%2E+2022%2E+Learning+to+prompt+for+continual+learning%2E+In+Proceedings+of+)
45. Yunchao Wei, Xiaodan Liang, Yunpeng Chen, Xiaohui Shen, Ming-Ming Cheng, Jiashi Feng, Yao Zhao, and Shuicheng Yan. 2016. Stc: A simple to complex framework for weakly-supervised semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(11):2314–2320. [find](https://scholar.google.com/scholar?q=Yunchao+Wei%2C+Xiaodan+Liang%2C+Yunpeng+Chen%2C+Xiaohui+Shen%2C+Ming%2DMing+Cheng%2C+Jiashi+Feng%2C+Yao+Zhao%2C+and+Shuicheng+Yan%2E+2016%2E+Stc%3A+A+simple+to+complex+framework+for+weakly%2Dsupervised+semantic+segmentation%2E)
46. Daphna Weinshall, Gad Cohen, and Dan Amir. 2018. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International conference on machine learning, pages 5238–5246. PMLR. [find](https://scholar.google.com/scholar?q=Daphna+Weinshall%2C+Gad+Cohen%2C+and+Dan+Amir%2E+2018%2E+Curriculum+learning+by+transfer+learning%3A+Theory+and+experiments+with+deep+networks%2E+In+International+conference+on+machine+learning%2C+pages+5238%E2%80%935246%2E+)
47. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) [find](https://scholar.google.com/scholar?q=Qianqian+Xie%2C+Weiguang+Han%2C+Xiao+Zhang%2C+Yanzhao+Lai%2C+Min+Peng%2C+Alejandro+Lopez%2DLira%2C+and+Jimin+Huang%2E+2023%2E+Pixiu%3A+A+large+language+model%2C+instruction+data+and+evaluation+benchmark+for+finance%2E+arXiv+)
48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
49. Yuanhao Yue, Chengyu Wang, Jun Huang, and Peng Wang. 2024. Distilling instruction-following abilities of large language models with task-aware curriculum planning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6030–6054, Miami, Florida, USA. Association for Computat… [find](https://scholar.google.com/scholar?q=Yuanhao+Yue%2C+Chengyu+Wang%2C+Jun+Huang%2C+and+Peng+Wang%2E+2024%2E+Distilling+instruction%2Dfollowing+abilities+of+large+language+models+with+task%2Daware+curriculum+planning%2E+In+Findings+of+the+Association+for+C)
50. Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR. [find](https://scholar.google.com/scholar?q=Friedemann+Zenke%2C+Ben+Poole%2C+and+Surya+Ganguli%2E+2017%2E+Continual+learning+through+synaptic+intelligence%2E+In+International+conference+on+machine+learning%2C+pages+3987%E2%80%933995%2E+PMLR%2E)
51. Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. 2023. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558. — [arXiv:2310.14558](https://arxiv.org/abs/2310.14558) [find](https://scholar.google.com/scholar?q=Xinlu+Zhang%2C+Chenxin+Tian%2C+Xianjun+Yang%2C+Lichang+Chen%2C+Zekun+Li%2C+and+Linda+Ruth+Petzold%2E+2023%2E+Alpacare%3A+Instruction%2Dtuned+large+language+models+for+medical+application%2E+arXiv+preprint+arXiv%3A2310%2E1455)
52. Xuan Zhang, Pamela Shapiro, Gaurav Kumar, Paul Mc-Namee, Marine Carpuat, and Kevin Duh. 2019. [Curriculum learning for domain adaptation in neural machine translation](https://md.research.elamur.ai/free/en/papers/zhang-2019-curriculum-learning-domain-adaptation-neural/). arXiv preprint arXiv:1905.05816.
53. Xuemiao Zhang, Xu Liangyu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. 2025. Preference curriculum: LLMs should always be pretrained on their preferred data. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21181–21198, Vienna, A… [find](https://scholar.google.com/scholar?q=Xuemiao+Zhang%2C+Xu+Liangyu%2C+Feiyu+Duan%2C+Yongwei+Zhou%2C+Sirui+Wang%2C+Rongxiang+Weng%2C+Jingang+Wang%2C+and+Xunliang+Cai%2E+2025%2E+Preference+curriculum%3A+LLMs+should+always+be+pretrained+on+their+preferred+data%2E+)
54. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations),…

*Updated 2 November 2026*