---
title: "MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning (2024)"
description: "The authors propose MoRAL, which places a router-controlled Mixture-of-Experts over LoRA adapters. 107 references, cited by 18 papers here."
authors: ["Shu Yang; Muhammad Asif Ali; Cheng-Long Wang; Lijie Hu;", "Di Wang"]
year: 2024
venue: "arXiv:2402.11260"
source: "https://arxiv.org/abs/2402.11260"
doi: "10.48550/arxiv.2402.11260"
arxiv: "2402.11260"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2024-moral-moe-augmented-lora-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2024-moral-moe-augmented-lora-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

# MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

**Authors:** Shu Yang; Muhammad Asif Ali; Cheng-Long Wang; Lijie Hu; and [Di Wang](https://md.research.elamur.ai/free/en/authors/di-wang/) · **Year:** 2024 · **Venue:** arXiv:2402.11260 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.11260) · [PDF](https://arxiv.org/pdf/2402.11260) · [DOI 10.48550/arxiv.2402.11260](https://doi.org/10.48550/arxiv.2402.11260) · [arXiv 2402.11260](https://arxiv.org/abs/2402.11260) · [OpenAlex](https://openalex.org/W4392011958)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [18](https://md.research.elamur.ai/free/en/papers/yang-2024-moral-moe-augmented-lora-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models go stale after training. They often hallucinate about recent events, and updating them risks catastrophic forgetting. Earlier lifelong-learning and model-editing methods mostly feed models sentences built from fact triplets, which are hard to extract from real text and lose information. The authors propose MoRAL, which places a router-controlled Mixture-of-Experts over LoRA adapters. Each expert can capture a different low-rank knowledge subspace, while training stays parameter-efficient. MoRAL learns from plain question-answer pairs drawn directly from unstructured documents.

To evaluate it, they build 5L-bench, which includes:
- a new QA dataset derived from recent Arxiv papers, to test adaptation to new domains;
- HotpotQA, as a holdout set for knowledge retention;
- metrics for open-book, closed-book and cross-setting evaluation, scored by GPT-4 and GLM-4.

Models answer far better when they can consult context. MoRAL's gains increase with model size. Unlike standard fine-tuning, which raises scores on new data but reduces them on old data, MoRAL keeps holdout performance relatively stable. This points to a practical recipe for keeping LLMs current.

## Key points
- MoRAL is compared against IA3, LLaMA-Adapter and LoRA on TinyLlama-1.1B-Chat, Phi-2-2.7B and Llama-2-7B-chat-hf.
- With MoRAL fine-tuning, Phi-2-2.7B shows up to a 30.15% "RA" improvement in open-book versus closed-book mode.
- 5L-bench metrics include Faith, Filter, RR and RA (open-book), RA (closed-book), plus QR and FL (cross-setting), all scored from 0 to 1.
- On the HotpotQA retention test, Llama-2-7B+MoRAL reaches open-book RA 0.89 versus 0.85 for LoRA, and Faith 0.69 versus 0.65.
- Without fine-tuning, TinyLlama-1.1B achieved the best baseline RA (0.60 closed-book, 0.86 open-book). However, it lagged larger models on refusing unanswerable questions (RR).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Macau](https://md.research.elamur.ai/free/en/universities/university-of-macau/) — University, Macao, China · 33 works · 2,223 citations
- [King Abdullah University of Science and Technology](https://md.research.elamur.ai/free/en/universities/king-abdullah-university-of-science-and-technology/) — University, Jeddah, Saudi Arabia · 27 works · 1,084 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Shu Yang; Muhammad Asif Ali; Cheng-Long Wang; Lijie Hu; and Di Wang. 2024. MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning. arXiv:2402.11260.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2024moral,
  title = {{MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning}},
  author = {Shu Yang; Muhammad Asif Ali; Cheng-Long Wang; Lijie Hu; and Di Wang},
  year = {2024},
  doi = {10.48550/arxiv.2402.11260},
  eprint = {2402.11260},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.11260}
}
```

- [← previous Language Models as Models of Language](https://md.research.elamur.ai/free/en/papers/milliere-2024-language-models-models-language/)
- [next → Personality Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-personality-alignment-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

107 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. [Muppet: Massive multi-task representations with pre-finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/).
2. Muhammad Asif Ali, Yifang Sun, Xiaoling Zhou, Wei Wang, and Xiang Zhao. 2019. Antonym-synonym classification based on new sub-space embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33(01), pages 6204–6211. [find](https://scholar.google.com/scholar?q=Muhammad+Asif+Ali%2C+Yifang+Sun%2C+Xiaoling+Zhou%2C+Wei+Wang%2C+and+Xiang+Zhao%2E+2019%2E+Antonym%2Dsynonym+classification+based+on+new+sub%2Dspace+embeddings%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Inte)
3. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. [find](https://scholar.google.com/scholar?q=Badr+AlKhamissi%2C+Millicent+Li%2C+Asli+Celikyilmaz%2C+Mona+Diab%2C+and+Marjan+Ghazvininejad%2E+2022%2E+A+review+on+language+models+as+knowledge+bases%2E)
4. Anthropic. 2023. Model card and evaluations for claude models. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2023%2E+Model+card+and+evaluations+for+claude+models%2E)
5. Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. 2024. Rag vs fine-tu… [find](https://scholar.google.com/scholar?q=Angels+Balaguer%2C+Vinamra+Benara%2C+Renato+Luiz+de+Freitas+Cunha%2C+Roberto+de+M%2E+Estev%C3%A3o+Filho%2C+Todd+Hendry%2C+Daniel+Holstein%2C+Jennifer+Marsman%2C+Nick+Mecklenburg%2C+Sara+Malvar%2C+Leonardo+O%2E+Nunes%2C+Rafael+Pad)
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Wilker+Aziz%2C+and+Ivan+Titov%2E+2021%2E+Editing+factual+knowledge+in+language+models%2E)
8. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [https://github.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca). [↗](https://github.com/sahil280114/codealpaca)
9. Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cu. 2023. Lifelong language pretraining with distribution-specialized experts. [find](https://scholar.google.com/scholar?q=Wuyang+Chen%2C+Yanqi+Zhou%2C+Nan+Du%2C+Yanping+Huang%2C+James+Laudon%2C+Zhifeng+Chen%2C+and+Claire+Cu%2E+2023%2E+Lifelong+language+pretraining+with+distribution%2Dspecialized+experts%2E)
10. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. [find](https://scholar.google.com/scholar?q=Jiaxi+Cui%2C+Zongjian+Li%2C+Yang+Yan%2C+Bohua+Chen%2C+and+Li+Yuan%2E+2023%2E+Chatlaw%3A+Open%2Dsource+legal+large+language+model+with+integrated+external+knowledge+bases%2E)
11. Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. 2023. [Language modeling is compression](https://md.research.elamur.ai/free/en/papers/deletang-2023-language-modeling-compression/).
12. Leon Derczynski. 2016. Complementarity, F-score, and NLP evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 261–266, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Leon+Derczynski%2E+2016%2E+Complementarity%2C+F%2Dscore%2C+and+NLP+evaluation%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+pages+261%E2%80%93266%2C+Portoro%C5%BE%2C+Slove)
13. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. [Time-aware language models as temporal knowledge bases](https://md.research.elamur.ai/free/en/papers/dhingra-2022-time-aware-language-models-as/). Transactions of the Association for Computational Linguistics, 10:257–273.
14. Ning Ding, Yulin Chen, Bokai Xu, Shengding Hu, Yujia Qin, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Ultrachat: A large-scale auto-generated multi-round dialogue data. [https://github.com/](https://github.com/) thunlp/ultrachat. [↗](https://github.com/)
15. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. [Ragas: Automated evaluation of retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/).
16. Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation. [find](https://scholar.google.com/scholar?q=Tao+Fang%2C+Shu+Yang%2C+Kaixin+Lan%2C+Derek+F%2E+Wong%2C+Jinpeng+Hu%2C+Lidia+S%2E+Chao%2C+and+Yue+Zhang%2E+2023%2E+Is+chatgpt+a+highly+fluent+grammatical+error+correction+system%3F+a+comprehensive+evaluation%2E)
17. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2024. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/).
18. Alexander Gepperth and Cem Karaoguz. 2016. A bio-inspired incremental learning architecture for applied perceptual problems. Cognitive Computation, 8(5):924–934. [find](https://scholar.google.com/scholar?q=Alexander+Gepperth+and+Cem+Karaoguz%2E+2016%2E+A+bio%2Dinspired+incremental+learning+architecture+for+applied+perceptual+problems%2E+Cognitive+Computation%2C+8%285%29%3A924%E2%80%93934%2E)
19. Google. 2023. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/).
20. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. Studying large language model general… [find](https://scholar.google.com/scholar?q=Roger+Grosse%2C+Juhan+Bae%2C+Cem+Anil%2C+Nelson+Elhage%2C+Alex+Tamkin%2C+Amirhossein+Tajdini%2C+Benoit+Steiner%2C+Dustin+Li%2C+Esin+Durmus%2C+Ethan+Perez%2C+Evan+Hubinger%2C+Kamile%CC%87+Luko%C5%A1iu%CC%84te%CC%87%2C+Karina+Nguyen%2C+Nicholas+Jos)
21. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki)
22. Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Mike+Lewis%2C+Ari+Holtzman%2C+Noah+A%2E+Smith%2C+and+Luke+Zettlemoyer%2E+2021%2E+Demix+layers%3A+Disentangling+domains+for+modular+language+modeling%2E)
23. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E)
24. Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023. [Are large language model-based evaluators the solution to scaling up multilingual evaluation?](https://md.research.elamur.ai/free/en/papers/hada-2023-large-language-model-based-evaluators/)
25. Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Thomas+Hartvigsen%2C+Swami+Sankaranarayanan%2C+Hamid+Palangi%2C+Yoon+Kim%2C+and+Marzyeh+Ghassemi%2E+2023%2E+Aging+with+grace%3A+Lifelong+model+editing+with+discrete+key%2Dvalue+adaptors%2E+In+Advances+in+Neural+Informa)
26. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. [find](https://scholar.google.com/scholar?q=Peter+Hase%2C+Mohit+Bansal%2C+Been+Kim%2C+and+Asma+Ghandeharioun%2E+2023%2E+Does+localization+inform+editing%3F+surprising+differences+in+causality%2Dbased+localization+vs%2E+knowledge+editing+in+language+models%2E)
27. Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan. 2019. Memory efficient experience replay for streaming learning. [find](https://scholar.google.com/scholar?q=Tyler+L%2E+Hayes%2C+Nathan+D%2E+Cahill%2C+and+Christopher+Kanan%2E+2019%2E+Memory+efficient+experience+replay+for+streaming+learning%2E)
28. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/).
29. Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Kaihuo Zhang, Yuxiang Huang, Zhenning Dai, Baitao Gong, Chongyi Wang, Yuan Yao, Jie Zhou, Jie Cai, Xinrong Zhang, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong… [find](https://scholar.google.com/scholar?q=Shengding+Hu%2C+Yuge+Tu%2C+Xu+Han%2C+Ganqu+Cui%2C+Chaoqun+He%2C+Weilin+Zhao%2C+Xiang+Long%2C+Zhi+Zheng%2C+Yewei+Fang%2C+Kaihuo+Zhang%2C+Yuxiang+Huang%2C+Zhenning+Dai%2C+Baitao+Gong%2C+Chongyi+Wang%2C+Yuan+Yao%2C+Jie+Zhou%2C+Jie+Cai%2C)
30. Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. 2023. [Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/). arXiv preprint arXiv:2304.01933.
31. Cynthia Huang, Yuqing Xie, Zhiying Jiang, Jimmy Lin, and Ming Li. 2023. Approximating human-like few-shot learning with gpt-based compression. [find](https://scholar.google.com/scholar?q=Cynthia+Huang%2C+Yuqing+Xie%2C+Zhiying+Jiang%2C+Jimmy+Lin%2C+and+Ming+Li%2E+2023%2E+Approximating+human%2Dlike+few%2Dshot+learning+with+gpt%2Dbased+compression%2E)
32. Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63–S63. [find](https://scholar.google.com/scholar?q=Fred+Jelinek%2C+Robert+L+Mercer%2C+Lalit+R+Bahl%2C+and+James+K+Baker%2E+1977%2E+Perplexity%E2%80%94a+measure+of+the+difficulty+of+speech+recognition+tasks%2E+The+Journal+of+the+Acoustical+Society+of+America%2C+62%28S1%29%3AS63%E2%80%93S)
33. Yunjie Ji, Yong Deng, Yan Gong, Yiping Penga, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. 2023. [Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases](https://md.research.elamur.ai/free/en/papers/ji-2023-exploring-impact-instruction-data/). arXiv preprint arXiv:2303.14742.
34. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
35. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. [Challenges and applications of large language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/).
36. Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Yuille. 2021. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. [find](https://scholar.google.com/scholar?q=Prakhar+Kaushik%2C+Alex+Gain%2C+Adam+Kortylewski%2C+and+Alan+Yuille%2E+2021%2E+Understanding+catastrophic+forgetting+and+remembering+in+continual+learning+with+optimal+relevance+mapping%2E)
37. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
38. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. [Understanding the effects of rlhf on llm generalisation and diversity](https://md.research.elamur.ai/free/en/papers/kirk-2023-understanding-effects-rlhf-llm-generalisation/).
39. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
40. Dhireesha Kudithipudi, Mario Aguilar-Simon, Jonathan Babb, Maxim Bazhenov, Douglas Blackiston, Josh Bongard, Andrew P. Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, Sandeep Madireddy, and Angel Yanguas-Gil. 2022. Biological underpinnings for lifelong learning machines. Nature Machine Intel… [find](https://scholar.google.com/scholar?q=Dhireesha+Kudithipudi%2C+Mario+Aguilar%2DSimon%2C+Jonathan+Babb%2C+Maxim+Bazhenov%2C+Douglas+Blackiston%2C+Josh+Bongard%2C+Andrew+P%2E+Brna%2C+Suraj+Chakravarthi+Raja%2C+Nick+Cheney%2C+Jeff+Clune%2C+Sandeep+Madireddy%2C+and+An)
41. Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. 2019. Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. [find](https://scholar.google.com/scholar?q=Timoth%C3%A9e+Lesort%2C+Vincenzo+Lomonaco%2C+Andrei+Stoian%2C+Davide+Maltoni%2C+David+Filliat%2C+and+Natalia+D%C3%ADaz%2DRodr%C3%ADguez%2E+2019%2E+Continual+learning+for+robotics%3A+Definition%2C+framework%2C+learning+strategies%2C+opportu)
42. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/).
43. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/).
44. Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. [find](https://scholar.google.com/scholar?q=Yun+Luo%2C+Zhen+Yang%2C+Fandong+Meng%2C+Yafu+Li%2C+Jie+Zhou%2C+and+Yue+Zhang%2E+2023%2E+An+empirical+study+of+catastrophic+forgetting+in+large+language+models+during+continual+fine%2Dtuning%2E)
45. Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043. — [arXiv:2401.17043](https://arxiv.org/abs/2401.17043) [find](https://scholar.google.com/scholar?q=Yuanjie+Lyu%2C+Zhiyu+Li%2C+Simin+Niu%2C+Feiyu+Xiong%2C+Bo+Tang%2C+Wenjin+Wang%2C+Hao+Wu%2C+Huanyong+Liu%2C+Tong+Xu%2C+and+Enhong+Chen%2E+2024%2E+Crud%2Drag%3A+A+comprehensive+chinese+benchmark+for+retrieval%2Daugmented+generatio)
46. Davide Maltoni and Vincenzo Lomonaco. 2019. Continuous learning in single-incremental-task scenarios. [find](https://scholar.google.com/scholar?q=Davide+Maltoni+and+Vincenzo+Lomonaco%2E+2019%2E+Continuous+learning+in+single%2Dincremental%2Dtask+scenarios%2E)
47. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft). [↗](https://github.com/huggingface/peft)
48. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023a. Locating and editing factual associations in gpt. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2023a%2E+Locating+and+editing+factual+associations+in+gpt%2E)
49. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. [Camel: Communicative agents for "mind" exploration of large language model society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). In Thirty-seventh Conference on Neural Information Processing Systems.
50. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023b. Mass-editing memory in a transformer. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+Arnab+Sen+Sharma%2C+Alex+Andonian%2C+Yonatan+Belinkov%2C+and+David+Bau%2E+2023b%2E+Mass%2Dediting+memory+in+a+transformer%2E)
51. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. An open source data contamination report for large language models. [find](https://scholar.google.com/scholar?q=Yucheng+Li%2C+Frank+Guerin%2C+and+Chenghua+Lin%2E+2024%2E+An+open+source+data+contamination+report+for+large+language+models%2E)
52. Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. Clues: Few-shot learning evaluation in natural language understanding. [find](https://scholar.google.com/scholar?q=Subhabrata+Mukherjee%2C+Xiaodong+Liu%2C+Guoqing+Zheng%2C+Saghar+Hosseini%2C+Hao+Cheng%2C+Greg+Yang%2C+Christopher+Meek%2C+Ahmed+Hassan+Awadallah%2C+and+Jianfeng+Gao%2E+2021%2E+Clues%3A+Few%2Dshot+learning+evaluation+in+natur)
53. Yen-Ting Lin and Yun-Nung Chen. 2023. [Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/).
54. Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Wei Cheng, Haoyu Wang, Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Jian Pei, and Liang Zhao. 2023. Domain specializ… [find](https://scholar.google.com/scholar?q=Chen+Ling%2C+Xujiang+Zhao%2C+Jiaying+Lu%2C+Chengyuan+Deng%2C+Can+Zheng%2C+Junxiang+Wang%2C+Tanmoy+Chowdhury%2C+Yun+Li%2C+Hejie+Cui%2C+Xuchao+Zhang%2C+Tianjiao+Zhao%2C+Amit+Panalkar%2C+Wei+Cheng%2C+Haoyu+Wang%2C+Yanchi+Liu%2C+Zheng)
55. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/).
56. Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. 2020. Mnemonics training: Multi-class incremental learning without forgetting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. [find](https://scholar.google.com/scholar?q=Yaoyao+Liu%2C+Yuting+Su%2C+An%2DAn+Liu%2C+Bernt+Schiele%2C+and+Qianru+Sun%2E+2020%2E+Mnemonics+training%3A+Multi%2Dclass+incremental+learning+without+forgetting%2E+In+2020+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patte)
57. Vincenzo Lomonaco and Davide Maltoni. 2017. Core50: a new dataset and benchmark for continuous object recognition. [find](https://scholar.google.com/scholar?q=Vincenzo+Lomonaco+and+Davide+Maltoni%2E+2017%2E+Core50%3A+a+new+dataset+and+benchmark+for+continuous+object+recognition%2E)
58. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022. Fast model editing at scale. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+2022%2E+Fast+model+editing+at+scale%2E)
59. Alexander New, Megan Baker, Eric Nguyen, and Gautam Vallabha. 2022. Lifelong learning metrics. [find](https://scholar.google.com/scholar?q=Alexander+New%2C+Megan+Baker%2C+Eric+Nguyen%2C+and+Gautam+Vallabha%2E+2022%2E+Lifelong+learning+metrics%2E)
60. Kai Nylund, Suchin Gururangan, and Noah A. Smith. 2023. Time is encoded in the weights of finetuned language models. [find](https://scholar.google.com/scholar?q=Kai+Nylund%2C+Suchin+Gururangan%2C+and+Noah+A%2E+Smith%2E+2023%2E+Time+is+encoded+in+the+weights+of+finetuned+language+models%2E)
61. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
62. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2024. [Fine-tuning or retrieval? comparing knowledge injection in llms](https://md.research.elamur.ai/free/en/papers/ovadia-2024-fine-tuning-or-retrieval-comparing/).
63. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
64. Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. 2023. In-context unlearning: Language models as few shot unlearners. [find](https://scholar.google.com/scholar?q=Martin+Pawelczyk%2C+Seth+Neel%2C+and+Himabindu+Lakkaraju%2E+2023%2E+In%2Dcontext+unlearning%3A+Language+models+as+few+shot+unlearners%2E)
65. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2023%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
66. Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, SM Tonmoy, Aman Chadha, Amit P Sheth, and Amitava Das. 2023. [The troubling emergence of hallucination in large language models–an extensive definition, quantification, and prescriptive remediations](https://md.research.elamur.ai/free/en/papers/rawte-2023-troubling-emergence-hallucination-large-language/). arXiv preprint arXiv:2310.04988.
67. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
68. Nils Reimers and Iryna Gurevych. 2020. [Making monolingual sentence embeddings multilingual using knowledge distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
69. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2022. Progressive neural networks. [find](https://scholar.google.com/scholar?q=Andrei+A%2E+Rusu%2C+Neil+C%2E+Rabinowitz%2C+Guillaume+Desjardins%2C+Hubert+Soyer%2C+James+Kirkpatrick%2C+Koray+Kavukcuoglu%2C+Razvan+Pascanu%2C+and+Raia+Hadsell%2E+2022%2E+Progressive+neural+networks%2E)
70. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/).
71. Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomènech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. 2018. Memory-based parameter adaptation. [find](https://scholar.google.com/scholar?q=Pablo+Sprechmann%2C+Siddhant+M%2E+Jayakumar%2C+Jack+W%2E+Rae%2C+Alexander+Pritzel%2C+Adri%C3%A0+Puigdom%C3%A8nech+Badia%2C+Benigno+Uria%2C+Oriol+Vinyals%2C+Demis+Hassabis%2C+Razvan+Pascanu%2C+and+Charles+Blundell%2E+2018%2E+Memory%2Dbased)
72. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jife… [find](https://scholar.google.com/scholar?q=Jiankai+Sun%2C+Chuanyang+Zheng%2C+Enze+Xie%2C+Zhengying+Liu%2C+Ruihang+Chu%2C+Jianing+Qiu%2C+Jiaqi+Xu%2C+Mingyu+Ding%2C+Hongyang+Li%2C+Mengzhe+Geng%2C+Yue+Wu%2C+Wenhai+Wang%2C+Junsong+Chen%2C+Zhangyue+Yin%2C+Xiaozhe+Ren%2C+Jie+Fu%2C)
73. Sebastian Thrun and Tom M Mitchell. 1995. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25–46. [find](https://scholar.google.com/scholar?q=Sebastian+Thrun+and+Tom+M+Mitchell%2E+1995%2E+Lifelong+robot+learning%2E+Robotics+and+autonomous+systems%2C+15%281%2D2%29%3A25%E2%80%9346%2E)
74. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
75. Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023a. [Survey on factuality in large language models: Knowledge, retrieval and domain-specif…](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
76. Haochun Wang, Sendong Zhao, Zewen Qiang, Zijian Li, Nuwa Xi, Yanrui Du, MuZhen Cai, Haoqiang Guo, Yuhan Chen, Haoming Xu, Bing Qin, and Ting Liu. 2023b. Knowledge-tuning large language models with structured medical knowledge bases for reliable response generation in chinese. [find](https://scholar.google.com/scholar?q=Haochun+Wang%2C+Sendong+Zhao%2C+Zewen+Qiang%2C+Zijian+Li%2C+Nuwa+Xi%2C+Yanrui+Du%2C+MuZhen+Cai%2C+Haoqiang+Guo%2C+Yuhan+Chen%2C+Haoming+Xu%2C+Bing+Qin%2C+and+Ting+Liu%2E+2023b%2E+Knowledge%2Dtuning+large+language+models+with+str)
77. Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023c. A comprehensive survey of continual learning: Theory, method and application. [find](https://scholar.google.com/scholar?q=Liyuan+Wang%2C+Xingxing+Zhang%2C+Hang+Su%2C+and+Jun+Zhu%2E+2023c%2E+A+comprehensive+survey+of+continual+learning%3A+Theory%2C+method+and+application%2E)
78. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2023d. Knowledge editing for [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/).
79. Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. [Larger language models do in-context learning differently](https://md.research.elamur.ai/free/en/papers/wei-2023-larger-language-models-context-learning/). CoRR, abs/2303.03846.
80. Cheng Wen, Xianghui Sun, Shuaijiang Zhao, Xiaoquan Fang, Liangyu Chen, and Wei Zou. 2023. Chathome: Development and evaluation of a domain-specific language model for home renovation. [find](https://scholar.google.com/scholar?q=Cheng+Wen%2C+Xianghui+Sun%2C+Shuaijiang+Zhao%2C+Xiaoquan+Fang%2C+Liangyu+Chen%2C+and+Wei+Zou%2E+2023%2E+Chathome%3A+Development+and+evaluation+of+a+domain%2Dspecific+language+model+for+home+renovation%2E)
81. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023a. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. [find](https://scholar.google.com/scholar?q=Haoran+Wu%2C+Wenxuan+Wang%2C+Yuxuan+Wan%2C+Wenxiang+Jiao%2C+and+Michael+Lyu%2E+2023a%2E+Chatgpt+or+grammarly%3F+evaluating+chatgpt+on+grammatical+error+correction+benchmark%2E)
82. Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. 2023b. Eva-kellm: A new benchmark for evaluating knowledge editing of llms. [find](https://scholar.google.com/scholar?q=Suhang+Wu%2C+Minlong+Peng%2C+Yue+Chen%2C+Jinsong+Su%2C+and+Mingming+Sun%2E+2023b%2E+Eva%2Dkellm%3A+A+new+benchmark+for+evaluating+knowledge+editing+of+llms%2E)
83. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. [find](https://scholar.google.com/scholar?q=Shitao+Xiao%2C+Zheng+Liu%2C+Peitian+Zhang%2C+and+Niklas+Muennighoff%2E+2023%2E+C%2Dpack%3A+Packaged+resources+to+advance+general+chinese+embedding%2E)
84. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… — [Effective Long-Context Scaling of Foundation Models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
85. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024a. [Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation](https://md.research.elamur.ai/free/en/papers/xu-2024-contrastive-preference-optimization-pushing-boundaries/).
86. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. [Retrieval meets long context large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/).
87. Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. 2024. [Openmoe: An early effort on open mixture-of-experts language models](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/).
88. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/).
89. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023a. [Large language model unlearning](https://md.research.elamur.ai/free/en/papers/yao-2023-large-language-model-unlearning/).
90. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023b. Editing large language models: Problems, methods, and opportunities. [find](https://scholar.google.com/scholar?q=Yunzhi+Yao%2C+Peng+Wang%2C+Bozhong+Tian%2C+Siyuan+Cheng%2C+Zhoubo+Li%2C+Shumin+Deng%2C+Huajun+Chen%2C+and+Ningyu+Zhang%2E+2023b%2E+Editing+large+language+models%3A+Problems%2C+methods%2C+and+opportunities%2E)
91. Lang Yu, Qin Chen, Jie Zhou, and Liang He. 2023. Melo: Enhancing model editing with neuron-indexed dynamic lora. [find](https://scholar.google.com/scholar?q=Lang+Yu%2C+Qin+Chen%2C+Jie+Zhou%2C+and+Liang+He%2E+2023%2E+Melo%3A+Enhancing+model+editing+with+neuron%2Dindexed+dynamic+lora%2E)
92. Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653.
93. Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker. 2023. [Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning](https://md.research.elamur.ai/free/en/papers/zadouri-2023-pushing-mixture-experts-limit-extremely/).
94. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023c. [Siren’s song in the ai ocean: A survey on hallucination in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/).
95. Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023d. How do large language models capture the ever-changing world knowledge? a review of recent advances. [find](https://scholar.google.com/scholar?q=Zihan+Zhang%2C+Meng+Fang%2C+Ling+Chen%2C+Mohammad%2DReza+Namazi%2DRad%2C+and+Jun+Wang%2E+2023d%2E+How+do+large+language+models+capture+the+ever%2Dchanging+world+knowledge%3F+a+review+of+recent+advances%2E)
96. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. [A survey of l…](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
97. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? [find](https://scholar.google.com/scholar?q=Ce+Zheng%2C+Lei+Li%2C+Qingxiu+Dong%2C+Yuxuan+Fan%2C+Zhiyong+Wu%2C+Jingjing+Xu%2C+and+Baobao+Chang%2E+2023%2E+Can+we+edit+factual+knowledge+by+in%2Dcontext+learning%3F)
98. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. A comprehensive survey on pretrained foundation models: A history from bert to c… [find](https://scholar.google.com/scholar?q=Ce+Zhou%2C+Qian+Li%2C+Chen+Li%2C+Jun+Yu%2C+Yixin+Liu%2C+Guangjing+Wang%2C+Kai+Zhang%2C+Cheng+Ji%2C+Qiben+Yan%2C+Lifang+He%2C+Hao+Peng%2C+Jianxin+Li%2C+Jia+Wu%2C+Ziwei+Liu%2C+Pengtao+Xie%2C+Caiming+Xiong%2C+Jian+Pei%2C+Philip+S%2E+Yu%2C+an)
99. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023. Evaluating large language models at evaluating instruction following. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2023%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E)
100. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023a. [Multilingual machine translation with large language models: Empirical results and analysis](https://md.research.elamur.ai/free/en/papers/zhu-2024-multilingual-machine-translation-large-language/).
101. Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. [find](https://scholar.google.com/scholar?q=Friedemann+Zenke%2C+Ben+Poole%2C+and+Surya+Ganguli%2E+2017%2E+Continual+learning+through+synaptic+intelligence%2E)
102. Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023b. [A survey on model compression for large language models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/).
103. Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, Xiaowei Zhu, Jun Zhou, and Huajun Chen. 2024a. A comprehensive… [find](https://scholar.google.com/scholar?q=Ningyu+Zhang%2C+Yunzhi+Yao%2C+Bozhong+Tian%2C+Peng+Wang%2C+Shumin+Deng%2C+Mengru+Wang%2C+Zekun+Xi%2C+Shengyu+Mao%2C+Jintian+Zhang%2C+Yuansheng+Ni%2C+Siyuan+Cheng%2C+Ziwen+Xu%2C+Xin+Xu%2C+Jia%2DChen+Gu%2C+Yong+Jiang%2C+Pengjun+Xie%2C+F)
104. Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. St-moe: Designing stable and transferable sparse expert models. [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Irwan+Bello%2C+Sameer+Kumar%2C+Nan+Du%2C+Yanping+Huang%2C+Jeff+Dean%2C+Noam+Shazeer%2C+and+William+Fedus%2E+2022%2E+St%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2E)
105. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024b. [Tinyllama: An open-source small language model](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/).
106. Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023a. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/).
107. Yue Zhang, Leyang Cui, Deng Cai, Xinting Huang, Tao Fang, and Wei Bi. 2023b. Multi-task instruction tuning of llama for specific scenarios: A preliminary study on writing assistance. [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Leyang+Cui%2C+Deng+Cai%2C+Xinting+Huang%2C+Tao+Fang%2C+and+Wei+Bi%2E+2023b%2E+Multi%2Dtask+instruction+tuning+of+llama+for+specific+scenarios%3A+A+preliminary+study+on+writing+assistance%2E)

*Updated 30 October 2026*