---
title: "LoRAMoE — paper by Dou et al. (2024) — summary and ideas"
description: "The authors build a 5-million-sample instruction set covering seven task categories and fine-tune… 43 references, cited by 34 papers here."
authors: ["Shihan Dou", "Enyu Zhou", "Yan Liu", "Songyang Gao", "Wei Shen", "Limao Xiong", "Yuhao Zhou", "Xiao Wang", "Zhiheng Xi", "Xiaoran Fan", "Shiliang Pu", "Jiang Zhu", "Rui Zheng", "Tao Gui", "Qi Zhang", "Xuanjing Huang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.106/"
doi: "10.18653/v1/2024.acl-long.106"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dou-2024-loramoe-alleviating-world-knowledge-forgetting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dou-2024-loramoe-alleviating-world-knowledge-forgetting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

# LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

**Authors:** [Shihan Dou](https://md.research.elamur.ai/free/en/authors/shihan-dou/), [Enyu Zhou](https://md.research.elamur.ai/free/en/authors/enyu-zhou/), [Yan Liu](https://md.research.elamur.ai/free/en/authors/yan-liu/), [Songyang Gao](https://md.research.elamur.ai/free/en/authors/songyang-gao/), [Wei Shen](https://md.research.elamur.ai/free/en/authors/wei-shen/), [Limao Xiong](https://md.research.elamur.ai/free/en/authors/limao-xiong/), [Yuhao Zhou](https://md.research.elamur.ai/free/en/authors/yuhao-zhou/), [Xiao Wang](https://md.research.elamur.ai/free/en/authors/xiao-wang/), [Zhiheng Xi](https://md.research.elamur.ai/free/en/authors/zhiheng-xi/), [Xiaoran Fan](https://md.research.elamur.ai/free/en/authors/xiaoran-fan/), Shiliang Pu, Jiang Zhu, [Rui Zheng](https://md.research.elamur.ai/free/en/authors/rui-zheng/), [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/) and [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.106/) · [PDF](https://aclanthology.org/2024.acl-long.106.pdf) · [DOI 10.18653/v1/2024.acl-long.106](https://doi.org/10.18653/v1/2024.acl-long.106) · [OpenAlex](https://openalex.org/W4402671841)

**Citations:** 59 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/dou-2024-loramoe-alleviating-world-knowledge-forgetting/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines a trade-off in supervised fine-tuning (SFT) of LLMs. Adding much more instruction data keeps improving downstream tasks, but it also erodes the factual knowledge the model gained in pre-training. The authors build a 5-million-sample instruction set covering seven task categories and fine-tune Llama-2-7B on it. Summarization, NLI and translation improve and then plateau, while closed-book QA (CBQA) scores fall below the original model. Re-tuning on CBQA afterwards does not restore them, so the authors argue the loss is irreversible.

To address this, they propose LoRAMoE, a Mixture-of-Experts-style plugin:
- The backbone stays frozen.
- Several LoRA adapters act as experts in the FFN layers, weighted by a trainable softmax router.
- A Localized Balancing Constraint nudges one group of experts to apply stored world knowledge and another to handle downstream tasks.

The authors report that LoRAMoE improves multi-task performance as instruction data grows while largely preserving world knowledge. Router visualizations show that each expert group is favored on the task types it was meant for. This matters for anyone scaling SFT data without wanting to sacrifice the model's factual recall.

## Key points
- **Forgetting setup:** Llama-2-7B was fine-tuned on up to 5M samples spanning seven task categories. Knowledge was measured on TriviaQA, Natural Questions and HotpotQA, plus Filtered NQ and Filtered TriviaQA, which the authors filtered to remove train-test overlap.
- **Two-stage evidence of lasting damage:** The model was first tuned on 3M instructions without CBQA, then tuned on CBQA. It scored 13.7 / 3.6 / 7.1 on TriviaQA / NQ / HotpotQA. The untuned baseline scored 33.5 / 7.8 / 11.2, and CBQA-only tuning scored 36.22 / 12.8 / 16.1.
- **Knowledge comes from pre-training:** When tuning only on CBQA (250k samples), almost all of the gain arrives within roughly the first 1k examples. The authors read this as alignment of existing knowledge rather than acquisition of new facts.
- **Architecture:** Each FFN linear layer computes W₀x plus a router-weighted sum of LoRA experts, with the router defined as G(x) = Softmax(xW_g). Only the experts and the router are trained.
- **Expert routing and limitations:** Knowledge-oriented experts dominate on CBQA benchmarks, but some downstream tasks still draw on them; for example, about 45% of router weight goes to them on WSC. Experiments are limited to 7B models and to two expert types.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 167 works · 20,987 citations

Also on the byline: [Hikvision](https://ror.org/02jzypx27).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang and Xuanjing Huang. 2024. LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dou2024loramoe,
  title = {{LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin}},
  author = {Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang and Xuanjing Huang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.106},
  url = {https://doi.org/10.18653/v1/2024.acl-long.106}
}
```

- [← previous LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/)
- [next → Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031. [find](https://scholar.google.com/scholar?q=Badr+AlKhamissi%2C+Millicent+Li%2C+Asli+Celikyilmaz%2C+Mona+Diab%2C+and+Marjan+Ghazvininejad%2E+2022%2E+A+review+on+language+models+as+knowledge+bases%2E+arXiv+preprint+arXiv%3A2204%2E06031%2E)
2. Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang, Yu Qiao, and Jing Shao. 2023b. Octavius: Mitigating task interference in mllms via moe. arXiv preprint arXiv:2311.02684. — [arXiv:2311.02684](https://arxiv.org/abs/2311.02684) [find](https://scholar.google.com/scholar?q=Zeren+Chen%2C+Ziqin+Wang%2C+Zhen+Wang%2C+Huayang+Liu%2C+Zhenfei+Yin%2C+Si+Liu%2C+Lu+Sheng%2C+Wanli+Ouyang%2C+Yu+Qiao%2C+and+Jing+Shao%2E+2023b%2E+Octavius%3A+Mitigating+task+interference+in+mllms+via+moe%2E+arXiv+preprint+arXi)
3. Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, et al. 2022. On the representation collapse of sparse mixture of experts. Advances in Neural Information Processing Systems, 35:34600–34613. [find](https://scholar.google.com/scholar?q=Zewen+Chi%2C+Li+Dong%2C+Shaohan+Huang%2C+Damai+Dai%2C+Shuming+Ma%2C+Barun+Patra%2C+Saksham+Singhal%2C+Payal+Bajaj%2C+Xia+Song%2C+Xian%2DLing+Mao%2C+et+al%2E+2022%2E+On+the+representation+collapse+of+sparse+mixture+of+experts%2E+)
4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+a)
5. Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. [Glam: Efficient scaling of language models with mixture-of-experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/). In International Conference on Machine Learning, pages 5547–5569. PMLR.
6. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv: Learning,arXiv: Learning. [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Barret+Zoph%2C+and+Noam+Shazeer%2E+2021%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv%3A+Learning%2CarXiv%3A+Learning%2E)
7. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. [The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/). arXiv preprint arXiv:1902.01382.
8. Moonsu Han, Minki Kang, Hyunwoo Jung, and Sung Ju Hwang. 2019. Episodic memory reader: Learning what to remember for question answering from streaming data. arXiv preprint arXiv:1903.06164. [find](https://scholar.google.com/scholar?q=Moonsu+Han%2C+Minki+Kang%2C+Hyunwoo+Jung%2C+and+Sung+Ju+Hwang%2E+2019%2E+Episodic+memory+reader%3A+Learning+what+to+remember+for+question+answering+from+streaming+data%2E+arXiv+preprint+arXiv%3A1903%2E06164%2E)
9. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). Cornell University - arXiv,Cornell University - arXiv.
10. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
11. Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290. [find](https://scholar.google.com/scholar?q=Yihan+Cao%2C+Yanbin+Kang%2C+and+Lichao+Sun%2E+2023%2E+Instruction+mining%3A+High%2Dquality+instruction+data+selection+for+large+language+models%2E+arXiv+preprint+arXiv%3A2307%2E06290%2E)
12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
13. Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2023a. Punica: Multi-tenant lora serving. arXiv preprint arXiv:2310.18547. [find](https://scholar.google.com/scholar?q=Lequn+Chen%2C+Zihao+Ye%2C+Yongji+Wu%2C+Danyang+Zhuo%2C+Luis+Ceze%2C+and+Arvind+Krishnamurthy%2E+2023a%2E+Punica%3A+Multi%2Dtenant+lora+serving%2E+arXiv+preprint+arXiv%3A2310%2E18547%2E)
14. Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. 2023. [Lorahub: Efficient cross-task generalization via dynamic lora composition](https://md.research.elamur.ai/free/en/papers/huang-2023-lorahub-efficient-cross-task-generalization/). arXiv preprint arXiv:2307.13269.
15. Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation, 3(1):79–87. [find](https://scholar.google.com/scholar?q=Robert+A+Jacobs%2C+Michael+I+Jordan%2C+Steven+J+Nowlan%2C+and+Geoffrey+E+Hinton%2E+1991%2E+Adaptive+mixtures+of+local+experts%2E+Neural+computation%2C+3%281%29%3A79%E2%80%9387%2E)
16. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. [Looking beyond the surface: A challenge set for reading comprehension over multiple sentences](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:…
17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
18. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). arXiv preprint arXiv:1704.04683.
19. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+2020%2E+Gshard%3A+Scaling+giant+models+with+conditional+computation+and)
20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059.
21. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+winograd+schema+challenge%2E+In+Thirteenth+international+conference+on+the+principles+of+knowledge+representation+and+reasoning%2E)
22. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2020. Question and answer test-train overlap in open-domain question answering datasets. arXiv preprint arXiv:2008.02637. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2020%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+arXiv+preprint+arXiv%3A2008%2E02637%2E)
23. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. [Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/). Advances in Neural Information Processing Systems, 35:1950–1965.
24. Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2023. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint arXiv:2310.18339. — [arXiv:2310.18339](https://arxiv.org/abs/2310.18339) [find](https://scholar.google.com/scholar?q=Qidong+Liu%2C+Xian+Wu%2C+Xiangyu+Zhao%2C+Yuanshao+Zhu%2C+Derong+Xu%2C+Feng+Tian%2C+and+Yefeng+Zheng%2E+2023%2E+Moelora%3A+An+moe%2Dbased+parameter+efficient+fine%2Dtuning+method+for+multi%2Dtask+medical+applications%2E+arXiv+p)
25. Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, et al. 2020. The microsoft toolkit of [multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). arXiv preprint arXiv:2002.07972.
26. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.
27. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2022. [Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering](https://md.research.elamur.ai/free/en/papers/neeman-2023-disentqa-disentangling-parametric-contextual-knowledge/). arXiv preprint arXiv:2211.05655.
28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
29. Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. 2022. St-adapter: Parameter-efficient image-to-video transfer learning. Advances in Neural Information Processing Systems, 35:26462–26477. [find](https://scholar.google.com/scholar?q=Junting+Pan%2C+Ziyi+Lin%2C+Xiatian+Zhu%2C+Jing+Shao%2C+and+Hongsheng+Li%2E+2022%2E+St%2Dadapter%3A+Parameter%2Defficient+image%2Dto%2Dvideo+transfer+learning%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A26462%E2%80%93264)
30. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) arXiv preprint arXiv:1909.01066.
31. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. 2019. [Answering complex open-domain questions through iterative query generation](https://md.research.elamur.ai/free/en/papers/qi-2019-answering-complex-open-domain-questions/). arXiv preprint arXiv:1910.07000.
32. Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595. [find](https://scholar.google.com/scholar?q=Carlos+Riquelme%2C+Joan+Puigcerver%2C+Basil+Mustafa%2C+Maxim+Neumann%2C+Rodolphe+Jenatton%2C+Andr%C3%A9+Susano+Pinto%2C+Daniel+Keysers%2C+and+Neil+Houlsby%2E+2021%2E+Scaling+vision+with+sparse+mixture+of+experts%2E+Advances+i)
33. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+arXiv+preprint+arXiv%3A2002%2E08910%2E)
34. Amrita Saha, Rahul Aralikatte, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018. [Duorc: Towards complex language understanding with paraphrased reading comprehension](https://md.research.elamur.ai/free/en/papers/saha-2018-duorc-complex-language-understanding-paraphrased/). arXiv preprint arXiv:1804.07927.
35. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
36. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2016. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2016%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+In+Inter)
37. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al. 2023. S-lora: Serving thousands of concurrent lora adapters. arXiv preprint arXiv:2311.03285. [find](https://scholar.google.com/scholar?q=Ying+Sheng%2C+Shiyi+Cao%2C+Dacheng+Li%2C+Coleman+Hooper%2C+Nicholas+Lee%2C+Shuo+Yang%2C+Christopher+Chou%2C+Banghua+Zhu%2C+Lianmin+Zheng%2C+Kurt+Keutzer%2C+et+al%2E+2023%2E+S%2Dlora%3A+Serving+thousands+of+concurrent+lora+adapte)
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
39. Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. 2023a. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152. [find](https://scholar.google.com/scholar?q=Xiao+Wang%2C+Tianze+Chen%2C+Qiming+Ge%2C+Han+Xia%2C+Rong+Bao%2C+Rui+Zheng%2C+Qi+Zhang%2C+Tao+Gui%2C+and+Xuanjing+Huang%2E+2023a%2E+Orthogonal+subspace+learning+for+language+model+continual+learning%2E+arXiv+preprint+arXiv%3A)
40. Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023b. [Instructuie: Multi-task instruction tuning for unified information extraction](https://md.research.elamur.ai/free/en/papers/wang-2023-instructuie-multi-task-instruction-tuning/). arXiv preprint arXiv:2304.08085.
41. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+2022%2E+Super%2Dnaturalinst)
42. Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. [Bilateral multi-perspective matching for natural language sentences](https://md.research.elamur.ai/free/en/papers/wang-2017-bilateral-multi-perspective-matching-natural/). arXiv preprint arXiv:1702.03814.
43. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. [Record: Bridging the gap between human and machine commonsense reading comprehension](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/). arXiv preprint arXiv:1810.12885.

*Updated 29 October 2026*