--- title: "Who cites Sparse Upcycling: Training — 86 citing papers" description: "86 papers of the Elamur Research library cite «Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints», found in their…" canonical: "https://research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/cited-by/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/cited-by/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [Sparse Upcycling: Training Mixture-of-Experts…](https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/) / cited by # Who cites «Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints» 86 papers of our library cite it (of the 149,439 whose bibliographies we read) · 19 with a page here, the most cited first · [← back to «Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints»](https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/) 1. **[GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Joshua Ainslie, James Lee-Thorp, Michiel de Jong, et al.*2023 · 1,026 citations* 2. [**Crosslingual Generalization through Multitask Finetuning arXiv:2211.01786**Niklas Muennighoff, Thomas Wang, Lintang Sutawika, et al.*2022 · 703 citations*](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/) 3. [**MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies arXiv:2404.06395**Shengding Hu, Yuge Tu, Xu Han, et al.*2024 · 269 citations*](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/) 4. [**A Survey on Efficient Inference for Large Language Models arXiv:2404.14294**Zixuan Zhou, Xuefei Ning, Ke Hong, et al.*2024 · 113 citations*](https://md.research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/) 5. [**Arcee's MergeKit: A Toolkit for Merging Large Language Models arXiv:2403.13257**Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, et al.*2024 · 105 citations*](https://md.research.elamur.ai/free/en/papers/goddard-2024-arcee-s-mergekit-toolkit-merging/) 6. [**Retrieval Head Mechanistically Explains Long-Context Factuality arXiv:2404.15574**Wenhao Wu, Yizhong Wang, Guangxuan Xiao, et al.*2024 · 94 citations*](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/) 7. [**Accelerating Distributed MoE Training and Inference with Lina arXiv:2210.17223**Jiamin Li, Yimin Jiang, Yibo Zhu, et al.*2022 · 92 citations*](https://md.research.elamur.ai/free/en/papers/li-2022-accelerating-distributed-moe-training-inference/) 8. **[LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training](https://md.research.elamur.ai/free/en/papers/zhu-2024-llama-moe-building-mixture-experts/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Tong Zhu, Xiaoye Qu, Daize Dong, et al.*2024 · 85 citations* 9. [**SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling arXiv:2312.15166**Dahyun Kim, Chanjun Park, Sanghoon Kim, et al.*2023 · 77 citations*](https://md.research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/) 10. [**Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models arXiv:2305.14705**Sheng Shen, Le Hou, Yanqi Zhou, et al.*2023 · 70 citations*](https://md.research.elamur.ai/free/en/papers/shen-2023-mixture-experts-meets-instruction-tuning/) 11. [**Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning arXiv:2309.05444**Ted Zadouri, Ahmet Ustun, Arash Ahmadian, et al.*2023 · 69 citations*](https://md.research.elamur.ai/free/en/papers/zadouri-2023-pushing-mixture-experts-limit-extremely/) 12. [**OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models arXiv:2402.01739**Fuzhao Xue, Zian Zheng, Yao Fu, et al.*2024 · 64 citations*](https://md.research.elamur.ai/free/en/papers/xue-2024-openmoe-early-effort-open-mixture/) 13. [**Stable LM 2 1.6B Technical Report arXiv:2402.17834**Marco Bellagente, Jonathan Tow, Dakota Mahan, et al.*2024 · 46 citations*](https://md.research.elamur.ai/free/en/papers/bellagente-2024-stable-lm-2-1-6b/) 14. **[Towards the Law of Capacity Gap in Distilling Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-law-capacity-gap-distilling-language/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Chen Zhang, Qiuchi Li, Dawei Song, et al.*2025 · 27 citations* 15. [**Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding arXiv:2507.19427**StepFun, Bin Wang, Bojun Wang, et al.*2025 · 26 citations*](https://md.research.elamur.ai/free/en/papers/stepfun-2025-step-3-large-yet/) 16. **[FuseChat: Knowledge Fusion of Chat Models](https://md.research.elamur.ai/free/en/papers/wan-2025-fusechat-knowledge-fusion-chat-models/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Fanqi Wan, Longguang Zhong, Ziyi Yang, et al.*2025 · 25 citations* 17. [**Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models arXiv:2406.06563**Tianwen Wei, Bo Zhu, Liang Zhao, et al.*2024 · 21 citations*](https://md.research.elamur.ai/free/en/papers/wei-2024-skywork-moe-deep-dive/) 18. **[Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis](https://md.research.elamur.ai/free/en/papers/pei-2026-analytical-ffn-moe-restructuring-activation/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Zehua Pei, Hui-Ling Zhen, Lancheng Zou, et al.*2026 · 1 citation* 19. **[A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAMΔ Integration into Upcycled MoE](https://md.research.elamur.ai/free/en/papers/zhou-2026-data-efficient-path-multilingual-llms/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**Hao Zhou, Tianhao Li, Zhijun Wang, et al.*2026* 67 more papers of our library cite it; they have no page here yet. - [← back to](https://md.research.elamur.ai/free/en/papers/komatsuzaki-2022-sparse-upcycling-training-mixture-experts/) — Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints