--- title: "Recurrent Knowledge Identification and Fusion for Language" description: "The authors propose Recurrent-KIF, inspired by the Complementary Learning Systems theory of… 1 reference, cited by 9 papers here." authors: ["Yujie Feng", "Xujia Wang", "Zexin Lu", "Shenghong Fu", "Guangyuan Shi", "Yongxin Xu", "Yasha Wang", "Philip S. Yu", "Xu Chu", "Xiao-Ming Wu"] year: 2025 venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.acl-long.1328/" doi: "10.18653/v1/2025.acl-long.1328" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/feng-2025-recurrent-knowledge-identification-fusion-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/feng-2025-recurrent-knowledge-identification-fusion-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Recurrent Knowledge Identification and Fusion for Language Model Continual Learning # Recurrent Knowledge Identification and Fusion for Language Model Continual Learning **Authors:** [Yujie Feng](https://md.research.elamur.ai/free/en/authors/yujie-feng/), Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, [Yongxin Xu](https://md.research.elamur.ai/free/en/authors/yongxin-xu/), [Yasha Wang](https://md.research.elamur.ai/free/en/authors/yasha-wang/), [Philip S. Yu](https://md.research.elamur.ai/free/en/authors/philip-s-yu/), [Xu Chu](https://md.research.elamur.ai/free/en/authors/xu-chu/) and [Xiao-Ming Wu](https://md.research.elamur.ai/free/en/authors/xiao-ming-wu/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1328/) · [PDF](https://aclanthology.org/2025.acl-long.1328.pdf) · [DOI 10.18653/v1/2025.acl-long.1328](https://doi.org/10.18653/v1/2025.acl-long.1328) · [OpenAlex](https://openalex.org/W4412889804) **Citations:** 2 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/feng-2025-recurrent-knowledge-identification-fusion-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This paper tackles continual learning for large language models, where a model must absorb tasks one after another without losing earlier skills (catastrophic forgetting) while still benefiting from links between tasks (knowledge transfer). Earlier merging approaches decide which parameters matter using importance scores computed once per task. As training moves the weights, those fixed scores become less reliable. The authors propose Recurrent-KIF, inspired by the Complementary Learning Systems theory of hippocampus and neocortex. Training is split into repeated cycles with two loops: - **Inner loop:** quickly adapts to the new task and flags the parameters that matter for it. - **Outer loop:** re-estimates importance for earlier tasks from a small replay buffer, using the current weights. It then fuses the inner and outer task vectors with binary masks that prune redundant updates and merge the key ones, including shared regions. Because fusion happens many times during training rather than once at the end, the method can use intermediate states. On two CL benchmarks, with backbones from 770M to 13B parameters, it outperformed state-of-the-art baselines. ## Key points - **Model scale:** tested on backbones from 770M (T5-large) to 13B parameters, including LLaMA2-7B. Tasks include IMDB, AG News, Amazon and Copa. On IMDB and AG News, results approach multi-task learning. - **Ablation on the Long Sequence Benchmark (order 1):** the full model reaches OP 77.9 and BWT −3.4. - With static importance (−DIE), scores drop to 74.8 / −4.8. - Merging without knowledge identification collapses to 52.3 / −21.5. - **Alternative fusion schemes:** global merging gives 72.1 / −11.2 and soft adaptive fusion gives 76.1 / −4.1. Freezing task-shared regions gives 75.8 / −4.3. - **Multi-round fusion:** it beats single-step fusion, but gains level off and then decline when fusion happens too often. - **Visualizations on T5-large:** large update magnitude does not imply importance, with mostly encoder parameters being critical. Importance scores for earlier tasks shift as training goes on. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations - [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,268 citations - [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 70 works · 20,363 citations - [Hong Kong Polytechnic University](https://md.research.elamur.ai/free/en/universities/hong-kong-polytechnic-university/) — University, Hong Kong, China · 106 works · 5,190 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S. Yu, Xu Chu and Xiao-Ming Wu. 2025. Recurrent Knowledge Identification and Fusion for Language Model Continual Learning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{feng2025recurrent, title = {{Recurrent Knowledge Identification and Fusion for Language Model Continual Learning}}, author = {Yujie Feng, Xujia Wang, Zexin Lu, Shenghong Fu, Guangyuan Shi, Yongxin Xu, Yasha Wang, Philip S. Yu, Xu Chu and Xiao-Ming Wu}, booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.acl-long.1328}, url = {https://doi.org/10.18653/v1/2025.acl-long.1328} } ``` - [← previous ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning](https://md.research.elamur.ai/free/en/papers/sun-2025-reasonmed-370k-multi-agent-generated/) - [next → Retrofitting Large Language Models with Dynamic Tokenization](https://md.research.elamur.ai/free/en/papers/feher-2025-retrofitting-large-language-models-dynamic/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 15(3):1–45. \#\# Conclusion *Updated 2 November 2026*