--- title: "Liger — paper by Lan et al. (2025) — summary and ideas" description: "The authors also introduce Liger Attention, which combines sliding-window softmax attention with gated… 2 references, cited by 9 papers here." authors: ["Disen Lan", "Weigao Sun", "Jiaxi Hu", "Jusen Du", "Yu Cheng"] year: 2025 venue: "arXiv:2503.01496" source: "https://arxiv.org/abs/2503.01496" doi: "10.48550/arxiv.2503.01496" arxiv: "2503.01496" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Liger: Linearizing Large Language Models to Gated Recurrent Structures # Liger: Linearizing Large Language Models to Gated Recurrent Structures **Authors:** [Disen Lan](https://md.research.elamur.ai/free/en/authors/disen-lan/), [Weigao Sun](https://md.research.elamur.ai/free/en/authors/weigao-sun/), [Jiaxi Hu](https://md.research.elamur.ai/free/en/authors/jiaxi-hu/), [Jusen Du](https://md.research.elamur.ai/free/en/authors/jusen-du/) and [Yu Cheng](https://md.research.elamur.ai/free/en/authors/yu-cheng/) · **Year:** 2025 · **Venue:** arXiv:2503.01496 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2503.01496) · [PDF](https://arxiv.org/pdf/2503.01496) · [DOI 10.48550/arxiv.2503.01496](https://doi.org/10.48550/arxiv.2503.01496) · [arXiv 2503.01496](https://arxiv.org/abs/2503.01496) · [OpenAlex](https://openalex.org/W4415084856) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/lan-2025-liger-linearizing-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Transformer LLMs have attention cost that grows quadratically with sequence length, and their KV cache keeps growing during generation. Linear recurrent models avoid this with linear-time training and fixed-memory decoding, but pretraining them from scratch is expensive and risky. Earlier linearization methods (SUPRA, MambaInLlama, LoLCATs) convert pretrained Transformers instead. However, they add feature-map or gate modules that must be trained from zero, and they mostly ignore gating. Liger converts a pretrained LLM into a gated linear recurrent model without adding parameters. The existing key projection keeps its usual role and also produces gate values through a parameter-free pooling step. The converted model is then briefly fine-tuned with LoRA on an autoregressive objective. The authors also introduce Liger Attention, which combines sliding-window softmax attention with gated linear recurrence inside a single layer. On Llama-3 models from 1B to 8B, Liger keeps about 93% of the original quality using roughly 0.02% of the pretraining tokens, and it outperforms prior linearization approaches. This makes efficient recurrent LLMs practical without full pretraining. ## Key points - **Gate construction:** gates are computed from the pretrained key projection W_K via pooling, so no new gate parameters are trained. Ablations show this is competitive with randomly initialized gate projections, and that removing LoRA, GLA or SWA hurts results. - **Scale and cost:** Liger was tested on Llama-3 at 1B, 3B and 8B. It recovers about 93% of base performance at about 0.02% of pretraining tokens. The gap to the original model shrinks from 4.8% at 1B to 1.8% at 8B. By comparison, GLA-1B trained from scratch averages only 46.9%. - **Generality:** besides GLA, the method also works with other gated recurrent variants, HGRN2 and GSA. - **Efficiency:** Liger-GLA-8B was compared with Llama-3-8B, both with and without Flash-Attention-2, on a single A800 80GB GPU. The setup used batch size 16, a 128-token prefix and decoding lengths from 1K to 32K. Liger shows linear-time decoding with constant memory. - **Resources:** the paper appeared at ICML 2025. Code is at github.com/OpenSparseLLMs/Linearization, and models are in the Hugging Face collection linear-moe-hub. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du and Yu Cheng. 2025. Liger: Linearizing Large Language Models to Gated Recurrent Structures. arXiv:2503.01496. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{lan2025liger, title = {{Liger: Linearizing Large Language Models to Gated Recurrent Structures}}, author = {Disen Lan, Weigao Sun, Jiaxi Hu, Jusen Du and Yu Cheng}, year = {2025}, doi = {10.48550/arxiv.2503.01496}, eprint = {2503.01496}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2503.01496} } ``` - [← previous Language Models Fail to Introspect About Their Knowledge of Language](https://md.research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/) - [next → Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://md.research.elamur.ai/free/en/papers/shang-2025-nile-chat-egyptian-language-models/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) ## References 2 references, 2 of them with a card of ours. ↗ leads to the work. 1. Aghajanyan, A., Zettlemoyer, L., and Gupta, S. [Intrinsic dimensionality explains the effectiveness of language model fine-tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/), 2020. URL https://arxiv.org/abs/2012.13255. [↗](https://arxiv.org/abs/2012.13255) 2. Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Alberti, S., Dylan Zinsley, J. Z., Rudra, A., and Ré, C. [Simple linear attention language models balance the recall-throughput tradeoff](https://md.research.elamur.ai/free/en/papers/arora-2024-simple-linear-attention-language-models/). arXiv preprint arXiv:2402.18668, 2024. \#\# 6. Conclusion *Updated 30 October 2026*