--- title: "Efficient models & compression — papers" description: "Efficient models & compression: 60 papers of the Elamur Research library, the most cited first." canonical: "https://research.elamur.ai/free/en/topics/efficient-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/efficient-models/) [home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Efficient models & compression # 🪶 Efficient models & compression 60 papers · page 1 of 3 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/) Smaller and faster models: quantization, pruning, distillation and sparse computation. 💡 Explained: [Minimum description length](https://md.research.elamur.ai/free/en/concepts/minimum-description-length/) · [Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) **1** [2](https://md.research.elamur.ai/free/en/topics/efficient-models/page/2/) [3](https://md.research.elamur.ai/free/en/topics/efficient-models/page/3/) [next →](https://md.research.elamur.ai/free/en/topics/efficient-models/page/2/) - [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) — Edward J. Hu, Yelong Shen, Phillip Wallis, et al. · 2021 · 5 972 citations - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. · 2023 · 2 738 citations - [Efficient Transformers: A Survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/) — Yi Tay, Mostafa Dehghani, Dara Bahri, et al. · 2020 · 1 513 citations - [Sequence-Level Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/) — Yoon Kim, Alexander M. Rush · 2016 · 1 440 citations - [FastText.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/) — Armand Joulin, Edouard Grave, Piotr Bojanowski, et al. · 2016 · 1 209 citations - [Efficient Low-rank Multimodal Fusion With Modality-Specific Factors](https://md.research.elamur.ai/free/en/papers/liu-2018-efficient-low-rank-multimodal-fusion/) — Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, et al. · 2018 · 1 174 citations - [Patient Knowledge Distillation for BERT Model Compression](https://md.research.elamur.ai/free/en/papers/sun-2019-patient-knowledge-distillation-bert-model/) — Siqi Sun, Yu Cheng, Zhe Gan, et al. · 2019 · 939 citations - [Reformer: The Efficient Transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/) — Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya · 2020 · 774 citations - [Efficient Content-Based Sparse Attention with Routing Transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/) — Aurko Roy, Mohammad Saffar, Ashish Vaswani, et al. · 2020 · 660 citations - [LLM-Pruner: On the Structural Pruning of Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/) — Xinyin Ma, Gongfan Fang, Xinchao Wang · 2023 · 447 citations - [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://md.research.elamur.ai/free/en/papers/gordon-2020-compressing-bert-studying-effects-weight/) — Mitchell Gordon, Kevin Duh, Nicholas Andrews · 2020 · 400 citations - [Long Range Arena: A Benchmark for Efficient Transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/) — Yi Tay, Mostafa Dehghani, Samira Abnar, et al. · 2020 · 396 citations - [A Survey on Model Compression for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2024-survey-model-compression-large-language/) — Xunyu Zhu, Jian Li, Yong Liu, et al. · 2024 · 381 citations - [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/) — Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al. · 2024 · 355 citations - [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/) — Demi Guo, Alexander Rush, Yoon Kim · 2021 · 355 citations - [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/) — Peter West, Chandra Bhagavatula, Jack Hessel, et al. · 2022 · 343 citations - [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/) — Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, et al. · 2023 · 335 citations - [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/) — Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, et al. · 2022 · 301 citations - [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://md.research.elamur.ai/free/en/papers/hooper-2024-kvquant-towards-10-million-context/) — Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, et al. · 2024 · 293 citations - [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/) — Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, et al. · 2023 · 288 citations - [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2024-llm-qat-data-free-quantization/) — Zechun Liu, Barlas Oguz, Changsheng Zhao, et al. · 2024 · 275 citations - [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/) — Zirui Liu, Jiayi Yuan, Hongye Jin, et al. · 2024 · 243 citations - [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/) — Zeyu Han, Chao Gao, Jinyang Liu, et al. · 2024 · 236 citations - [Structured Pruning Learns Compact and Accurate Models](https://md.research.elamur.ai/free/en/papers/xia-2022-structured-pruning-learns-compact-accurate/) — Mengzhou Xia, Zexuan Zhong, Danqi Chen · 2022 · 216 citations - [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/) — Yucheng Li, Bo Dong, Frank Guerin, et al. · 2023 · 208 citations **1** [2](https://md.research.elamur.ai/free/en/topics/efficient-models/page/2/) [3](https://md.research.elamur.ai/free/en/topics/efficient-models/page/3/) [next →](https://md.research.elamur.ai/free/en/topics/efficient-models/page/2/) - [← previous topic Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) - [next topic → Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) - [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)