---
title: "GRASPrune — paper by Wang et al. (2026) — summary and ideas"
description: "Serving large language models is costly because weights, attention computation and the KV cache all consume memory and… 49 references."
authors: ["Ziyang Wang", "Jiangfeng Xiao", "Chuan Xiao", "Ruoxiang LI", "Rui Mao", "Jianbin Qin"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.491/"
doi: "10.18653/v1/2026.acl-long.491"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-grasprune-global-gating-budgeted-structured/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-grasprune-global-gating-budgeted-structured/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/) / GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

# GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models

**Authors:** [Ziyang Wang](https://md.research.elamur.ai/free/en/authors/ziyang-wang/), Jiangfeng Xiao, [Chuan Xiao](https://md.research.elamur.ai/free/en/authors/chuan-xiao/), Ruoxiang LI, [Rui Mao](https://md.research.elamur.ai/free/en/authors/rui-mao/) and [Jianbin Qin](https://md.research.elamur.ai/free/en/authors/jianbin-qin/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.491/) · [PDF](https://aclanthology.org/2026.acl-long.491.pdf) · [DOI 10.18653/v1/2026.acl-long.491](https://doi.org/10.18653/v1/2026.acl-long.491) · [OpenAlex](https://openalex.org/W7166839910)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Serving large language models is costly because weights, attention computation and the KV cache all consume memory and add latency. GRASPrune is a post-training structured pruning method that removes FFN channels and KV head groups together under one shared parameter budget. Earlier pipelines typically score units first and enforce the budget afterwards. GRASPrune instead keeps the budget active throughout optimization. It learns only scalar gates, with the backbone frozen, using a projected straight-through estimator. At every training step, the gates are turned into a hard mask that already satisfies the budget. Each KV head group receives a cost weight, so it can be compared fairly with FFN channels. Once the mask is fixed, per-unit scaling factors are calibrated and folded into the sliced weights. The result is a smaller dense checkpoint with no extra inference parameters. On LLaMA-2-7B at 50% parameter retention, the pruned model reaches 12.18 WikiText-2 perplexity with competitive zero-shot accuracy, and no full fine-tuning is needed. This makes compression cheap and directly compatible with standard serving stacks.

## Key points
- **Training cost:** gate learning uses 512 unlabeled calibration sequences for four epochs on a single NVIDIA A100 80GB GPU, taking roughly 6 minutes including calibration.
- **Perplexity on LLaMA-2-7B (WikiText-2):** 12.18 at 50% parameter retention and 16.65 at 40%. Baselines include LLM-Pruner, SliceGPT and FLAP; LLaMA-2-13B is also analyzed.
- **Ranking ablation:** at 80% retention, ranking units by value-per-cost (p/c) instead of the learned score raises WikiText-2 perplexity from 6.47 to 9.37. It also tilts the allocation toward cheaper FFN units.
- **Scaling-stage ablation:** calibrated scaling helps most under heavy pruning. At 40% retention, WikiText-2 perplexity drops from 20.40 to 16.65 and average zero-shot accuracy rises to 0.406.
- **Serving benefits:** removing KV head groups cuts KV-cache and peak prefill memory roughly in proportion. Throughput gains grow with batch size, measured with a 2048-token prompt and 256 generated tokens. Mid-depth layers lose the most KV heads.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beijing Institute of Technology](https://md.research.elamur.ai/free/en/universities/beijing-institute-of-technology/) — University, Beijing, China · 34 works · 3,652 citations
- [The University of Osaka](https://md.research.elamur.ai/free/en/universities/the-university-of-osaka/) — University, Osaka, Japan · 15 works · 517 citations
- [Shenzhen University](https://md.research.elamur.ai/free/en/universities/shenzhen-university/) — University, Shenzhen, China · 16 works · 95 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang LI, Rui Mao and Jianbin Qin. 2026. GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026grasprune,
  title = {{GRASPrune: Global Gating for Budgeted Structured Pruning of Large Language Models}},
  author = {Ziyang Wang, Jiangfeng Xiao, Chuan Xiao, Ruoxiang LI, Rui Mao and Jianbin Qin},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.491},
  url = {https://doi.org/10.18653/v1/2026.acl-long.491}
}
```

- [← previous GraphRAG-Rad: Concept-Aware Radiology Report Generation via Latent Visual-Semantic Retrieval](https://md.research.elamur.ai/free/en/papers/safari-2026-graphrag-rad-concept-aware-radiology/)
- [next → GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuning](https://md.research.elamur.ai/free/en/papers/tian-2026-grass-gradient-based-adaptive-layer/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/32/)

## References

49 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2024. Fluctuation-based adaptive structured pruning for large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10):10865–10873. [find](https://scholar.google.com/scholar?q=Yongqi+An%2C+Xu+Zhao%2C+Tao+Yu%2C+Ming+Tang%2C+and+Jinqiao+Wang%2E+2024%2E+Fluctuation%2Dbased+adaptive+structured+pruning+for+large+language+models%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+3)
2. Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Saleh+Ashkboos%2C+Maximilian+L%2E+Croci%2C+Marcelo+Gennari+do+Nascimento%2C+Torsten+Hoefler%2C+and+James+Hensman%2E+2024%2E+SliceGPT%3A+Compress+large+language+models+by+deleting+rows+and+columns%2E+In+The+Twelfth+Inte)
3. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint, arXiv:1308.3432. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+L%C3%A9onard%2C+and+Aaron+Courville%2E+2013%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+Preprint%2C+arXiv%3A1308%2E3432%2E)
4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [PIQA: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439.
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). Preprint, arXiv:2110.14168.
10. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Matthieu+Courbariaux%2C+Yoshua+Bengio%2C+and+Jean%2DPierre+David%2E+2015%2E+Binaryconnect%3A+Training+deep+neural+networks+with+binary+weights+during+propagations%2E+In+Advances+in+Neural+Information+Processing+Sys)
11. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
12. Shangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. 2024. DISP-LLM: Dimension-independent structural pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Shangqian+Gao%2C+Chi%2DHeng+Lin%2C+Ting+Hua%2C+Zheng+Tang%2C+Yilin+Shen%2C+Hongxia+Jin%2C+and+Yen%2DChang+Hsu%2E+2024%2E+DISP%2DLLM%3A+Dimension%2Dindependent+structural+pruning+for+large+language+models%2E+In+The+Thirty%2Deighth+)
13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
14. Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. 2025. SlimLLM: Accurate structured pruning for large language models. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jialong+Guo%2C+Xinghao+Chen%2C+Yehui+Tang%2C+and+Yunhe+Wang%2E+2025%2E+SlimLLM%3A+Accurate+structured+pruning+for+large+language+models%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
15. Jiujun He and Huazhen Lin. 2025. Olica: Efficient structured pruning of large language models without retraining. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Jiujun+He+and+Huazhen+Lin%2E+2025%2E+Olica%3A+Efficient+structured+pruning+of+large+language+models+without+retraining%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
17. Hanyu Hu, Pengxiang Zhao, Ping Li, Yi Zheng, Zhefeng Wang, and Xiaoming Yuan. 2025. Fasp: Fast and accurate structured pruning of large language models. Preprint, arXiv:2501.09412. [find](https://scholar.google.com/scholar?q=Hanyu+Hu%2C+Pengxiang+Zhao%2C+Ping+Li%2C+Yi+Zheng%2C+Zhefeng+Wang%2C+and+Xiaoming+Yuan%2E+2025%2E+Fasp%3A+Fast+and+accurate+structured+pruning+of+large+language+models%2E+Preprint%2C+arXiv%3A2501%2E09412%2E)
18. Eric Jang, Shixiang Gu, and Ben Poole. 2017. [Categorical reparameterization with gumbel-softmax](https://md.research.elamur.ai/free/en/papers/jang-2016-categorical-reparameterization-gumbel-softmax/). Preprint, arXiv:1611.01144.
19. Huiqiang Jiang, YUCHENG LI, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2024. [MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/). In The Thirty-eighth Annual Conferen…
20. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. [Challenges and applications of large language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/). Preprint, arXiv:2307.10169.
21. Eldar Kurtic, Elias Frantar, and Dan Alistarh. 2023. ZipLM: Inference-aware structured pruning of language models. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Eldar+Kurtic%2C+Elias+Frantar%2C+and+Dan+Alistarh%2E+2023%2E+ZipLM%3A+Inference%2Daware+structured+pruning+of+language+models%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2E)
22. Guangyan Li, Yongqiang Tang, and Wensheng Zhang. 2024a. LoRAP: Transformer sub-layers deserve differentiated structured compression for large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 28657–… [find](https://scholar.google.com/scholar?q=Guangyan+Li%2C+Yongqiang+Tang%2C+and+Wensheng+Zhang%2E+2024a%2E+LoRAP%3A+Transformer+sub%2Dlayers+deserve+differentiated+structured+compression+for+large+language+models%2E+In+Proceedings+of+the+41st+International+)
23. Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024b. [SnapKV: LLM knows what you are looking for before generation](https://md.research.elamur.ai/free/en/papers/li-2024-snapkv-llm-knows-looking-before/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
24. Gui Ling, Ziyang Wang, Yuliang Yan, and Qingwen Liu. 2024. SlimGPT: Layer-wise structured pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Gui+Ling%2C+Ziyang+Wang%2C+Yuliang+Yan%2C+and+Qingwen+Liu%2E+2024%2E+SlimGPT%3A+Layer%2Dwise+structured+pruning+for+large+language+models%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Information+Processing+Sys)
25. Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through l 0 regularization. Preprint, arXiv:1712.01312. [find](https://scholar.google.com/scholar?q=Christos+Louizos%2C+Max+Welling%2C+and+Diederik+P%2E+Kingma%2E+2018%2E+Learning+sparse+neural+networks+through+l+0+regularization%2E+Preprint%2C+arXiv%3A1712%2E01312%2E)
26. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. [LLM-Pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). In Advances in Neural Information Processing Systems, volume 36, pages 21702–21720. Curran Associates, Inc.
27. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. [The concrete distribution: A continuous relaxation of discrete random variables](https://md.research.elamur.ai/free/en/papers/maddison-2016-concrete-distribution-continuous-relaxation-discrete/). Preprint, arXiv:1611.00712.
28. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19(2):313–330.
29. Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. 2024. OSSCAR: One-shot structured pruning in vision and language models with combinatorial optimization. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedi… [find](https://scholar.google.com/scholar?q=Xiang+Meng%2C+Shibal+Ibrahim%2C+Kayhan+Behdin%2C+Hussein+Hazimeh%2C+Natalia+Ponomareva%2C+and+Rahul+Mazumder%2E+2024%2E+OSSCAR%3A+One%2Dshot+structured+pruning+in+vision+and+language+models+with+combinatorial+optimizat)
30. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). In International Conference on Learning Representations.
31. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
32. Preprint, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
33. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligenc)
34. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J+Zico+Kolter%2E+2024%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
35. Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Stella Biderman, Baber Abbasi, Jonathan Tow, ben fattori, Charles Lovering, farzanehnakhaee70, Jason Phang, Anish Thite, Fazz, Aflah, Niklas Muennighoff, Thomas Wang, sdtblck, gakada, nopperl, researcher2, and 11 others. 2024. Eleutherai/lm-evaluation-ha… [find](https://scholar.google.com/scholar?q=Lintang+Sutawika%2C+Hailey+Schoelkopf%2C+Leo+Gao%2C+Stella+Biderman%2C+Baber+Abbasi%2C+Jonathan+Tow%2C+ben+fattori%2C+Charles+Lovering%2C+farzanehnakhaee70%2C+Jason+Phang%2C+Anish+Thite%2C+Fazz%2C+Aflah%2C+Niklas+Muennighoff%2C+)
36. Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023. Structured pruning for efficient generative pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10880– 10895, Toronto, Canada. Association for Comp… [find](https://scholar.google.com/scholar?q=Chaofan+Tao%2C+Lu+Hou%2C+Haoli+Bai%2C+Jiansheng+Wei%2C+Xin+Jiang%2C+Qun+Liu%2C+Ping+Luo%2C+and+Ngai+Wong%2E+2023%2E+Structured+pruning+for+efficient+generative+pre%2Dtrained+language+models%2E+In+Findings+of+the+Associatio)
37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Preprin…
38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
40. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
41. Jeffrey T. H. Wong, Cheng Zhang, Xinye Cao, Pedro Gimenes, George A. Constantinides, Wayne Luk, and Yiren Zhao. 2025. A3 : an analytical low-rank approximation framework for attention. Preprint, arXiv:2505.12942. — [arXiv:2505.12942](https://arxiv.org/abs/2505.12942) [find](https://scholar.google.com/scholar?q=Jeffrey+T%2E+H%2E+Wong%2C+Cheng+Zhang%2C+Xinye+Cao%2C+Pedro+Gimenes%2C+George+A%2E+Constantinides%2C+Wayne+Luk%2C+and+Yiren+Zhao%2E+2025%2E+A3+%3A+an+analytical+low%2Drank+approximation+framework+for+attention%2E+Preprint%2C+arXiv)
42. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2024a. [InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory](https://md.research.elamur.ai/free/en/papers/xiao-2024-infllm-training-free-long-context/). In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
43. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024b. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024b%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representation)
44. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
45. Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FitzGerald, and Abhishek Kumar. 2025b. Wanda++: Pruning large language models via regional… [find](https://scholar.google.com/scholar?q=Yifan+Yang%2C+Kai+Zhen%2C+Bhavana+Ganesh%2C+Aram+Galstyan%2C+Goeric+Huybrechts%2C+Markus+M%C3%BCller%2C+Jonas+M%2E+K%C3%BCbler%2C+Rupak+Vignesh+Swaminathan%2C+Athanasios+Mouchtaris%2C+Sravan+Babu+Bodapati%2C+Nathan+Susanj%2C+Zheng+Zha)
46. Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. 2024. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In Proceedings… [find](https://scholar.google.com/scholar?q=Lu+Yin%2C+You+Wu%2C+Zhenyu+Zhang%2C+Cheng%2DYu+Hsieh%2C+Yaqing+Wang%2C+Yiling+Jia%2C+Gen+Li%2C+Ajay+Kumar+Jaiswal%2C+Mykola+Pechenizkiy%2C+Yi+Liang%2C+Michael+Bendersky%2C+Zhangyang+Wang%2C+and+Shiwei+Liu%2E+2024%2E+Outlier+weighe)
47. Tao Yuan, Haoli Bai, Yinfei Pan, Xuyang Cao, Tianyu Zhang, Lu Hou, Ting Hu, and Xianzhi Yu. 2025. E 3 -pruner: Towards efficient, economical, and effective layer pruning for large language models. Preprint, arXiv:2511.17205. — [arXiv:2511.17205](https://arxiv.org/abs/2511.17205) [find](https://scholar.google.com/scholar?q=Tao+Yuan%2C+Haoli+Bai%2C+Yinfei+Pan%2C+Xuyang+Cao%2C+Tianyu+Zhang%2C+Lu+Hou%2C+Ting+Hu%2C+and+Xianzhi+Yu%2E+2025%2E+E+3+%2Dpruner%3A+Towards+efficient%2C+economical%2C+and+effective+layer+pruning+for+large+language+models%2E+Pre)
48. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
49. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Thirty-seventh Conference on Neural… [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+Re%2C+Clark+Barrett%2C+Zhangyang+Wang%2C+and+Beidi+Chen%2E+2023%2E+H2o%3A+Heavy%2Dhitter+oracle+)

*Updated 1 November 2026*