---
title: "Beyond Variance — paper by Shihab et al. (2026)"
description: "The authors argue that this harms factual knowledge, which often sits in low-variance but… 5.9 points. 72 references, cited by 1 paper here."
authors: ["Ibne Farabi Shihab", "Sanjeda Akter", "Anuj Sharma"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2214/"
doi: "10.18653/v1/2026.acl-long.2214"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shihab-2026-beyond-variance-knowledge-aware-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shihab-2026-beyond-variance-knowledge-aware-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics

# Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics

**Authors:** Ibne Farabi Shihab, Sanjeda Akter and [Anuj Sharma](https://md.research.elamur.ai/free/en/authors/anuj-sharma/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2214/) · [PDF](https://aclanthology.org/2026.acl-long.2214.pdf) · [DOI 10.18653/v1/2026.acl-long.2214](https://doi.org/10.18653/v1/2026.acl-long.2214) · [OpenAlex](https://openalex.org/W7166854504)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Post-training activation compression helps fit LLMs onto limited hardware. Standard SVD-style methods keep the directions with the largest activation variance and ignore how strongly the loss depends on each direction. The authors argue that this harms factual knowledge, which often sits in low-variance but gradient-sensitive dimensions. Their method, Fisher-Aligned Subspace Compression (FASC), chooses a rank-k projection by minimizing a second-order estimate of the loss increase. This estimate is built from activation–gradient statistics and the empirical Fisher Information Matrix and is solved as a generalized eigenproblem. Randomized sketching makes it feasible for wide layers. A companion diagnostic, the Dependence Violation Score (ρ), measures how strongly activations and gradients are coupled in each layer. It shows where the extra cost of gradient-aware compression pays off. On Mistral-7B, Llama-3-8B and other models, FASC keeps 6–8% more accuracy on MMLU and LAMA than variance-based compression at 50% rank. This matters for deploying smaller models in knowledge-heavy applications, and ρ offers a way to locate stored facts without labeled probes.

## Key points
- With FASC at 50% rank reduction, a 7B model reaches the factual recall of an uncompressed 13B model. Its margin over SliceGPT and ASVD, two comparison methods from 2024, ranges from 3.7 to 5.9 points, and it remains effective when combined with GPTQ-4bit quantization.
- The authors recommend applying FASC only to layers with ρ > 0.3, estimated from n = 4096 calibration samples. In the top 20% of layers by ρ, FASC outperforms SVD by 11.8–13.5 percentage points. In the bottom 20%, the two methods perform about the same.
- Layer-wise ρ correlates with FASC's gain over SVD (r = 0.73 for Mistral-7B, r = 0.71 for Llama-3-8B). Early layers (0–10) and the final layers show low ρ.
- Dynamic sketch sizing (m = 2k for low-ρ layers, m = 4k for high-ρ layers) keeps the cost at about 1.5× that of SVD. Memory drops from 14.2GB to 8.1GB at batch size 1, but savings shrink to about 44% at B = 32.
- FASC subspaces overlap by more than 85% across the C4, WikiText and Alpaca calibration sets. On Llama-3-1B they overlap 87% with Hessian eigenvectors in high-ρ layers. Gains on BBH are smaller, at about 4 pp.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Iowa State University](https://md.research.elamur.ai/free/en/universities/iowa-state-university/) — University, Ames, United States · 8 works · 400 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

💡 Concepts (1): [💡 Latent semantic indexing](https://md.research.elamur.ai/free/en/concepts/latent-semantic-indexing/) · 29 works

## Cite

Cite as: Ibne Farabi Shihab, Sanjeda Akter and Anuj Sharma. 2026. Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shihab2026beyond,
  title = {{Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics}},
  author = {Ibne Farabi Shihab, Sanjeda Akter and Anuj Sharma},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2214},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2214}
}
```

- [← previous Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts](https://md.research.elamur.ai/free/en/papers/zheng-2026-beyond-value-benchmarks-measuring-value/)
- [next → Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations](https://md.research.elamur.ai/free/en/papers/wei-2026-bias-ear-listener-assessing-sensitivity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

72 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shun-Ichi Amari. 1998. Natural gradient works efficiently in learning. Neural computation, 10(2):251– 276. [find](https://scholar.google.com/scholar?q=Shun%2DIchi+Amari%2E+1998%2E+Natural+gradient+works+efficiently+in+learning%2E+Neural+computation%2C+10%282%29%3A251%E2%80%93+276%2E)
2. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. Neural networks: Tricks of the trade, pages 437–478. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2E+2012%2E+Practical+recommendations+for+gradient%2Dbased+training+of+deep+architectures%2E+Neural+networks%3A+Tricks+of+the+trade%2C+pages+437%E2%80%93478%2E)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901. \#\# Limitations Tianlon…
4. While we evaluate across diverse architectures (Mistral-7B, Llama-3-8B, Gemma-2-9B, Mixtral-Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [find](https://scholar.google.com/scholar?q=While+we+evaluate+across+diverse+architectures+%28Mistral%2D7B%2C+Llama%2D3%2D8B%2C+Gemma%2D2%2D9B%2C+Mixtral%2DKevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+2020%2E+Electra%3A+Pre%2Dtraining+text+encode)
5. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502. [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
6. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems, volume 35, pages 30318–30332. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+Gpt3%2E+int8+%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume)
7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). Advances in Neural Information Processing Systems, 36.
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
9. Abhimanyu Dubey, Abhinav Jauhri, Abhishek Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alex Schelten, Angela Yang, Angelina Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
10. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Transactions of the Association for Computational Linguistics, 9:1012–1031. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Shauli+Ravfogel%2C+Alon+Jacovi%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+improving+consistency+in+pretrained+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9)
11. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, and 1 others. 2021. A mathematical framework for transformer circuits. Anthropic, Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+and+1+others%2E+2021%2E+A+mathematical+framework+for+transformer+ci)
12. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323–10337. PMLR. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+In+International+Conference+on+Machine+Learning%2C+pages+10323%E2%80%9310337%2E+PMLR%2E)
13. Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783.
14. Shaopeng Guo, David Alvarez-Melis, and Diederik P Kingma. 2019. Powerful, simpler model compression. arXiv preprint arXiv:1906.04473. — [arXiv:1906.04473](https://arxiv.org/abs/1906.04473) [find](https://scholar.google.com/scholar?q=Shaopeng+Guo%2C+David+Alvarez%2DMelis%2C+and+Diederik+P+Kingma%2E+2019%2E+Powerful%2C+simpler+model+compression%2E+arXiv+preprint+arXiv%3A1906%2E04473%2E+%E2%80%94+%5BarXiv%3A1906%2E04473%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1906%2E04473%29)
15. Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. 2011. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217– 288. [find](https://scholar.google.com/scholar?q=Nathan+Halko%2C+Per%2DGunnar+Martinsson%2C+and+Joel+A+Tropp%2E+2011%2E+Finding+structure+with+randomness%3A+Probabilistic+algorithms+for+constructing+approximate+matrix+decompositions%2E+SIAM+review%2C+53%282%29%3A217%E2%80%93+288)
16. Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, volume 28. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Jeff+Pool%2C+John+Tran%2C+and+William+Dally%2E+2015%2E+Learning+both+weights+and+connections+for+efficient+neural+network%2E+In+Advances+in+neural+information+processing+systems%2C+volume+28%2E)
17. Babak Hassibi and David G Stork. 1993. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 6. [find](https://scholar.google.com/scholar?q=Babak+Hassibi+and+David+G+Stork%2E+1993%2E+Second+order+derivatives+for+network+pruning%3A+Optimal+brain+surgeon%2E+Advances+in+neural+information+processing+systems%2C+6%2E)
18. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations.
19. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. [find](https://scholar.google.com/scholar?q=Geoffrey+E+Hinton%2C+Nitish+Srivastava%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+R+Salakhutdinov%2E+2012%2E+Improving+neural+networks+by+preventing+co%2Dadaptation+of+feature+detectors%2E+arXiv+preprint+arXi)
20. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. [What does bert learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657.
21. Albert Q Jiang, Arthur Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Arthur+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+and+1+others%2E+2023%2E+)
22. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. [Convolutional sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). International conference on machine learning, pages 1243–1252.
23. Albert Q Jiang, Arthur Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma B Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Arthur+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+B+Hanna%2C+Florian+Bressand%2C+and+1+others%2E+2024%2E+Mixtral)
24. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
25. Nora Kassner, Hinrich Schütze, and Benjamin Roth. 2021. [Multilingual lama: Investigating knowledge in multilingual pretrained language models](https://md.research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/). arXiv preprint arXiv:2102.00894.
26. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Sehoon Kim, Coleman Hooper, Amir Gholami, Zhewei Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutze… — [SqueezeLLM: Dense-and-Sparse Quantization](https://md.research.elamur.ai/free/en/papers/kim-2023-squeezellm-dense-sparse-quantization/)
27. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980.
28. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. [Regularizing and optimizing lstm language models](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/). arXiv preprint arXiv:1708.02182.
29. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(1… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+and+1+others%2E+2017%2E+Overcomin)
30. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843.
31. Tom Kwiatkowski, Jennimae Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). In Transactions of the Association for Computational Li…
32. Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. 2012. Efficient backprop. Neural networks: Tricks of the trade, pages 9–50. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+L%C3%A9on+Bottou%2C+Genevieve+B+Orr%2C+and+Klaus%2DRobert+M%C3%BCller%2E+2012%2E+Efficient+backprop%2E+Neural+networks%3A+Tricks+of+the+trade%2C+pages+9%E2%80%9350%2E)
33. Yann LeCun, John S Denker, and Sara A Solla. 1989. Optimal brain damage. In Advances in neural information processing systems, volume 2. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+John+S+Denker%2C+and+Sara+A+Solla%2E+1989%2E+Optimal+brain+damage%2E+In+Advances+in+neural+information+processing+systems%2C+volume+2%2E)
34. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
35. Xiang Li, Ying Wang, Shiqing Zhang, Jiangwei Yang, Wenshuo Li, Peng Lv, Hangjie Dai, Luming Wang, Yuqing Liu, Shouhong Ding, and 1 others. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438. [find](https://scholar.google.com/scholar?q=Xiang+Li%2C+Ying+Wang%2C+Shiqing+Zhang%2C+Jiangwei+Yang%2C+Wenshuo+Li%2C+Peng+Lv%2C+Hangjie+Dai%2C+Luming+Wang%2C+Yuqing+Liu%2C+Shouhong+Ding%2C+and+1+others%2E+2022%2E+Smoothquant%3A+Accurate+and+efficient+post%2Dtraining+quant)
36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). In arXiv preprint arXiv:1907.11692.
37. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). Advances in neural information processing systems, 36.
38. Yiqiu Mao, Xuefei Chen, Xiaoxuan Ma, Yanqi Yang, Yuhan Wang, Zhenhua Wang, Xiang Shi, Qirong Zhang, Yuxuan Li, Guodong Chen, and 1 others. 2023. Activation compression of large language models for efficient serving. arXiv preprint arXiv:2312.09838. — [arXiv:2312.09838](https://arxiv.org/abs/2312.09838) [find](https://scholar.google.com/scholar?q=Yiqiu+Mao%2C+Xuefei+Chen%2C+Xiaoxuan+Ma%2C+Yanqi+Yang%2C+Yuhan+Wang%2C+Zhenhua+Wang%2C+Xiang+Shi%2C+Qirong+Zhang%2C+Yuxuan+Li%2C+Guodong+Chen%2C+and+1+others%2E+2023%2E+Activation+compression+of+large+language+models+for+eff)
39. James Martens. 2020. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1–76. [find](https://scholar.google.com/scholar?q=James+Martens%2E+2020%2E+New+insights+and+perspectives+on+the+natural+gradient+method%2E+Journal+of+Machine+Learning+Research%2C+21%28146%29%3A1%E2%80%9376%2E)
40. Per-Gunnar Martinsson and Joel A Tropp. 2020. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica, 29:403–572. [find](https://scholar.google.com/scholar?q=Per%2DGunnar+Martinsson+and+Joel+A+Tropp%2E+2020%2E+Randomized+numerical+linear+algebra%3A+Foundations+and+algorithms%2E+Acta+Numerica%2C+29%3A403%E2%80%93572%2E)
41. Tomas Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Martin+Karafi%C3%A1t%2C+Luk%C3%A1%C5%A1+Burget%2C+Jan+C%CC%8Cernocky%CC%80%2C+and+Sanjeev+Khudanpur%2E+2010%2E+Recurrent+neural+network+based+language+model%2E+In+Eleventh+Annual+Conference+of+the+International+Speech+Comm)
42. Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2015. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342. [find](https://scholar.google.com/scholar?q=Emilio+Parisotto%2C+Jimmy+Lei+Ba%2C+and+Ruslan+Salakhutdinov%2E+2015%2E+Actor%2Dmimic%3A+Deep+multitask+and+transfer+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1511%2E06342%2E)
43. Razvan Pascanu and Yoshua Bengio. 2013. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584. — [arXiv:1301.3584](https://arxiv.org/abs/1301.3584) [find](https://scholar.google.com/scholar?q=Razvan+Pascanu+and+Yoshua+Bengio%2E+2013%2E+Revisiting+natural+gradient+for+deep+networks%2E+arXiv+preprint+arXiv%3A1301%2E3584%2E+%E2%80%94+%5BarXiv%3A1301%2E3584%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1301%2E3584%29)
44. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, and 1 others. 2020. [Kilt: a benchmark for knowledge intensive language tasks](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/). arXiv preprint arXiv:2009.02252.
45. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura…
46. Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. Preprint, arXiv:24… [find](https://scholar.google.com/scholar?q=Qwen+Team%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiax)
47. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
49. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5414–5426. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
50. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in bertology: What we know about how bert works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866.
51. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
52. Wenzhe Shi, Damai Cui, Qi Li, Jiaxin Zhang, Furu Wei, and Ming Zhou. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 8493–8502. [find](https://scholar.google.com/scholar?q=Wenzhe+Shi%2C+Damai+Cui%2C+Qi+Li%2C+Jiaxin+Zhang%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+)
53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
54. Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. 2025a. Differentiable entropy regularization: A complexity-aware approach for neural optimization. arXiv preprint arXiv:2509.03733. — [arXiv:2509.03733](https://arxiv.org/abs/2509.03733) [find](https://scholar.google.com/scholar?q=Ibne+Farabi+Shihab%2C+Sanjeda+Akter%2C+and+Anuj+Sharma%2E+2025a%2E+Differentiable+entropy+regularization%3A+A+complexity%2Daware+approach+for+neural+optimization%2E+arXiv+preprint+arXiv%3A2509%2E03733%2E+%E2%80%94+%5BarXiv%3A2509%2E03)
55. Weitao Wan, Yansong Zhong, Tianpeng Li, and Jiansheng Xu. 2020. Improving neural network compression via knowledge distillation and pruning. IEEE Transactions on Knowledge and Data Engineering, 34(4):1888–1902. [find](https://scholar.google.com/scholar?q=Weitao+Wan%2C+Yansong+Zhong%2C+Tianpeng+Li%2C+and+Jiansheng+Xu%2E+2020%2E+Improving+neural+network+compression+via+knowledge+distillation+and+pruning%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+34%284%29%3A)
56. Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. 2025b. Efficient unstructured pruning of mamba state-space models for resource-constrained environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 11098–11126, Suzhou, China. Association for Co… [find](https://scholar.google.com/scholar?q=Ibne+Farabi+Shihab%2C+Sanjeda+Akter%2C+and+Anuj+Sharma%2E+2025b%2E+Efficient+unstructured+pruning+of+mamba+state%2Dspace+models+for+resource%2Dconstrained+environments%2E+In+Proceedings+of+the+2025+Conference+on+Em)
57. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+2019%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+p)
58. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+journal+of+machine+learning+)
59. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and 1 others. 2022. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261.
60. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. GitHub repository. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E+GitHub+repos)
61. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
62. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4593–4601.
63. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omar Levy, and Samuel R Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understa…](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
64. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omar Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–35…
65. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022a. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593. [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2022a%2E+Interpretability+in+the+wild%3A+a+circuit+for+indirect+object+identification+in+gpt%2D2+small%2E+arXiv+preprint+a)
66. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022b. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Wang%2C+Alexandre+Variengien%2C+Arthur+Conmy%2C+Buck+Shlegeris%2C+and+Jacob+Steinhardt%2E+2022b%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A173)
67. Yizhong Wang, Yeganeh Li, Kuangliu Liu, Yujia Huang, Yeganeh Kordi, Swaroop Mishra, Aristo Nalmpantis, Jane Yao, Behzad Honarpisheh, Narasimhan Aditya, and 1 others. 2024. A survey on efficient inference for large language models. arXiv preprint arXiv:2404.14294. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Li%2C+Kuangliu+Liu%2C+Yujia+Huang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Aristo+Nalmpantis%2C+Jane+Yao%2C+Behzad+Honarpisheh%2C+Narasimhan+Aditya%2C+and+1+others%2E+2024%2E+A+survey+on+efficient+infere)
68. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. [Blimp: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). In Transactions of the Association for Computational Linguistics, volume 8, pages 377–392.
69. David P Woodruff. 2014. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1–2):1–157. [find](https://scholar.google.com/scholar?q=David+P+Woodruff%2E+2014%2E+Sketching+as+a+tool+for+numerical+linear+algebra%2E+Foundations+and+Trends+in+Theoretical+Computer+Science%2C+10%281%E2%80%932%29%3A1%E2%80%93157%2E)
70. Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Cheng Li Li, and Yuxiong He. 2022. [Zeroquant: Efficient and affordable post-training quantization for large-scale transformers](https://md.research.elamur.ai/free/en/papers/yao-2022-zeroquant-efficient-affordable-post-training/). Advances in Neural Information Processing Systems, 35:27168– 27183.
71. Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2021. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630. — [arXiv:2103.13630](https://arxiv.org/abs/2103.13630) [find](https://scholar.google.com/scholar?q=Ofir+Zafrir%2C+Guy+Boudoukh%2C+Peter+Izsak%2C+and+Moshe+Wasserblat%2E+2021%2E+A+survey+of+quantization+methods+for+efficient+neural+network+inference%2E+arXiv+preprint+arXiv%3A2103%2E13630%2E+%E2%80%94+%5BarXiv%3A2103%2E13630%5D%28https)
72. Zhenhua Zhou, Amir Gholami, Zhewei Dong, Zhen Yao, Michael W Mahoney, and Kurt Keutzer. 2021. Post-training quantization for vision transformer. In Advances in Neural Information Processing Systems, volume 34, pages 28092–28103. [find](https://scholar.google.com/scholar?q=Zhenhua+Zhou%2C+Amir+Gholami%2C+Zhewei+Dong%2C+Zhen+Yao%2C+Michael+W+Mahoney%2C+and+Kurt+Keutzer%2E+2021%2E+Post%2Dtraining+quantization+for+vision+transformer%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+v)

*Updated 1 November 2026*