---
title: "Dual Activation-Weight Sparsity — paper by Sun et al. (2026)"
description: "The authors propose DAWS, a training-free method that merges both. 25 references. Running large language models is costly."
authors: ["Luoyang Sun", "Guangyan Li", "Cheng Deng", "Haifeng Zhang", "Jian Zhao", "Yongqiang Tang", "Wensheng Zhang", "Jun Wang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.378/"
doi: "10.18653/v1/2026.acl-long.378"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sun-2026-dual-activation-weight-sparsity-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sun-2026-dual-activation-weight-sparsity-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/) / Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression

# Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression

**Authors:** Luoyang Sun, Guangyan Li, [Cheng Deng](https://md.research.elamur.ai/free/en/authors/cheng-deng/), Haifeng Zhang, [Jian Zhao](https://md.research.elamur.ai/free/en/authors/jian-zhao/), Yongqiang Tang, Wensheng Zhang and [Jun Wang](https://md.research.elamur.ai/free/en/authors/jun-wang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.378/) · [PDF](https://aclanthology.org/2026.acl-long.378.pdf) · [DOI 10.18653/v1/2026.acl-long.378](https://doi.org/10.18653/v1/2026.acl-long.378) · [OpenAlex](https://openalex.org/W7166825207)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Running large language models is costly. Existing compression tools usually address only one side of the problem. Weight pruning (SparseGPT, WANDA) produces fixed masks, while activation sparsity (TEAL) skips small inputs uniformly. Combining the two has so far required extra machinery, such as the SVD components in R-Sparse. The authors propose DAWS, a training-free method that merges both.

An analysis of pretrained Llama-3-8B shows two things. About 15% of neurons carry roughly 75% of output energy, and sparsity behaviour differs strongly between projection matrices. Each linear layer therefore splits its input by magnitude into three groups:
- the largest values are multiplied with the original dense weights;
- mid-range values use a magnitude-pruned copy of the weights;
- the smallest values are dropped.

A two-stage greedy search sets sparsity budgets across and within modules (q/k/v/o_proj, gate/up/down_proj). The method needs no retraining, decompositions or learned routers.

On Llama-3-8B, Llama-2-7B and Mistral-7B at 50% sparsity, DAWS reportedly keeps over 98% of dense performance and beats WANDA, TEAL and R-Sparse by 1–3 points. This offers a simple path to cheaper inference.

## Key points
- **Ablation (Llama-3-8B, 50% sparsity):** Pure activation sparsity reaches PPL 8.53 and average accuracy 64.85. The full search (S1 + S2) improves this to PPL 7.17 and 67.83. The dense model scores PPL 6.13 and 69.46. The accuracy average spans eight tasks: Arc-C and Arc-E, BoolQ, HS, OBQA, SciQ, PIQA, and WG.
- **Magnitude vs WANDA-style pruning inside DAWS:** Magnitude pruning wins on PPL (7.17 vs 7.30) and on accuracy (67.83 vs 67.18). Masks were compared across WikiText-2 sample pairs. WANDA masks reach IoU of only 0.80–0.90 at best and are least stable in self_attn.o_proj. Magnitude masks do not depend on the data.
- **Dynamic vs static activation masks:** Activation masks overlap little between samples (IoU about 0.20–0.35). Replacing them with fixed masks drops average accuracy from 68.15 to 52.36, so thresholds must be computed per input.
- **Theory:** The authors give a proof that, under heavy-tailed activations, the dual scheme has lower approximation error than either sparsity type alone.
- **Caveats for readers:** The extracted text has internal inconsistencies:
  - DAWS on Llama-3-8B is reported as PPL 6.55 / 68.15 in one table but 7.17 / 67.83 in others. The latter is about 97.7% of dense, below the ">98%" claim.
  - The method overview mentions WANDA-style scoring, while the rest of the paper uses magnitude pruning.
  - Weight-mask IoU figures vary between >0.92 and 0.80–0.90.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 381 works · 80,747 citations
- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 107 works · 22,923 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 246 works · 24,952 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 87 works · 14,883 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 188 works · 8,744 citations
- [Guangzhou University](https://md.research.elamur.ai/free/en/universities/guangzhou-university/) — University, Guangzhou, China · 6 works · 7 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 297 works

## Cite

Cite as: Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang and Jun Wang. 2026. Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sun2026dual,
  title = {{Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression}},
  author = {Luoyang Sun, Guangyan Li, Cheng Deng, Haifeng Zhang, Jian Zhao, Yongqiang Tang, Wensheng Zhang and Jun Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.378},
  url = {https://doi.org/10.18653/v1/2026.acl-long.378}
}
```

- [← previous DSMentor: Curriculum-Guided Inference with Online Memory for Data-Science LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2026-dsmentor-curriculum-guided-inference-online/)
- [next → Dual-Axis Compositional Contrastive Few-Shot Learning using Prototypes Across Linguistic and Semantic Dimensions for Indic Low-Resource Multilingual NLU](https://md.research.elamur.ai/free/en/papers/mitra-2026-dual-axis-compositional-contrastive-few/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/30/)

## References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tai An, Ruwu Cai, Yanzhe Zhang, Yang Liu, Hao Chen, Pengcheng Xie, Sheng Chang, Yiwu Yao, and Gongyi Wang. 2025. Amber pruner: Leveraging n:m activation sparsity for efficient prefill in large language models. Preprint, arXiv:2508.02128. — [arXiv:2508.02128](https://arxiv.org/abs/2508.02128) [find](https://scholar.google.com/scholar?q=Tai+An%2C+Ruwu+Cai%2C+Yanzhe+Zhang%2C+Yang+Liu%2C+Hao+Chen%2C+Pengcheng+Xie%2C+Sheng+Chang%2C+Yiwu+Yao%2C+and+Gongyi+Wang%2E+2025%2E+Amber+pruner%3A+Leveraging+n%3Am+activation+sparsity+for+efficient+prefill+in+large+languag)
2. Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proc. AAAI, page 7432–7439.
3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proc. NAACL.
4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
5. Harry Dong, Beidi Chen, and Yuejie Chi. 2024. Prompt-prompted mixture of experts for efficient llm generation. CoRR. [find](https://scholar.google.com/scholar?q=Harry+Dong%2C+Beidi+Chen%2C+and+Yuejie+Chi%2E+2024%2E+Prompt%2Dprompted+mixture+of+experts+for+efficient+llm+generation%2E+CoRR%2E)
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
7. Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. Preprint, arXiv:1803.03635. [find](https://scholar.google.com/scholar?q=Jonathan+Frankle+and+Michael+Carbin%2E+2019%2E+The+lottery+ticket+hypothesis%3A+Finding+sparse%2C+trainable+neural+networks%2E+Preprint%2C+arXiv%3A1803%2E03635%2E)
8. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference on machine learning, pages 10323–10337. PMLR. [find](https://scholar.google.com/scholar?q=Elias+Frantar+and+Dan+Alistarh%2E+2023%2E+Sparsegpt%3A+Massive+language+models+can+be+accurately+pruned+in+one%2Dshot%2E+In+International+conference+on+machine+learning%2C+pages+10323%E2%80%9310337%2E+PMLR%2E)
9. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
10. Song Han, Huizi Mao, and William J. Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. Preprint, arXiv:1510.00149. [find](https://scholar.google.com/scholar?q=Song+Han%2C+Huizi+Mao%2C+and+William+J%2E+Dally%2E+2016%2E+Deep+compression%3A+Compressing+deep+neural+networks+with+pruning%2C+trained+quantization+and+huffman+coding%2E+Preprint%2C+arXiv%3A1510%2E00149%2E)
11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
12. Donghyun Lee, Je-Yong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. 2024. Cats: Contextually-aware thresholding for sparsity in large language models. arXiv preprint arXiv:2404.08763. [find](https://scholar.google.com/scholar?q=Donghyun+Lee%2C+Je%2DYong+Lee%2C+Genghan+Zhang%2C+Mo+Tiwari%2C+and+Azalia+Mirhoseini%2E+2024%2E+Cats%3A+Contextually%2Daware+thresholding+for+sparsity+in+large+language+models%2E+arXiv+preprint+arXiv%3A2404%2E08763%2E)
13. James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. 2024. Training-free activation sparsity in large language models. arXiv preprint arXiv:2408.14690. [find](https://scholar.google.com/scholar?q=James+Liu%2C+Pragaash+Ponnusamy%2C+Tianle+Cai%2C+Han+Guo%2C+Yoon+Kim%2C+and+Ben+Athiwaratkun%2E+2024%2E+Training%2Dfree+activation+sparsity+in+large+language+models%2E+arXiv+preprint+arXiv%3A2408%2E14690%2E)
14. Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. [Llm-pruner: On the structural pruning of large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-llm-pruner-structural-pruning-large/). Preprint, arXiv:2305.11627.
15. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843.
16. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proc. EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proc%2E+EMNLP%2E)
17. Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. 2023. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564. — [arXiv:2310.04564](https://arxiv.org/abs/2310.04564) [find](https://scholar.google.com/scholar?q=Iman+Mirzadeh%2C+Keivan+Alizadeh%2C+Sachin+Mehta%2C+Carlo+C+Del+Mundo%2C+Oncel+Tuzel%2C+Golnoosh+Samei%2C+Mohammad+Rastegari%2C+and+Mehrdad+Farajtabar%2E+2023%2E+Relu+strikes+back%3A+Exploiting+activation+sparsity+in+lar)
18. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proc. AAAI, page 8732–8740. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+Proc%2E+AAAI%2C+page+8732%E2%80%938740%2E)
19. Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. 2024. [Powerinfer: Fast large language model serving with a consumer-grade gpu](https://md.research.elamur.ai/free/en/papers/song-2023-powerinfer-fast-large-language-model/). Preprint, arXiv:2312.12456.
20. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695. [find](https://scholar.google.com/scholar?q=Mingjie+Sun%2C+Zhuang+Liu%2C+Anna+Bair%2C+and+J%2E+Zico+Kolter%2E+2023%2E+A+simple+and+effective+pruning+approach+for+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E11695%2E)
21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
22. Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106.
23. Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, and 1 others. 2025. Wanda++: Pruning large language models via regional gradients. arXiv preprint arXiv:2503.04992. [find](https://scholar.google.com/scholar?q=Yifan+Yang%2C+Kai+Zhen%2C+Bhavana+Ganesh%2C+Aram+Galstyan%2C+Goeric+Huybrechts%2C+Markus+M%C3%BCller%2C+Jonas+M+K%C3%BCbler%2C+Rupak+Vignesh+Swaminathan%2C+Athanasios+Mouchtaris%2C+Sravan+Babu+Bodapati%2C+and+1+others%2E+2025%2E+Wanda)
24. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proc. ACL. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proc%2E+ACL%2E)
25. Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, and Steven Li. 2025. R-sparse: Rank-aware activation sparsity for efficient llm inference. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Zechun+Liu%2C+Yuandong+Tian%2C+Harshit+Khaitan%2C+Zhangyang+Wang%2C+and+Steven+Li%2E+2025%2E+R%2Dsparse%3A+Rank%2Daware+activation+sparsity+for+efficient+llm+inference%2E+In+The+Thirteenth+International+Con)

*Updated 2 November 2026*