---
title: "Softpick — paper by Zuhri et al. (2026) — summary and ideas"
description: "The authors propose softpick, which swaps softmax for a rectified function. 49 references, cited by 5 papers here."
authors: ["Zayd Muhammad Kawakibi Zuhri", "Erland Hilman Fuadi", "Alham Fikri Aji"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.905/"
doi: "10.18653/v1/2026.findings-acl.905"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zuhri-2026-softpick-no-attention-sink-no/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zuhri-2026-softpick-no-attention-sink-no/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

# Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

**Authors:** Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and [Alham Fikri Aji](https://md.research.elamur.ai/free/en/authors/alham-fikri-aji/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.905/) · [PDF](https://aclanthology.org/2026.findings-acl.905.pdf) · [DOI 10.18653/v1/2026.findings-acl.905](https://doi.org/10.18653/v1/2026.findings-acl.905) · [OpenAlex](https://openalex.org/W4417172974)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/zuhri-2026-softpick-no-attention-sink-no/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Softmax attention forces each head's scores to add up to one. This pushes heads to dump weight on uninformative tokens such as BOS, a pattern known as attention sinks. It also produces extremely large hidden-state values, called massive activations, which make quantization and low-precision training difficult. The authors propose softpick, which swaps softmax for a rectified function. Its numerator is ReLU(e^x − 1) and its denominator sums the absolute values |e^x − 1|. Because of this asymmetry, outputs can be exactly zero and no longer need to total one. Negative scores still pass gradients, and the function adds no parameters and works with FlashAttention-2. Llama-style models with 340M and 1.8B parameters were trained from scratch on fineweb-edu, each with a softmax and a softpick variant. Softpick removed sinks entirely and gave sparser attention maps and hidden states with much lower kurtosis. Its quantized models beat softmax, especially at low bit widths. At 340M it roughly matched softmax quality, but at 1.8B it fell behind. That gap matters for anyone weighing it for quantization, pruning or interpretability work.

## Key points
- **Training setup:** the 340M models saw 52B tokens and the 1.8B models 104B tokens from the fineweb-edu 100B subset, trained on 8×H100 GPUs with the flash-linear-attention / Flame framework.
- **Sinks and loss at 340M:** the sink rate fell from 63.41% with softmax to 0.00% with softpick. The final training loss differed by only 0.004.
- **Quantization:** quantized softpick models outperformed softmax, with the largest advantage at the lowest precisions, down to 2-bit.
- **Weakness at 1.8B:**
  - Wikitext word perplexity rose from 15.10 to 17.87.
  - Arc Easy accuracy dropped from 67.21 to 62.04.
  - The authors offer two explanations: long-context underscoring and "dead" heads.
- **Dead heads:** their share declined over training, from 17.19% to 9.11% at 340M and from 42.87% to 20.90% at 1.8B. The authors suggest training on more tokens could help. Passkey-style long-context retrieval did not improve.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and Alham Fikri Aji. 2026. Softpick: No Attention Sink, No Massive Activations with Rectified Softmax. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zuhri2026softpick,
  title = {{Softpick: No Attention Sink, No Massive Activations with Rectified Softmax}},
  author = {Zayd Muhammad Kawakibi Zuhri, Erland Hilman Fuadi and Alham Fikri Aji},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.905},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.905}
}
```

- [← previous Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs](https://md.research.elamur.ai/free/en/papers/siddique-2026-shifting-perspectives-steering-vectors-robust/)
- [next → SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2026-superwriter-reflection-driven-long-form/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samira Abnar and Willem Zuidema. 2020. [Quantifying attention flow in transformers](https://md.research.elamur.ai/free/en/papers/abnar-2020-quantifying-attention-flow-transformers/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197, Online. Association for Computational Linguistics.
2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, 18187 9 Che Chang, Kai Chen, and 106 others. 2025. [gpt-oss-120b &…](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
3. Hicham Badri and Appu Shaji. 2023. HQQ: Half-quadratic quantization for large language models (blog post). [https://mobiusml.github.io/hqq_](https://mobiusml.github.io/hqq_) blog/. Accessed: 2025-12-30. [↗](https://mobiusml.github.io/hqq_)
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
5. Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. 2025. Why do LLMs attend to the first token? Preprint, arXiv:2504.02732. — [arXiv:2504.02732](https://arxiv.org/abs/2504.02732) [find](https://scholar.google.com/scholar?q=Federico+Barbero%2C+%C3%81lvaro+Arroyo%2C+Xiangming+Gu%2C+Christos+Perivolaropoulos%2C+Michael+Bronstein%2C+Petar+Velic%CC%8Ckovic%CC%81%2C+and+Razvan+Pascanu%2E+2025%2E+Why+do+LLMs+attend+to+the+first+token%3F+Preprint%2C+arXiv%3A2504%2E0)
6. Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. [PIQA: reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
7. Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. Quantizable transformers: Removing outliers by helping attention heads do nothing. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA,… [find](https://scholar.google.com/scholar?q=Yelysei+Bondarenko%2C+Markus+Nagel%2C+and+Tijmen+Blankevoort%2E+2023%2E+Quantizable+transformers%3A+Removing+outliers+by+helping+attention+heads+do+nothing%2E+In+Advances+in+Neural+Information+Processing+Systems+)
8. David Chiang. 2025. Transformers in uniform tc 0 . Preprint, arXiv:2409.13629. — [arXiv:2409.13629](https://arxiv.org/abs/2409.13629) [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2025%2E+Transformers+in+uniform+tc+0+%2E+Preprint%2C+arXiv%3A2409%2E13629%2E+%E2%80%94+%5BarXiv%3A2409%2E13629%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2409%2E13629%29)
9. Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Timoth%C3%A9e+Darcet%2C+Maxime+Oquab%2C+Julien+Mairal%2C+and+Piotr+Bojanowski%2E+2024%2E+Vision+transformers+need+registers%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C+Au)
10. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Mike+Lewis%2C+Younes+Belkada%2C+and+Luke+Zettlemoyer%2E+2022%2E+LLM%2Eint8%28%29%3A+8%2Dbit+matrix+multiplication+for+transformers+at+scale%2E)
11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized llms. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 20… [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+QLoRA%3A+Efficient+fine%2Dtuning+of+quantized+llms%2E+In+Advances+in+Neural+Information+Processing+Systems+36%3A+Annual+Conference+on+)
12. Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. 2024. Scaling FP8 training to trillion-token llms. ArXiv preprint, abs/2409.12517. [find](https://scholar.google.com/scholar?q=Maxim+Fishman%2C+Brian+Chmiel%2C+Ron+Banner%2C+and+Daniel+Soudry%2E+2024%2E+Scaling+FP8+training+to+trillion%2Dtoken+llms%2E+ArXiv+preprint%2C+abs%2F2409%2E12517%2E)
13. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. [find](https://scholar.google.com/scholar?q=Elias+Frantar%2C+Saleh+Ashkboos%2C+Torsten+Hoefler%2C+and+Dan+Alistarh%2E+2022%2E+GPTQ%3A+Accurate+post%2Dtraining+quantization+for+generative+pre%2Dtrained+transformers%2E)
14. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
15. Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2024. When attention sink emerges in language models: An empirical view. Preprint, arXiv:2410.10781. — [arXiv:2410.10781](https://arxiv.org/abs/2410.10781) [find](https://scholar.google.com/scholar?q=Xiangming+Gu%2C+Tianyu+Pang%2C+Chao+Du%2C+Qian+Liu%2C+Fengzhuo+Zhang%2C+Cunxiao+Du%2C+Ye+Wang%2C+and+Min+Lin%2E+2024%2E+When+attention+sink+emerges+in+language+models%3A+An+empirical+view%2E+Preprint%2C+arXiv%3A2410%2E10781%2E+%E2%80%94+%5B)
16. Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I. Jordan, and Song Mei. 2024. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. Preprint, arXiv:2410.13835. — [arXiv:2410.13835](https://arxiv.org/abs/2410.13835) [find](https://scholar.google.com/scholar?q=Tianyu+Guo%2C+Druv+Pai%2C+Yu+Bai%2C+Jiantao+Jiao%2C+Michael+I%2E+Jordan%2C+and+Song+Mei%2E+2024%2E+Active%2Ddormant+attention+heads%3A+Mechanistically+demystifying+extreme%2Dtoken+phenomena+in+llms%2E+Preprint%2C+arXiv%3A2410%2E13)
17. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). Preprint, arXiv:1803.05457.
18. Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. 2024. From attention to activation: Unravelling the enigmas of large language models. Preprint, arXiv:2410.17174. — [arXiv:2410.17174](https://arxiv.org/abs/2410.17174) [find](https://scholar.google.com/scholar?q=Prannay+Kaul%2C+Chengcheng+Ma%2C+Ismail+Elezi%2C+and+Jiankang+Deng%2E+2024%2E+From+attention+to+activation%3A+Unravelling+the+enigmas+of+large+language+models%2E+Preprint%2C+arXiv%3A2410%2E17174%2E+%E2%80%94+%5BarXiv%3A2410%2E17174%5D%28htt)
19. Tri Dao. 2024. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
20. Shashi Kumar, Srikanth R. Madikeri, Juan Pablo Zuluaga, Esaú Villatoro-Tello, Iuliia Nigmatulina, Petr Motlícek, E ManjunathK, and Aravind Ganapathiraju. 2024. XLSR-Transducer: Streaming asr for self-supervised pretrained models. [find](https://scholar.google.com/scholar?q=Shashi+Kumar%2C+Srikanth+R%2E+Madikeri%2C+Juan+Pablo+Zuluaga%2C+Esa%C3%BA+Villatoro%2DTello%2C+Iuliia+Nigmatulina%2C+Petr+Motl%C3%ADcek%2C+E+ManjunathK%2C+and+Aravind+Ganapathiraju%2E+2024%2E+XLSR%2DTransducer%3A+Streaming+asr+for+self%2D)
21. ICASSP 2025 -2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). [find](https://scholar.google.com/scholar?q=ICASSP+2025+%2D2025+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E)
22. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans… [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Sy)
23. Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Iris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. 2025. TorchTitan: One-stop pytorch native solution for production ready LLM pretraining. In The Thirteenth 18188… [find](https://scholar.google.com/scholar?q=Wanchao+Liang%2C+Tianyu+Liu%2C+Less+Wright%2C+Will+Constable%2C+Andrew+Gu%2C+Chien%2DChin+Huang%2C+Iris+Zhang%2C+Wei+Feng%2C+Howard+Huang%2C+Junjie+Wang%2C+Sanket+Purandare%2C+Gokul+Nadathur%2C+and+Stratos+Idreos%2E+2025%2E+TorchT)
24. International Conference on Learning Representations. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, and 180 others. 20… [find](https://scholar.google.com/scholar?q=International+Conference+on+Learning+Representations%2E+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+)
25. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. FineWeb-Edu: the finest collection of educational content. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Loubna+Ben+Allal%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2024%2E+FineWeb%2DEdu%3A+the+finest+collection+of+educational+content%2E)
26. Yi Lu, Xin Zhou, Wei He, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. LongHeads: Multi-head attention is secretly a long context processor. Preprint, arXiv:2402.10685. [find](https://scholar.google.com/scholar?q=Yi+Lu%2C+Xin+Zhou%2C+Wei+He%2C+Jun+Zhao%2C+Tao+Ji%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2024%2E+LongHeads%3A+Multi%2Dhead+attention+is+secretly+a+long+context+processor%2E+Preprint%2C+arXiv%3A2402%2E10685%2E)
27. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
28. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. [find](https://scholar.google.com/scholar?q=Meta%2E+2025%2E+The+llama+4+herd%3A+The+beginning+of+a+new+era+of+natively+multimodal+ai+innovation%2E)
29. Evan Miller. 2023. Attention is off by one. [find](https://scholar.google.com/scholar?q=Evan+Miller%2E+2023%2E+Attention+is+off+by+one%2E)
30. Ken M. Nakanishi. 2025. Scalable-softmax is superior for attention. Preprint, arXiv:2501.19399. — [arXiv:2501.19399](https://arxiv.org/abs/2501.19399) [find](https://scholar.google.com/scholar?q=Ken+M%2E+Nakanishi%2E+2025%2E+Scalable%2Dsoftmax+is+superior+for+attention%2E+Preprint%2C+arXiv%3A2501%2E19399%2E+%E2%80%94+%5BarXiv%3A2501%2E19399%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E19399%29)
31. Louis Owen, Nilabhra Roy Chowdhury, Abhay Kumar, and Fabian Güra. 2025. A refined analysis of massive activations in llms. Preprint, arXiv:2503.22329. — [arXiv:2503.22329](https://arxiv.org/abs/2503.22329) [find](https://scholar.google.com/scholar?q=Louis+Owen%2C+Nilabhra+Roy+Chowdhury%2C+Abhay+Kumar%2C+and+Fabian+G%C3%BCra%2E+2025%2E+A+refined+analysis+of+massive+activations+in+llms%2E+Preprint%2C+arXiv%3A2503%2E22329%2E+%E2%80%94+%5BarXiv%3A2503%2E22329%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E2)
32. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Proceedings of the 54th Annual Meeting of the Association fo…
33. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neu… [find](https://scholar.google.com/scholar?q=Zihan+Qiu%2C+Zekun+Wang%2C+Bo+Zheng%2C+Zeyu+Huang%2C+Kaiyue+Wen%2C+Songlin+Yang%2C+Rui+Men%2C+Le+Yu%2C+Fei+Huang%2C+Suozhi+Huang%2C+Dayiheng+Liu%2C+Jingren+Zhou%2C+and+Junyang+Lin%2E+2025%2E+Gated+attention+for+large+language+mo)
34. Hemanth Saratchandran, Jianqiao Zheng, Yiping Ji, Wenbo Zhang, and Simon Lucey. 2024. Rethinking softmax: Self-attention with polynomial activations. Preprint, arXiv:2410.18613. — [arXiv:2410.18613](https://arxiv.org/abs/2410.18613) [find](https://scholar.google.com/scholar?q=Hemanth+Saratchandran%2C+Jianqiao+Zheng%2C+Yiping+Ji%2C+Wenbo+Zhang%2C+and+Simon+Lucey%2E+2024%2E+Rethinking+softmax%3A+Self%2Dattention+with+polynomial+activations%2E+Preprint%2C+arXiv%3A2410%2E18613%2E+%E2%80%94+%5BarXiv%3A2410%2E18613%5D%28h)
35. Kyuhong Shim, Iksoo Choi, Wonyong Sung, and Jungwook Choi. 2021. Layer-wise pruning of transformer attention heads for efficient language modeling. In 2021 18th International SoC Design Conference (ISOCC), page 357–358. IEEE. [find](https://scholar.google.com/scholar?q=Kyuhong+Shim%2C+Iksoo+Choi%2C+Wonyong+Sung%2C+and+Jungwook+Choi%2E+2021%2E+Layer%2Dwise+pruning+of+transformer+attention+heads+for+efficient+language+modeling%2E+In+2021+18th+International+SoC+Design+Conference+%28IS)
36. Ethan Smith. 2025. Softmax attention is a fluke. [find](https://scholar.google.com/scholar?q=Ethan+Smith%2E+2025%2E+Softmax+attention+is+a+fluke%2E)
37. Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [find](https://scholar.google.com/scholar?q=Daria+Soboleva%2C+Faisal+Al%2DKhateeb%2C+Robert+Myers%2C+Jacob+R+Steeves%2C+Joel+Hestness%2C+and+Nolan+Dey%2E+2023%2E+SlimPajama%3A+A+627B+token+cleaned+and+deduplicated+version+of+RedPajama%2E)
38. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2440–2448. [find](https://scholar.google.com/scholar?q=Sainbayar+Sukhbaatar%2C+Arthur+Szlam%2C+Jason+Weston%2C+and+Rob+Fergus%2E+2015%2E+End%2Dto%2Dend+memory+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+28%3A+Annual+Conference+on+Neural+Information+Pro)
39. Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. [Massive activations in large language models](https://md.research.elamur.ai/free/en/papers/sun-2024-massive-activations-large-language-models/). Preprint, arXiv:2402.17762.
40. Philippe Tillet, H. T. Kung, and David Cox. 2019. [Triton: an intermediate language and compiler for tiled neural network computations](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/). In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, page 10–19, New York, NY, USA. Association for…
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
42. Hanrui Wang, Zhekai Zhang, and Song Han. 2021. [SpAtten: Efficient sparse attention architecture with cascade token and head pruning](https://md.research.elamur.ai/free/en/papers/wang-2020-spatten-efficient-sparse-attention-architecture/). In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE.
43. Hongyu Wang, Shuming Ma, and Furu Wei. 2025. Bit-Net v2: Native 4-bit activations with hadamard transformation for 1-bit llms. [find](https://scholar.google.com/scholar?q=Hongyu+Wang%2C+Shuming+Ma%2C+and+Furu+Wei%2E+2025%2E+Bit%2DNet+v2%3A+Native+4%2Dbit+activations+with+hadamard+transformation+for+1%2Dbit+llms%2E)
44. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
45. Yuxin Wen, Jim Wu, Ajay Jain, Tom Goldstein, and Ashwinee Panda. 2025. Analysis of attention in video diffusion transformers. Preprint, arXiv:2504.10317. — [arXiv:2504.10317](https://arxiv.org/abs/2504.10317) [find](https://scholar.google.com/scholar?q=Yuxin+Wen%2C+Jim+Wu%2C+Ajay+Jain%2C+Tom+Goldstein%2C+and+Ashwinee+Panda%2E+2025%2E+Analysis+of+attention+in+video+diffusion+transformers%2E+Preprint%2C+arXiv%3A2504%2E10317%2E+%E2%80%94+%5BarXiv%3A2504%2E10317%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F250)
46. Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings… [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Ji+Lin%2C+Micka%C3%ABl+Seznec%2C+Hao+Wu%2C+Julien+Demouth%2C+and+Song+Han%2E+2023%2E+SmoothQuant%3A+Accurate+and+efficient+post%2Dtraining+quantization+for+large+language+models%2E+In+International+Conferenc)
47. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth 18189 11 [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+18189+11)
48. International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=International+Conference+on+Learning+Representations%2C+ICLR+2024%2C+Vienna%2C+Austria%2C+May+7%2D11%2C+2024%2E+OpenReview%2Enet%2E)
49. Songlin Yang and Yu Zhang. 2024. FLA: A triton-based library for hardware-efficient implementations of linear attention mechanism. 18190 12 [find](https://scholar.google.com/scholar?q=Songlin+Yang+and+Yu+Zhang%2E+2024%2E+FLA%3A+A+triton%2Dbased+library+for+hardware%2Defficient+implementations+of+linear+attention+mechanism%2E+18190+12)

*Updated 2 November 2026*