---
title: "LM-Infinite — paper by Han et al. (2024) — summary and ideas"
description: "Han et al. study this failure on Llama-2 through both theory and experiments. 1.2 points on Qasper. 75 references, cited by 75 papers here."
authors: ["Chi Han", "Qifan Wang", "Hao Peng", "Wenhan Xiong", "Yu Chen", "Heng Ji", "Sinong Wang"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.222/"
doi: "10.18653/v1/2024.naacl-long.222"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

# LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

**Authors:** [Chi Han](https://md.research.elamur.ai/free/en/authors/chi-han/), [Qifan Wang](https://md.research.elamur.ai/free/en/authors/qifan-wang/), [Hao Peng](https://md.research.elamur.ai/free/en/authors/hao-peng/), [Wenhan Xiong](https://md.research.elamur.ai/free/en/authors/wenhan-xiong/), [Yu Chen](https://md.research.elamur.ai/free/en/authors/yu-chen/), [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/) and Sinong Wang · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.222/) · [PDF](https://aclanthology.org/2024.naacl-long.222.pdf) · [DOI 10.18653/v1/2024.naacl-long.222](https://doi.org/10.18653/v1/2024.naacl-long.222) · [OpenAlex](https://openalex.org/W4401042914)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Transformer LLMs are usually pre-trained on segments shorter than 4K tokens, and their output degrades sharply on much longer inputs. Han et al. study this failure on Llama-2 through both theory and experiments. They trace it to three causes. First, relative distances never seen during training force attention logits to grow without bound. Second, capping the logits instead lets attention entropy rise as more tokens are attended. Third, the opening tokens implicitly carry absolute-position information, so discarding them hurts the model. The authors argue that sliding-window attention and relative encodings such as RoPE or AliBi cannot fix these problems alone.

Their method, LM-Infinite, requires no fine-tuning and has two parts. A Λ-shaped attention mask keeps only the initial tokens plus a recent window. A distance ceiling clamps relative positions to the pre-training maximum. Optionally, top-k middle tokens can be attended again. On off-the-shelf models, perplexity stays stable on ArXiv and OpenWebText2 up to 200M tokens. Zero-shot Passkey Retrieval and Qasper scores improve, and compute costs drop. This makes long-context LLM use practical for researchers with modest resources.

## Key points
- **Three failure factors:** exploding attention logits at unseen distances, growing attention entropy over unseen numbers of tokens, and a distinct feature region occupied by the starting tokens.
- **Downstream results:** Llama-2-7b-chat used top-5 middle tokens in layers above the 5th. It gained 37.2 percentage points on Passkey Retrieval and 1.2 points on Qasper (1.5K test QA pairs) over the original model and truncation baselines.
- **Efficiency:** on 32K-token sequences, LM-Infinite decodes 2.7× faster and uses 7.5× less GPU memory than the unmodified LLM.
- **Generation quality:** BLEU and ROUGE-L scores on ArXiv and OpenWebText2 are comparable to the fine-tuned MPT-7B-SW, while vanilla models produce mostly nonsense. Against repeated truncation on 10k-token generation, LM-Infinite scores about 5 BLEU higher at similar compute. It matches the baseline's BLEU with less than 25% of its computation.
- **Limitations:** the method applies only to relative-positional-encoding Transformers. It was not tested on proprietary models such as ChatGPT or on lengths beyond 200M tokens (e.g., 1G).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji and Sinong Wang. 2024. LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{han2024lm,
  title = {{LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models}},
  author = {Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji and Sinong Wang},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.222},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.222}
}
```

- [← previous Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/li-2024-language-ranker-metric-quantifying-llm/)
- [next → SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Support](https://md.research.elamur.ai/free/en/papers/qiu-2024-smile-single-turn-multi-turn/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

75 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. [Exploring length generalization in large language models](https://md.research.elamur.ai/free/en/papers/anil-2022-exploring-length-generalization-large-language/). Advances in Neural Information Processing Systems, 35:38546–38556.
2. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. [Longalign: A recipe for long context alignment of large language models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/). arXiv preprint arXiv:2401.18058.
3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. Longbench: A bilingual, multi-task benchmark for long context understanding. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+2023%2E+Longbench%3A+A+bilingual%2C+multi%2Dtask)
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
5. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+2022%2E)
6. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR. [find](https://scholar.google.com/scholar?q=Improving+language+models+by+retrieving+from+trillions+of+tokens%2E+In+International+conference+on+machine+learning%2C+pages+2206%E2%80%932240%2E+PMLR%2E)
7. Mirelle Candida Bueno, Carlos Gemmell, Jeff Dalton, Roberto Lotufo, and Rodrigo Nogueira. 2022. Induced natural language rationales and interleaved markup tokens enable extrapolation in large language models. In Proceedings of the 1st Workshop on Mathematical Natural Language Processing (MathNLP), p… [find](https://scholar.google.com/scholar?q=Mirelle+Candida+Bueno%2C+Carlos+Gemmell%2C+Jeff+Dalton%2C+Roberto+Lotufo%2C+and+Rodrigo+Nogueira%2E+2022%2E+Induced+natural+language+rationales+and+interleaved+markup+tokens+enable+extrapolation+in+large+language)
8. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023a. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023a%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2E)
9. Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. 2021. Skyformer: Remodel self-attention with gaussian kernel and nyström method. In Proc. Thirty-fifth Annual Conference on Neural Information Processing Systems (NeurIPS2021). [find](https://scholar.google.com/scholar?q=Yifan+Chen%2C+Qi+Zeng%2C+Heng+Ji%2C+and+Yun+Yang%2E+2021%2E+Skyformer%3A+Remodel+self%2Dattention+with+gaussian+kernel+and+nystr%C3%B6m+method%2E+In+Proc%2E+Thirty%2Dfifth+Annual+Conference+on+Neural+Information+Processing+Sy)
10. Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. 2022. Sketching as a tool for understanding and accelerating self-attention for long sequences. In Proc. [find](https://scholar.google.com/scholar?q=Yifan+Chen%2C+Qi+Zeng%2C+Heng+Ji%2C+and+Yun+Yang%2E+2022%2E+Sketching+as+a+tool+for+understanding+and+accelerating+self%2Dattention+for+long+sequences%2E+In+Proc%2E)
11. The 2022 Conference of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT2022). [find](https://scholar.google.com/scholar?q=The+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%2D+Human+Language+Technologies+%28NAACL%2DHLT2022%29%2E)
12. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). In The Twelfth International Conference on Learning Representations.
13. Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. 2023. Dissecting transformer length extrapolation via the lens of receptive field analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13522–13537. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Alexander+Rudnicky%2C+and+Peter+Ramadge%2E+2023%2E+Dissecting+transformer+length+extrapolation+via+the+lens+of+receptive+field+analysis%2E+In+Proceedings+of+the+61st+Annual+Meeting)
14. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+G+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+An)
15. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang…
16. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. 2023. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486. [find](https://scholar.google.com/scholar?q=Jiayu+Ding%2C+Shuming+Ma%2C+Li+Dong%2C+Xingxing+Zhang%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+and+Furu+Wei%2E+2023%2E+Longnet%3A+Scaling+transformers+to+1%2C000%2C000%2C000+tokens%2E+arXiv+preprint+arXiv%3A2307%2E02486%2E)
17. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753.
18. Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. 2024. Get more with less: Synthesizing recurrence with kv cache compression for efficient llm inference. arXiv preprint arXiv:2402.09398. — [arXiv:2402.09398](https://arxiv.org/abs/2402.09398) [find](https://scholar.google.com/scholar?q=Harry+Dong%2C+Xinyu+Yang%2C+Zhenyu+Zhang%2C+Zhangyang+Wang%2C+Yuejie+Chi%2C+and+Beidi+Chen%2E+2024%2E+Get+more+with+less%3A+Synthesizing+recurrence+with+kv+cache+compression+for+efficient+llm+inference%2E+arXiv+preprin)
19. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+Pile%3A+An+800gb+datas)
20. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Mingwei+Chang%2E+2020%2E+Retrieval+augmented+language+model+pre%2Dtraining%2E+In+International+conference+on+machine+learning%2C+pages+3929%E2%80%933938%2E+PMLR%2E)
21. David Haussler. 2018. Decision theoretic generalizations of the pac model for neural net and other learning applications. In The Mathematics of Generalization, pages 37–116. CRC Press. [find](https://scholar.google.com/scholar?q=David+Haussler%2E+2018%2E+Decision+theoretic+generalizations+of+the+pac+model+for+neural+net+and+other+learning+applications%2E+In+The+Mathematics+of+Generalization%2C+pages+37%E2%80%93116%2E+CRC+Press%2E)
22. Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. 2021. Shape: Shifted absolute position embedding for transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3309–3321. [find](https://scholar.google.com/scholar?q=Shun+Kiyono%2C+Sosuke+Kobayashi%2C+Jun+Suzuki%2C+and+Kentaro+Inui%2E+2021%2E+Shape%3A+Shifted+absolute+position+embedding+for+transformers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+La)
23. Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. 2024. A human-inspired reading agent with gist memory of very long contexts. arXiv preprint arXiv:2402.09727. [find](https://scholar.google.com/scholar?q=Kuang%2DHuei+Lee%2C+Xinyun+Chen%2C+Hiroki+Furuta%2C+John+Canny%2C+and+Ian+Fischer%2E+2024%2E+A+human%2Dinspired+reading+agent+with+gist+memory+of+very+long+contexts%2E+arXiv+preprint+arXiv%3A2402%2E09727%2E)
24. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can open-source llms truly promise on context length? [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Rulin+Shao%2C+Anze+Xie%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+Xuezhe+Ma%2C+and+Hao+Zhang%2E+2023%2E+How+long+can+open%2Dsource+llms+truly+promise+on+context+length%3F)
25. Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. 2021. Cape: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34:16079– 16092. [find](https://scholar.google.com/scholar?q=Tatiana+Likhomanenko%2C+Qiantong+Xu%2C+Gabriel+Synnaeve%2C+Ronan+Collobert%2C+and+Alex+Rogozhnikov%2E+2021%2E+Cape%3A+Encoding+relative+positions+with+continuous+augmented+positional+embeddings%2E+Advances+in+Neural+)
26. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
27. Xijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, and Mao Yang. 2023. Boosting llm reasoning: Push the limits of few-shot learning with reinforced in-context pruning. arXiv preprint arXiv:2312.08901. — [arXiv:2312.08901](https://arxiv.org/abs/2312.08901) [find](https://scholar.google.com/scholar?q=Xijie+Huang%2C+Li+Lyna+Zhang%2C+Kwang%2DTing+Cheng%2C+and+Mao+Yang%2E+2023%2E+Boosting+llm+reasoning%3A+Push+the+limits+of+few%2Dshot+learning+with+reinforced+in%2Dcontext+pruning%2E+arXiv+preprint+arXiv%3A2312%2E08901%2E+%E2%80%94+%5Ba)
28. Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2023. [Scaling laws of rope-based extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/). In The Twelfth International Conference on Learning Representations.
29. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. [Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/). arXiv preprint arXiv:2310.06839.
30. Kun Luo, Zheng Liu, Shitao Xiao, and Kang Liu. 2024. Bge landmark embedding: A chunking-free embedding method for retrieval augmented long-context large language models. arXiv preprint arXiv:2402.11573. [find](https://scholar.google.com/scholar?q=Kun+Luo%2C+Zheng+Liu%2C+Shitao+Xiao%2C+and+Kang+Liu%2E+2024%2E+Bge+landmark+embedding%3A+A+chunking%2Dfree+embedding+method+for+retrieval+augmented+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E1157)
31. Kaiokendev. 2023. Things iḿ learning while training superhot. [https://kaiokendev.github.io/til\#](https://kaiokendev.github.io/til\#) extending-context-to-8k. [↗](https://kaiokendev.github.io/til\#)
32. Kai Lv, Xiaoran Liu, Qipeng Guo, Hang Yan, Conghui He, Xipeng Qiu, and Dahua Lin. 2024. Longwanjuan: Towards systematic measurement for long text quality. arXiv preprint arXiv:2402.13583. [find](https://scholar.google.com/scholar?q=Kai+Lv%2C+Xiaoran+Liu%2C+Qipeng+Guo%2C+Hang+Yan%2C+Conghui+He%2C+Xipeng+Qiu%2C+and+Dahua+Lin%2E+2024%2E+Longwanjuan%3A+Towards+systematic+measurement+for+long+text+quality%2E+arXiv+preprint+arXiv%3A2402%2E13583%2E)
33. Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. 2016. Learning to remember rare events. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser%2C+Ofir+Nachum%2C+Aurko+Roy%2C+and+Samy+Bengio%2E+2016%2E+Learning+to+remember+rare+events%2E+In+International+Conference+on+Learning+Representations%2E)
34. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. The impact of positional encoding on length generalization in transformers. arXiv preprint arXiv:2305.19466. [find](https://scholar.google.com/scholar?q=Amirhossein+Kazemnejad%2C+Inkit+Padhi%2C+Karthikeyan+Natesan+Ramamurthy%2C+Payel+Das%2C+and+Siva+Reddy%2E+2023%2E+The+impact+of+positional+encoding+on+length+generalization+in+transformers%2E+arXiv+preprint+arXiv%3A2)
35. Guolin Ke, Di He, and Tie-Yan Liu. 2020. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guolin+Ke%2C+Di+He%2C+and+Tie%2DYan+Liu%2E+2020%2E+Rethinking+positional+encoding+in+language+pre%2Dtraining%2E+In+International+Conference+on+Learning+Representations%2E)
36. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pages 4171–4186.
37. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2019%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+In+International+Conference+on+Learning+Repr)
38. Amirkeivan Mohtashami and Martin Jaggi. 2023. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+2023%2E+Landmark+attention%3A+Random%2Daccess+infinite+context+length+for+transformers%2E+arXiv+preprint+arXiv%3A2305%2E16300%2E)
39. Matanel Oren, Michael Hassid, Yossi Adi, and Roy Schwartz. 2024. Transformers are multi-state rnns. arXiv preprint arXiv:2401.06104. [find](https://scholar.google.com/scholar?q=Matanel+Oren%2C+Michael+Hassid%2C+Yossi+Adi%2C+and+Roy+Schwartz%2E+2024%2E+Transformers+are+multi%2Dstate+rnns%2E+arXiv+preprint+arXiv%3A2401%2E06104%2E)
40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
41. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2023%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat)
42. Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427. — [arXiv:2310.01427](https://arxiv.org/abs/2310.01427) [find](https://scholar.google.com/scholar?q=Alexander+Peysakhovich+and+Adam+Lerer%2E+2023%2E+Attention+sorting+combats+recency+bias+in+long+context+language+models%2E+arXiv+preprint+arXiv%3A2310%2E01427%2E+%E2%80%94+%5BarXiv%3A2310%2E01427%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E01)
43. David Pollard. 1990. Empirical processes: Theory and applications. In NSF-CBMS Regional Conference Series in Probability and Statistics, pages i–86. JSTOR. [find](https://scholar.google.com/scholar?q=David+Pollard%2E+1990%2E+Empirical+processes%3A+Theory+and+applications%2E+In+NSF%2DCBMS+Regional+Conference+Series+in+Probability+and+Statistics%2C+pages+i%E2%80%9386%2E+JSTOR%2E)
44. Ofir Press, Noah Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2021%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
45. Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. arXiv preprint arXiv:2403.03514. [find](https://scholar.google.com/scholar?q=Zexuan+Qiu%2C+Jingjing+Li%2C+Shijue+Huang%2C+Wanjun+Zhong%2C+and+Irwin+King%2E+2024%2E+Clongeval%3A+A+chinese+benchmark+for+evaluating+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E03514%2E)
46. Zhijie Qu. 2023. Gpt rotational position embedding for length extrapolation. In Proceedings of the 2023 6th International Conference on Machine Learning and Natural Language Processing, pages 166–170. [find](https://scholar.google.com/scholar?q=Zhijie+Qu%2E+2023%2E+Gpt+rotational+position+embedding+for+length+extrapolation%2E+In+Proceedings+of+the+2023+6th+International+Conference+on+Machine+Learning+and+Natural+Language+Processing%2C+pages+166%E2%80%93170%2E)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
48. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
49. Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. [Parallel context windows for large language models](https://md.research.elamur.ai/free/en/papers/ratner-2023-parallel-context-windows-large-language/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1…
50. Siyu Ren and Kenny Q Zhu. 2024. On the efficacy of eviction policy for key-value constrained generative language model inference. arXiv preprint arXiv:2402.06262. [find](https://scholar.google.com/scholar?q=Siyu+Ren+and+Kenny+Q+Zhu%2E+2024%2E+On+the+efficacy+of+eviction+policy+for+key%2Dvalue+constrained+generative+language+model+inference%2E+arXiv+preprint+arXiv%3A2402%2E06262%2E)
51. Ninglu Shao, Shitao Xiao, Zheng Liu, and Peitian Zhang. 2024. Flexibly scaling large language models contexts through extensible tokenization. arXiv preprint arXiv:2401.07793. — [arXiv:2401.07793](https://arxiv.org/abs/2401.07793) [find](https://scholar.google.com/scholar?q=Ninglu+Shao%2C+Shitao+Xiao%2C+Zheng+Liu%2C+and+Peitian+Zhang%2E+2024%2E+Flexibly+scaling+large+language+models+contexts+through+extensible+tokenization%2E+arXiv+preprint+arXiv%3A2401%2E07793%2E+%E2%80%94+%5BarXiv%3A2401%2E07793%5D%28htt)
52. Kaiqiang Song, Xiaoyang Wang, Sangwoo Cho, Xiaoman Pan, and Dong Yu. 2023. Zebra: Extending context window with layerwise grouped local-global attention. arXiv preprint arXiv:2312.08618. — [arXiv:2312.08618](https://arxiv.org/abs/2312.08618) [find](https://scholar.google.com/scholar?q=Kaiqiang+Song%2C+Xiaoyang+Wang%2C+Sangwoo+Cho%2C+Xiaoman+Pan%2C+and+Dong+Yu%2E+2023%2E+Zebra%3A+Extending+context+window+with+layerwise+grouped+local%2Dglobal+attention%2E+arXiv+preprint+arXiv%3A2312%2E08618%2E+%E2%80%94+%5BarXiv%3A2312)
53. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2021. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864.
54. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). arXiv preprint arXiv:2212.10554.
55. Mingxu Tao, Yansong Feng, and Dongyan Zhao. 2023. A frustratingly easy improvement for position embeddings via random padding. arXiv preprint arXiv:2305.04859. — [arXiv:2305.04859](https://arxiv.org/abs/2305.04859) [find](https://scholar.google.com/scholar?q=Mingxu+Tao%2C+Yansong+Feng%2C+and+Dongyan+Zhao%2E+2023%2E+A+frustratingly+easy+improvement+for+position+embeddings+via+random+padding%2E+arXiv+preprint+arXiv%3A2305%2E04859%2E+%E2%80%94+%5BarXiv%3A2305%2E04859%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
56. MosaicML NLP Team. 2023. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML+NLP+Team%2E+2023%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E+Accessed%3A+2023%2D05%2D05%2E)
57. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
58. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023b%2E+Llama+2%3A+Open+foundation+and+f)
59. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2023. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170. — [arXiv:2307.03170](https://arxiv.org/abs/2307.03170) [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+2023%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+arXiv+preprint+arXiv%3A2307%2E0)
60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
61. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/](https://github.com/kingoflolz/) mesh-transformer-jax. [↗](https://github.com/kingoflolz/)
62. Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guangsheng Bao, Qian Wang, and Yue Zhang. 2024. Novelqa: A benchmark for long-range novel question answering. arXiv preprint arXiv:2403.12766. [find](https://scholar.google.com/scholar?q=Cunxiang+Wang%2C+Ruoxi+Ning%2C+Boqi+Pan%2C+Tonghui+Wu%2C+Qipeng+Guo%2C+Cheng+Deng%2C+Guangsheng+Bao%2C+Qian+Wang%2C+and+Yue+Zhang%2E+2024%2E+Novelqa%3A+A+benchmark+for+long%2Drange+novel+question+answering%2E+arXiv+preprint+ar)
63. Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2021. Memorizing transformers. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yuhuai+Wu%2C+Markus+Norman+Rabe%2C+DeLesley+Hutchins%2C+and+Christian+Szegedy%2E+2021%2E+Memorizing+transformers%2E+In+International+Conference+on+Learning+Representations%2E)
64. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
65. Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023. [DOC: Improving long story coherence with detailed outline control](https://md.research.elamur.ai/free/en/papers/yang-2023-doc-improving-long-story-coherence/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3378–3465, Toronto, Canada. Association for Comput…
66. Zi Yang and Nan Hua. 2024. Attendre: Wait to attend by retrieval with evicted queries in memory-based transformers for long context processing. arXiv preprint arXiv:2401.04881. — [arXiv:2401.04881](https://arxiv.org/abs/2401.04881) [find](https://scholar.google.com/scholar?q=Zi+Yang+and+Nan+Hua%2E+2024%2E+Attendre%3A+Wait+to+attend+by+retrieval+with+evicted+queries+in+memory%2Dbased+transformers+for+long+context+processing%2E+arXiv+preprint+arXiv%3A2401%2E04881%2E+%E2%80%94+%5BarXiv%3A2401%2E04881%5D%28ht)
67. Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong. 2021. Adaptive semiparametric language models. Transactions of the Association for Computational Linguistics, 9:362–373. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+and+Lingpeng+Kong%2E+2021%2E+Adaptive+semiparametric+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A362%E2%80%93373%2E)
68. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. 2024. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136. [find](https://scholar.google.com/scholar?q=Tao+Yuan%2C+Xuefei+Ning%2C+Dong+Zhou%2C+Zhijie+Yang%2C+Shiyao+Li%2C+Minghui+Zhuang%2C+Zheyue+Tan%2C+Zhuyu+Yao%2C+Dahua+Lin%2C+Boxun+Li%2C+et+al%2E+2024%2E+Lv%2Deval%3A+A+balanced+long%2Dcontext+benchmark+with+5+length+levels+up+to)
69. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in neural information processing systems, 33:17283–17297.
70. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. 2024a. Soaring from 4k to 400k: Extending llm’s context with activation beacon. arXiv preprint arXiv:2401.03462. [find](https://scholar.google.com/scholar?q=Peitian+Zhang%2C+Zheng+Liu%2C+Shitao+Xiao%2C+Ninglu+Shao%2C+Qiwei+Ye%2C+and+Zhicheng+Dou%2E+2024a%2E+Soaring+from+4k+to+400k%3A+Extending+llm%E2%80%99s+context+with+activation+beacon%2E+arXiv+preprint+arXiv%3A2401%2E03462%2E)
71. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024b. ∞ bench: Extending long context evaluation beyond 100k tokens. arXiv preprint arXiv:2402.13718. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Khai+Hao%2C+Xu+Han%2C+Zhen+Leng+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+et+al%2E+2024b%2E+%E2%88%9E+bench%3A+Extending+long+context+evaluation+beyond+100k+tok)
72. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024c. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797. — [arXiv:2403.04797](https://arxiv.org/abs/2403.04797) [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Runjin+Chen%2C+Shiwei+Liu%2C+Zhewei+Yao%2C+Olatunji+Ruwase%2C+Beidi+Chen%2C+Xiaoxia+Wu%2C+and+Zhangyang+Wang%2E+2024c%2E+Found+in+the+middle%3A+How+language+models+use+long+contexts+better+via+plug%2Dand%2Dpl)
73. Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. 2024d. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Ying+Sheng%2C+Tianyi+Zhou%2C+Tianlong+Chen%2C+Lianmin+Zheng%2C+Ruisi+Cai%2C+Zhao+Song%2C+Yuandong+Tian%2C+Christopher+R%C3%A9%2C+Clark+Barrett%2C+et+al%2E+2024d%2E+H2o%3A+Heavy%2Dhitter+oracle+for+efficient+generative)
74. Wangchunshu Zhou, Yuchen Eleanor Jiang, Peng Cui, Tiannan Wang, Zhenxin Xiao, Yifan Hou, Ryan Cotterell, and Mrinmaya Sachan. 2023. Recurrentgpt: Interactive generation of (arbitrarily) long text. arXiv preprint arXiv:2305.13304. [find](https://scholar.google.com/scholar?q=Wangchunshu+Zhou%2C+Yuchen+Eleanor+Jiang%2C+Peng+Cui%2C+Tiannan+Wang%2C+Zhenxin+Xiao%2C+Yifan+Hou%2C+Ryan+Cotterell%2C+and+Mrinmaya+Sachan%2E+2023%2E+Recurrentgpt%3A+Interactive+generation+of+%28arbitrarily%29+long+text%2E+arX)
75. Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2023. Pose: Efficient context window extension of llms via positional skip-wise training. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dawei+Zhu%2C+Nan+Yang%2C+Liang+Wang%2C+Yifan+Song%2C+Wenhao+Wu%2C+Furu+Wei%2C+and+Sujian+Li%2E+2023%2E+Pose%3A+Efficient+context+window+extension+of+llms+via+positional+skip%2Dwise+training%2E+In+The+Twelfth+International+)

*Updated 24 October 2026*