---
title: "LLM Maybe LongLM — paper by Jin et al. (2024)"
description: "In this ICML 2024 paper, the authors argue that the problem is not a lack of long-text understanding. 43 references, cited by 65 papers here."
authors: ["Hongye Jin", "Xiaotian Han", "Jingfeng Yang", "Zhimeng Jiang", "Zirui Liu", "Chia-Yuan Chang", "Huiyuan Chen", "Xia Hu"]
year: 2024
venue: "arXiv:2401.01325"
source: "https://arxiv.org/abs/2401.01325"
doi: "10.48550/arxiv.2401.01325"
arxiv: "2401.01325"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

# LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

**Authors:** [Hongye Jin](https://md.research.elamur.ai/free/en/authors/hongye-jin/), [Xiaotian Han](https://md.research.elamur.ai/free/en/authors/xiaotian-han/), [Jingfeng Yang](https://md.research.elamur.ai/free/en/authors/jingfeng-yang/), Zhimeng Jiang, [Zirui Liu](https://md.research.elamur.ai/free/en/authors/zirui-liu/), Chia-Yuan Chang, Huiyuan Chen and [Xia Hu](https://md.research.elamur.ai/free/en/authors/xia-hu/) · **Year:** 2024 · **Venue:** arXiv:2401.01325 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.01325) · [PDF](https://arxiv.org/pdf/2401.01325) · [DOI 10.48550/arxiv.2401.01325](https://doi.org/10.48550/arxiv.2401.01325) · [arXiv 2401.01325](https://arxiv.org/abs/2401.01325) · [OpenAlex](https://openalex.org/W6903985660)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [65](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models that use RoPE usually fail on inputs longer than their pretraining sequence length. Perplexity spikes and outputs become erratic. In this ICML 2024 paper, the authors argue that the problem is not a lack of long-text understanding. Instead, they blame a "positional O.O.D." effect: at inference, the model meets relative distances it never saw during training.

Their fix, SelfExtend, needs no fine-tuning and only a small code change. It mixes two kinds of attention, both computed with the model's own weights:
- **Neighbor attention** handles tokens inside a fixed local window, as usual.
- **Grouped attention** handles distant tokens. Their positions are compressed by floor division by a group size, so every relative distance stays within the pretrained range.

The authors test the method on Llama-2, Mistral, SOLAR and Phi-2. Tasks cover language modeling on PG-19, synthetic passkey retrieval and real-world LongBench tasks. SelfExtend keeps perplexity stable on longer inputs and often matches or beats long-context models built by fine-tuning. This matters because it is a cheap way to widen context windows without scarce long-text data or costly retraining.

## Key points
- **Core mechanism:** a FLOOR (`//`) operation with group size G_s maps large, unseen relative positions to ones seen in pretraining. Exact nearby positions are kept inside a neighbor window, for example 1024 tokens.
- **PG-19 perplexity:** Llama-2-7b-chat is stable at 4k (4096) tokens but explodes at 6k (6144). With grouped attention, it stays low and stable on longer sequences.
- **Passkey retrieval:** SelfExtend on Llama-2-chat-7b was compared with fine-tuned models: vicuna-1.5-7b-16k, llama-2-7b-32k, Yarn-llama-2-7b-64k and Llama-2-7b-Longlora-16k. Group sizes tested were 8, 12, 16 and 100,000. A harder variant used passkeys longer than the usual 5 digits, and SelfExtend kept its advantage without tuning.
- **LongBench:** longer contexts generally helped, with steady gains on NarrativeQA and Qmsum. Some datasets peaked and then declined, because large group sizes blur positional detail. Lcc stayed flat, likely because its code tasks are local and its inputs short.
- **Limitations:** naive implementations add extra attention cost, which blocked kernels such as Flash Attention largely offset. Quality drops with very large group sizes, so context cannot grow indefinitely. Code is available at github.com/datamllab/LongLM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen and Xia Hu. 2024. LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning. arXiv:2401.01325.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jin2024llm,
  title = {{LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning}},
  author = {Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen and Xia Hu},
  year = {2024},
  doi = {10.48550/arxiv.2401.01325},
  eprint = {2401.01325},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.01325}
}
```

- [← previous MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://md.research.elamur.ai/free/en/papers/ge-2024-mart-improving-llm-safety-multi/)
- [next → Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://md.research.elamur.ai/free/en/papers/shaham-2024-multilingual-instruction-tuning-just-pinch/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

43 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. amazon. Mistrallite model. [https://huggingface.co/](https://huggingface.co/) amazon/MistralLite, 2023. [Online; accessed 29-December-2023… [↗](https://huggingface.co/)
2. Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359, 2022. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+Re%CC%81%2C+C%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A16344%E2%80%9316359%2C+2022%2E)
3. An, C., Gong, S., Zhong, M., Li, M., Zhang, J., Kong, L., and Qiu, X. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). arXiv preprint arXiv:2307.11088, 2023.
4. Bai, T., Luo, J., Zhao, J., Wen, B., and Wang, Q. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01356, 2021. — [arXiv:2102.01356](https://arxiv.org/abs/2102.01356) [find](https://scholar.google.com/scholar?q=Bai%2C+T%2E%2C+Luo%2C+J%2E%2C+Zhao%2C+J%2E%2C+Wen%2C+B%2E%2C+and+Wang%2C+Q%2E+Recent+advances+in+adversarial+training+for+adversarial+robustness%2E+arXiv+preprint+arXiv%3A2102%2E01356%2C+2021%2E+%E2%80%94+%5BarXiv%3A2102%2E01356%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
5. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Abbasi%2C+B%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+Le+Noac%E2%80%99h%2C+A%2E%2C+Li%2C+H%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Ociepa%2C+C%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Schoelko)
6. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., et al. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023. gkamradt. Llmtest needleinahaystack: Doing simple retrieval from llm models. https:/…
7. Anothropic. Long context prompting for claude 2.1. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) claude-2-1-prompting, 2023. [↗](https://www.anthropic.com/news/)
8. Han, C., Wang, Q., Xiong, W., Chen, Y., Ji, H., and Wang, S. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137, 2023. context lengths in llms. arXiv preprint arXiv:2308.10882, 2023. — [arXiv:2308.16137](https://arxiv.org/abs/2308.16137) [find](https://scholar.google.com/scholar?q=Han%2C+C%2E%2C+Wang%2C+Q%2E%2C+Xiong%2C+W%2E%2C+Chen%2C+Y%2E%2C+Ji%2C+H%2E%2C+and+Wang%2C+S%2E+Lm%2Dinfinite%3A+Simple+on%2Dthe%2Dfly+length+generalization+for+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E16137%2C+2023%2E+context+lengths+in+l)
9. Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Peng%2C+B%2E%2C+Quesnelle%2C+J%2E%2C+Fan%2C+H%2E%2C+and+Shippole%2C+E%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E)
10. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
11. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2C+2021%2E)
12. Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Giorno, A. D., Eldan, R., Gopi, S., Gunasekar, S., Javaheripi, M., Kauffmann, P., Lee, Y. T., Li, Y., Nguyen, A., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Santacroce, M., Behl, H. S., Kalai, A. T., Wang… [find](https://scholar.google.com/scholar?q=Javaheripi%2C+M%2E%2C+Bubeck%2C+S%2E%2C+Abdin%2C+M%2E%2C+Aneja%2C+J%2E%2C+Bubeck%2C+S%2E%2C+Mendes%2C+C%2E+C%2E+T%2E%2C+Chen%2C+W%2E%2C+Giorno%2C+A%2E+D%2E%2C+Eldan%2C+R%2E%2C+Gopi%2C+S%2E%2C+Gunasekar%2C+S%2E%2C+Javaheripi%2C+M%2E%2C+Kauffmann%2C+P%2E%2C+Lee%2C+Y%2E+T%2E%2C+Li%2C+Y%2E%2C+Nguyen%2C+)
13. Rae, J. W., Potapenko, A., Jayakumar, S. M., and Lillicrap, T. P. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint arXiv:1911.05507, 2019.
14. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
15. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023a)
16. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
17. Jiang, H., Wu, Q., , Luo, X., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. [LongLLMLingua: Accelerating and enhancing llms in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/). ArXiv preprint, abs/2310.06839, 2023b. URL https://arxiv.org/abs/2310.06839. [↗](https://arxiv.org/abs/2310.06839)
18. Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021. — [arXiv:2108.13624](https://arxiv.org/abs/2108.13624) [find](https://scholar.google.com/scholar?q=Shen%2C+Z%2E%2C+Liu%2C+J%2E%2C+He%2C+Y%2E%2C+Zhang%2C+X%2E%2C+Xu%2C+R%2E%2C+Yu%2C+H%2E%2C+and+Cui%2C+P%2E+Towards+out%2Dof%2Ddistribution+generalization%3A+A+survey%2E+arXiv+preprint+arXiv%3A2108%2E13624%2C+2021%2E+%E2%80%94+%5BarXiv%3A2108%2E13624%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
19. Shi, H., Gao, J., Ren, X., Xu, H., Liang, X., Li, Z., and Kwok, J. T.-Y. Sparsebert: Rethinking the importance analysis in self-attention. In International Conference on Machine Learning, pp. 9547–9557. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Shi%2C+H%2E%2C+Gao%2C+J%2E%2C+Ren%2C+X%2E%2C+Xu%2C+H%2E%2C+Liang%2C+X%2E%2C+Li%2C+Z%2E%2C+and+Kwok%2C+J%2E+T%2E%2DY%2E+Sparsebert%3A+Rethinking+the+importance+analysis+in+self%2Dattention%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+9547%E2%80%93955)
20. Ke, G., He, D., and Liu, T.-Y. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020. [find](https://scholar.google.com/scholar?q=Ke%2C+G%2E%2C+He%2C+D%2E%2C+and+Liu%2C+T%2E%2DY%2E+Rethinking+positional+encoding+in+language+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2006%2E15595%2C+2020%2E)
21. Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.[15166, 2023](https://md.research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/).
22. Su, J. Rectified rotary position embeddings. [https://github.com/bojone/rerope](https://github.com/bojone/rerope), 2023. [↗](https://github.com/bojone/rerope)
23. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. [RoFormer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2022. arXiv: 2104.09864.
24. Lin, S., Hilton, J., and Evans, O. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2021.
25. Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. [Kivi: A tuning-free asymmetric 2bit quantization for kv cache](https://md.research.elamur.ai/free/en/papers/liu-2024-kivi-tuning-free-asymmetric-2bit/). arXiv preprint arXiv:2402.02750, 2024.
26. Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
27. Mohtashami, A. and Jaggi, M. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300, 2023. [find](https://scholar.google.com/scholar?q=Mohtashami%2C+A%2E+and+Jaggi%2C+M%2E+Landmark+attention%3A+Random%2Daccess+infinite+context+length+for+transformers%2E+arXiv+preprint+arXiv%3A2305%2E16300%2C+2023%2E)
28. Team, M. N. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www. mosaicml.com/blog/mpt-7b. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=Team%2C+M%2E+N%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2C+2023%2E+URL+www%2E+mosaicml%2Ecom%2Fblog%2Fmpt%2D7b%2E+Accessed%3A+2023%2D05%2D05%2E)
29. Pal, A., Karkhanis, D., Roberts, M., Dooley, S., Sundararajan, A., and Naidu, S. Giraffe: Adventures in expanding Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Liu, J., Shen, Z., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towar… — [arXiv:2108.13624](https://arxiv.org/abs/2108.13624) [find](https://scholar.google.com/scholar?q=Pal%2C+A%2E%2C+Karkhanis%2C+D%2E%2C+Roberts%2C+M%2E%2C+Dooley%2C+S%2E%2C+Sundararajan%2C+A%2E%2C+and+Naidu%2C+S%2E+Giraffe%3A+Adventures+in+expanding+Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+)
30. Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Bhosale%2C+S%2E%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2307%2E09288%2C+2023%2E)
31. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 2023.
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017.
33. Zhu, D., Yang, N., Wang, L., Song, Y., Wu, W., Wei, F., and Li, S. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400, 2023. [find](https://scholar.google.com/scholar?q=Zhu%2C+D%2E%2C+Yang%2C+N%2E%2C+Wang%2C+L%2E%2C+Song%2C+Y%2E%2C+Wu%2C+W%2E%2C+Wei%2C+F%2E%2C+and+Li%2C+S%2E+Pose%3A+Efficient+context+window+extension+of+llms+via+positional+skip%2Dwise+training%2E+arXiv+preprint+arXiv%3A2309%2E10400%2C+2023%2E)
34. Wu, K., Peng, H., Chen, M., Fu, J., and Chao, H. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10033–10041, 2021. [find](https://scholar.google.com/scholar?q=Wu%2C+K%2E%2C+Peng%2C+H%2E%2C+Chen%2C+M%2E%2C+Fu%2C+J%2E%2C+and+Chao%2C+H%2E+Rethinking+and+improving+relative+position+encoding+for+vision+transformer%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C)
35. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=Xiao%2C+G%2E%2C+Tian%2C+Y%2E%2C+Chen%2C+B%2E%2C+Han%2C+S%2E%2C+and+Lewis%2C+M%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E)
36. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039, 2023.
37. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). arXiv preprint arXiv:2010.11934, 2020.
38. Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., and Hu, X. [Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/). arXiv preprint arXiv:2304.13712, 2023.
39. Yin Song and Chen Wu and Eden Duthie. amazon/Mistral-Lite, 2023. URL [https://huggingface.co/amazon/](https://huggingface.co/amazon/) MistralLite. [↗](https://huggingface.co/amazon/)
40. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). Advances in neural information processing systems, 33:17283–17297, 2020.
41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
42. Zhang, J., Chao, H., Dhurandhar, A., Chen, P.-Y., Tajer, A., Xu, Y., and Yan, P. When neural networks fail to generalize? a model sensitivity perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11219–11227, 2023. [find](https://scholar.google.com/scholar?q=Zhang%2C+J%2E%2C+Chao%2C+H%2E%2C+Dhurandhar%2C+A%2E%2C+Chen%2C+P%2E%2DY%2E%2C+Tajer%2C+A%2E%2C+Xu%2C+Y%2E%2C+and+Yan%2C+P%2E+When+neural+networks+fail+to+generalize%3F+a+model+sensitivity+perspective%2E+In+Proceedings+of+the+AAAI+Conference+on+Arti)
43. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022.

*Updated 24 October 2026*