---
title: "Adaptive Zooming via Relevance-Informed Positional Resource"
description: "The authors propose RiPRA, which treats positional encoding as distributing a fixed budget (the pre-training length) across a… 40 references."
authors: ["Hongbo Zhao", "Huibin Wang", "Bin Tang", "Xianming Hu", "Yihong Huang", "Yijun Shen", "Nuoyi Chen", "Ping Li", "Kai Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1058/"
doi: "10.18653/v1/2026.findings-acl.1058"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2026-adaptive-zooming-relevance-informed-positional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2026-adaptive-zooming-relevance-informed-positional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension

# Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension

**Authors:** [Hongbo Zhao](https://md.research.elamur.ai/free/en/authors/hongbo-zhao/), Huibin Wang, Bin Tang, [Xianming Hu](https://md.research.elamur.ai/free/en/authors/xianming-hu/), [Yihong Huang](https://md.research.elamur.ai/free/en/authors/yihong-huang/), Yijun Shen, [Nuoyi Chen](https://md.research.elamur.ai/free/en/authors/nuoyi-chen/), [Ping Li](https://md.research.elamur.ai/free/en/authors/ping-li/) and [Kai Zhang](https://md.research.elamur.ai/free/en/authors/kai-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1058/) · [PDF](https://aclanthology.org/2026.findings-acl.1058.pdf) · [DOI 10.18653/v1/2026.findings-acl.1058](https://doi.org/10.18653/v1/2026.findings-acl.1058) · [OpenAlex](https://openalex.org/W7166847147)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLMs that use RoPE lose quality when inputs run past their pre-training context length. Existing training-free fixes, such as position reuse (Self-Extend, ChunkLlama, AdaGroPE) or frequency interpolation (NTK, YaRN, GALI), remap positions using distance alone and ignore content. As a result, important and unimportant tokens get compressed equally. The authors propose RiPRA, which treats positional encoding as distributing a fixed budget (the pre-training length) across a longer input. Tokens that are semantically close to the current query receive finer positional resolution, while less relevant regions are squeezed together. Relevance is computed from query–key dot products taken before positions are applied, at the chunk level, and only in a few "anchor" layers. Isotonic regression then enforces a monotonic, order-preserving mapping, and the result is rescaled to fit the budget. The method needs no training or auxiliary model. On LongBench, L-Eval and Passkey Retrieval it consistently beats prior training-free extension methods, and it adds only small latency. This shows that conditioning positions on content can improve long-context generalization.

## Key points
- **Core idea:** RiPRA assigns each token a positional "derivative" between 0 and 1, and these values must sum to the pre-training length. A short immediate window next to the query always gets full resolution.
- **Setup and hyperparameters:**
  - The backbone is Llama-3-8B-Instruct.
  - The best chunk size is S = 256. It even beat S = 64 on Musique, plausibly because very small chunks break up coherent content.
  - Cutting anchor layers from 32 to 2 costs very little accuracy, but using only layer 0 hurts noticeably.
- **Immediate window size:** In the ablation, NarrativeQA F1 rose from a 21.71 baseline to 26.11 with a 1024-token window, and Musique F1 rose from 21.49 to about 32. A 2048-token window degraded slightly on both datasets.
- **Cost:** Compared with SelfExtend on an NVIDIA A6000, per-token latency increases by 3.2–6.7% for contexts of 12k–32k. Extra memory is only about 2.6 MB at 32k.
- **Limitation:** The method was evaluated only on English text-only benchmarks. Multilingual use, including Ukrainian, has not been verified.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [East China Normal University](https://md.research.elamur.ai/free/en/universities/east-china-normal-university/) — University, Shanghai, China · 69 works · 2,490 citations
- [Southwest Petroleum University](https://md.research.elamur.ai/free/en/universities/southwest-petroleum-university/) — University, Chengdu, China · 2 works · 35 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hongbo Zhao, Huibin Wang, Bin Tang, Xianming Hu, Yihong Huang, Yijun Shen, Nuoyi Chen, Ping Li and Kai Zhang. 2026. Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2026adaptive,
  title = {{Adaptive Zooming via Relevance-Informed Positional Resource Allocation for Training-free LLM Context Extension}},
  author = {Hongbo Zhao, Huibin Wang, Bin Tang, Xianming Hu, Yihong Huang, Yijun Shen, Nuoyi Chen, Ping Li and Kai Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1058},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1058}
}
```

- [← previous Adaptive Text2GQL: Integrating Structural Twig Linking and Evolutionary In-Context Learning](https://md.research.elamur.ai/free/en/papers/niu-2026-adaptive-text2gql-integrating-structural-twig/)
- [next → AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations](https://md.research.elamur.ai/free/en/papers/yao-2026-adaptivek-complexity-driven-sparse-autoencoders/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

40 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. [Longlora: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). arXiv preprint arXiv:2309.12307.
2. Jishnu Ray Chowdhury and Cornelia Caragea. 2023. Monotonic location attention for length generalization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 28792–28808. PMLR. [find](https://scholar.google.com/scholar?q=Jishnu+Ray+Chowdhury+and+Cornelia+Caragea%2E+2023%2E+Monotonic+location+attention+for+length+generalization%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2023%2C+23%2D29+July+2023%2C+Honolulu%2C+Hawaii%2C+)
3. Jan De Leeuw, Kurt Hornik, and Patrick Mair. 2010. Isotone optimization in r: pool-adjacent-violators algorithm (pava) and active set methods. Journal of statistical software, 32:1–24. [find](https://scholar.google.com/scholar?q=Jan+De+Leeuw%2C+Kurt+Hornik%2C+and+Patrick+Mair%2E+2010%2E+Isotone+optimization+in+r%3A+pool%2Dadjacent%2Dviolators+algorithm+%28pava%29+and+active+set+methods%2E+Journal+of+statistical+software%2C+32%3A1%E2%80%9324%2E)
4. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753.
5. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
6. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024a. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
7. Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022. [Position information in transformers: An overview](https://md.research.elamur.ai/free/en/papers/dufter-2022-position-information-transformers-overview/). Comput. Linguistics, 48(3):733–763.
8. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024b. Training-free long-context scaling of large language models. arXiv preprint arXiv:2402.17463. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+2024b%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E17463%2E+%E2%80%94+%5BarXiv)
9. Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, and Songlin Hu. 2025. Nextlong: Toward effective long-context training without long documents. arXiv preprint arXiv:2501.12766. — [arXiv:2501.12766](https://arxiv.org/abs/2501.12766) [find](https://scholar.google.com/scholar?q=Chaochen+Gao%2C+Xing+Wu%2C+Zijia+Lin%2C+Debing+Zhang%2C+and+Songlin+Hu%2E+2025%2E+Nextlong%3A+Toward+effective+long%2Dcontext+training+without+long+documents%2E+arXiv+preprint+arXiv%3A2501%2E12766%2E+%E2%80%94+%5BarXiv%3A2501%2E12766%5D%28htt)
10. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). In Proceedings of the 62nd annual meeting of the association for computational lin…
11. Olga Golovneva, Tianlu Wang, Jason Weston, and Sainbayar Sukhbaatar. 2024. Contextual position encoding: Learning to count what’s important. arXiv preprint arXiv:2405.18719. bloc97. 2023. NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minima… — [arXiv:2405.18719](https://arxiv.org/abs/2405.18719) [find](https://scholar.google.com/scholar?q=Olga+Golovneva%2C+Tianlu+Wang%2C+Jason+Weston%2C+and+Sainbayar+Sukhbaatar%2E+2024%2E+Contextual+position+encoding%3A+Learning+to+count+what%E2%80%99s+important%2E+arXiv+preprint+arXiv%3A2405%2E18719%2E+bloc97%2E+2023%2E+NTK%2DAware+Sc)
12. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2024. CLEX: continuous length extrapolation for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Guanzheng+Chen%2C+Xin+Li%2C+Zaiqiao+Meng%2C+Shangsong+Liang%2C+and+Lidong+Bing%2E+2024%2E+CLEX%3A+continuous+length+extrapolation+for+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Repre)
13. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023a. Extending context window of large language models via positional interpolation. ArXiv, abs/2306.15595. emozilla. 2023. Dynamically Scaled RoPE further increases performance of long context LLaMA with zero fine-tuning. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023a%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+ArXiv%2C+abs%2F2306%2E15595%2E+emozilla%2E+2023%2E+Dynamicall)
14. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. [Lm-infinite: Zero-shot extreme length generalization for large language models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T…
15. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. [RULER: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) In First Conference on Language Modeling.
16. Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, and 1 others. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3. [find](https://scholar.google.com/scholar?q=Mojan+Javaheripi%2C+S%C3%A9bastien+Bubeck%2C+Marah+Abdin%2C+Jyoti+Aneja%2C+Sebastien+Bubeck%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Weizhu+Chen%2C+Allie+Del+Giorno%2C+Ronen+Eldan%2C+Sivakanth+Gopi%2C+and+1+others%2E+2023%2E+Phi%2D2%3A+The+su)
17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
18. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024a. [Llm maybe longlm: Self-extend llm context window without tuning](https://md.research.elamur.ai/free/en/papers/jin-2024-llm-maybe-longlm-self-extend/). arXiv preprint arXiv:2401.01325.
19. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024b. LLM maybe longlm: Selfextend LLM context window without tuning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. Open-Review.… [find](https://scholar.google.com/scholar?q=Hongye+Jin%2C+Xiaotian+Han%2C+Jingfeng+Yang%2C+Zhimeng+Jiang%2C+Zirui+Liu%2C+Chia%2DYuan+Chang%2C+Huiyuan+Chen%2C+and+Xia+Hu%2E+2024b%2E+LLM+maybe+longlm%3A+Selfextend+LLM+context+window+without+tuning%2E+In+Forty%2Dfirst+Inte)
20. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. 2023. [The impact of positional encoding on length generalization in transformers](https://md.research.elamur.ai/free/en/papers/kazemnejad-2023-impact-positional-encoding-length-generalization/). In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,…
21. Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. BOOKSUM: A collection of datasets for long-form narrative summarization. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pag… [find](https://scholar.google.com/scholar?q=Wojciech+Kryscinski%2C+Nazneen+Rajani%2C+Divyansh+Agarwal%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+2022%2E+BOOKSUM%3A+A+collection+of+datasets+for+long%2Dform+narrative+summarization%2E+In+Findings+of+the+Association+)
22. Rongsheng Li, Jin Xu, Zhixiong Cao, Hai-Tao Zheng, and Hong-Gee Kim. 2024. Extending context window in large language models with segmented base adjustment for rotary position embeddings. Applied Sciences, 14(7):3076. [find](https://scholar.google.com/scholar?q=Rongsheng+Li%2C+Jin+Xu%2C+Zhixiong+Cao%2C+Hai%2DTao+Zheng%2C+and+Hong%2DGee+Kim%2E+2024%2E+Extending+context+window+in+large+language+models+with+segmented+base+adjustment+for+rotary+position+embeddings%2E+Applied+Scie)
23. Yan Li, Tianyi Zhang, Zechuan Li, and Caren Han. 2025. A training-free length extrapolation approach for llms: Greedy attention logit interpolation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8784– 8804. [find](https://scholar.google.com/scholar?q=Yan+Li%2C+Tianyi+Zhang%2C+Zechuan+Li%2C+and+Caren+Han%2E+2025%2E+A+training%2Dfree+length+extrapolation+approach+for+llms%3A+Greedy+attention+logit+interpolation%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+)
24. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2023%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2E)
25. Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+The+Tenth+International+Conference+on+Learning+Representati)
26. Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. 2019. [Compressive transformers for long-range sequence modelling](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/). arXiv preprint arXiv:1911.05507.
27. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023a. Randomized positional encodings boost length generalization of transformers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+2023a%2E+Randomized+positional+encodings+boost+length+generalization+of+transfo)
28. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023b. Randomized positional encodings boost length generalization of transformers. arXiv preprint arXiv:2305.16843. [find](https://scholar.google.com/scholar?q=Anian+Ruoss%2C+Gr%C3%A9goire+Del%C3%A9tang%2C+Tim+Genewein%2C+Jordi+Grau%2DMoya%2C+R%C3%B3bert+Csord%C3%A1s%2C+Mehdi+Bennani%2C+Shane+Legg%2C+and+Joel+Veness%2E+2023b%2E+Randomized+positional+encodings+boost+length+generalization+of+transfo)
29. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
30. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. [Long range arena : A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, M…
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
32. Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. 2024. [Scaling laws of rope-based extrapolation](https://md.research.elamur.ai/free/en/papers/liu-2023-scaling-laws-rope-based/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
34. Amirkeivan Mohtashami and Martin Jaggi. 2023. Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Amirkeivan+Mohtashami+and+Martin+Jaggi%2E+2023%2E+Random%2Daccess+infinite+context+length+for+transformers%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Systems%2E)
35. Yu Wang, Sheng Shen, Rémi Munos, Hongyuan Zhan, and Yuandong Tian. 2025. Positional encoding via token-aware phase attention. arXiv preprint arXiv:2509.12635. — [arXiv:2509.12635](https://arxiv.org/abs/2509.12635) [find](https://scholar.google.com/scholar?q=Yu+Wang%2C+Sheng+Shen%2C+R%C3%A9mi+Munos%2C+Hongyuan+Zhan%2C+and+Yuandong+Tian%2E+2025%2E+Positional+encoding+via+token%2Daware+phase+attention%2E+arXiv+preprint+arXiv%3A2509%2E12635%2E+%E2%80%94+%5BarXiv%3A2509%2E12635%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
36. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2E)
37. Xinhao Xu, Jiaxin Li, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. 2025. Extending llm context window with adaptive grouped positional encoding: A training-free method. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5… [find](https://scholar.google.com/scholar?q=Xinhao+Xu%2C+Jiaxin+Li%2C+Hui+Chen%2C+Zijia+Lin%2C+Jungong+Han%2C+and+Guiguang+Ding%2E+2025%2E+Extending+llm+context+window+with+adaptive+grouped+positional+encoding%3A+A+training%2Dfree+method%2E+In+Proceedings+of+the+6)
38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
39. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, and 1 others. 2023. [Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery an…
40. Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summarization. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligenc… [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Yang+Liu%2C+Yichong+Xu%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+2022%2E+Dialoglm%3A+Pre%2Dtrained+model+for+long+dialogue+understanding+and+summarization%2E+In+Thirty%2DSixth+AAAI+Conference+on+Artificial+In)

*Updated 2 November 2026*