---
title: "Can Calibration of Positional Encodings Enhance Long Context"
description: "They then introduce CALIOPE, a framework that remaps the position indices passed to RoPE during inference. 29 references."
authors: ["Tom Zehle", "Matthias Aßenmacher"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.120/"
doi: "10.18653/v1/2026.findings-eacl.120"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zehle-2026-can-calibration-positional-encodings-enhance/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zehle-2026-can-calibration-positional-encodings-enhance/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/) / Can Calibration of Positional Encodings Enhance Long Context Utilization?

# Can Calibration of Positional Encodings Enhance Long Context Utilization?

**Authors:** [Tom Zehle](https://md.research.elamur.ai/free/en/authors/tom-zehle/) and [Matthias Aßenmacher](https://md.research.elamur.ai/free/en/authors/matthias-assenmacher/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.120/) · [PDF](https://aclanthology.org/2026.findings-eacl.120.pdf) · [DOI 10.18653/v1/2026.findings-eacl.120](https://doi.org/10.18653/v1/2026.findings-eacl.120) · [OpenAlex](https://openalex.org/W7140110649)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Large language models often fail to use long inputs well. Facts in the middle of a prompt tend to be overlooked (the "Lost in the Middle", LiM, effect), and recent tokens get extra weight (recency bias). The authors ask whether rotary position embeddings (RoPE) contribute to these biases and whether adjusting them can help.

They first show that both biases persist in current open models. They then introduce CALIOPE, a framework that remaps the position indices passed to RoPE during inference. It requires no fine-tuning, no access to hidden states and no parameter changes, so it can be dropped into existing RoPE-based models. Several calibrators are evaluated on needle-in-a-haystack retrieval (Natural-QA) and on reasoning across multiple chunks (Babilong-QA).

The effects depend on the task and the model. Some calibrators clearly flatten the U-shaped accuracy curve for Llama-3.1-8B on retrieval, but they slightly hurt its multi-fact reasoning. Qwen-2.5-14B shows the opposite pattern. The work matters because it positions RoPE remapping as a cheap, practical way to improve long-context use in RAG and similar settings.

## Key points
- **Models and data:** Llama-3.1-8B and Qwen-2.5-14B were tested on Natural-QA (single relevant chunk among distractors) and Babilong-QA (facts spread across the context).
- **Setup:** Contexts had 20, 50 or 100 chunks (about 3K, 8K and 16K tokens). The relevant chunk was placed at relative positions 0, 0.2, 0.5, 0.8 and 1.0. Each configuration used 500 questions and three seeds.
- **Evaluation:** Answers were scored by substring matching plus an LLM-as-a-judge check with Llama-3.3-70B. The strength of the U-shape was measured with a Pearson correlation between a position's distance from the middle and its accuracy.
- **Calibrators:** On Llama retrieval, the Moses and Decay calibrators flattened the positional curve. Hourglass raised overall accuracy without flattening it but underperformed on Qwen. Calibration hyperparameters were not tuned, so the results show feasibility rather than optimal performance.
- **Code:** The code is open-source at github.com/finitearth/caliope.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 113 works · 56,817 citations
- [Munich Center for Machine Learning](https://md.research.elamur.ai/free/en/universities/munich-center-for-machine-learning/) — University, Munich, Germany · 62 works · 1,167 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Tom Zehle and Matthias Aßenmacher. 2026. Can Calibration of Positional Encodings Enhance Long Context Utilization?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zehle2026can,
  title = {{Can Calibration of Positional Encodings Enhance Long Context Utilization?}},
  author = {Tom Zehle and Matthias Aßenmacher},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.120},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.120}
}
```

- [← previous Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences](https://md.research.elamur.ai/free/en/papers/modzelewski-2026-can-ai-generated-persuasion-be/)
- [next → Can Emotions Signal Gender? Investigating Implicit Cues in Human and LLM Translations of Amazon Reviews](https://md.research.elamur.ai/free/en/papers/manakhimova-2026-can-emotions-signal-gender-investigating/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/26/)

## References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. [Make your llm fully utilize the context](https://md.research.elamur.ai/free/en/papers/an-2024-make-llm-fully-utilize-context/). Preprint, arXiv:2404.16811.
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, and 1 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
3. Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. 2024. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205. — [arXiv:2410.06205](https://arxiv.org/abs/2410.06205) [find](https://scholar.google.com/scholar?q=Federico+Barbero%2C+Alex+Vitvitskyi%2C+Christos+Perivolaropoulos%2C+Razvan+Pascanu%2C+and+Petar+Velic%CC%8Ckovic%CC%81%2E+2024%2E+Round+and+round+we+go%21+what+makes+rotary+positional+encodings+useful%3F+arXiv+preprint+arXiv%3A2)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
5. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2E)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
7. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). arXiv preprint arXiv:2402.13753.
8. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, Liu YiBo, Sun Qianguo, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Proceedings of the 62nd Annual Meeting of… [find](https://scholar.google.com/scholar?q=Junqing+He%2C+Kunhao+Pan%2C+Xiaoqun+Dong%2C+Zhuoyang+Song%2C+Liu+YiBo%2C+Sun+Qianguo%2C+Yuxin+Liang%2C+Hao+Wang%2C+Enming+Zhang%2C+and+Jiaxing+Zhang%2E+2024%2E+Never+lost+in+the+middle%3A+Mastering+long%2Dcontext+question+answ)
9. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and 1 others. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008. [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Yung%2DSung+Chuang%2C+Chun%2DLiang+Li%2C+Zifeng+Wang%2C+Long+T+Le%2C+Abhishek+Kumar%2C+James+Glass%2C+Alexander+Ratner%2C+Chen%2DYu+Lee%2C+Ranjay+Krishna%2C+and+1+others%2E+2024%2E+Found+in+the+middle%3A+Calibratin)
10. Albert Qiaochu Jiang, Xiaohan Ren, Yixiao Chen, Michael Zhang, Tianle Wang, and 1 others. 2024. Longllmlingua: Compressing prompts for accelerating long-context llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Albert+Qiaochu+Jiang%2C+Xiaohan+Ren%2C+Yixiao+Chen%2C+Michael+Zhang%2C+Tianle+Wang%2C+and+1+others%2E+2024%2E+Longllmlingua%3A+Compressing+prompts+for+accelerating+long%2Dcontext+llms%2E+In+Proceedings+of+the+62nd+Annual)
11. Yuri Kuratov, Mikhail Belyaev, Nikita Smetanin, Aleksandr Shevchenko, Aleksandr Krotov, Maxim Arkhipov, and Andrey Trusov. 2024. Babilong: Benchmarking long-context reasoning in large language models. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Mikhail+Belyaev%2C+Nikita+Smetanin%2C+Aleksandr+Shevchenko%2C+Aleksandr+Krotov%2C+Maxim+Arkhipov%2C+and+Andrey+Trusov%2E+2024%2E+Babilong%3A+Benchmarking+long%2Dcontext+reasoning+in+large+language+models%2E)
12. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural que…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
14. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Processing Systems, volum…
15. Changqun Li, Linlin Wang, Xin Lin, Gerard de Melo, and Liang He. 2022. Curriculum prompt learning with self-training for abstractive dialogue summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1096–1106. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Changqun+Li%2C+Linlin+Wang%2C+Xin+Lin%2C+Gerard+de+Melo%2C+and+Liang+He%2E+2022%2E+Curriculum+prompt+learning+with+self%2Dtraining+for+abstractive+dialogue+summarization%2E+In+Proceedings+of+the+2022+Conference+on+Em)
16. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Percy Liang, and Tatsunori Hashimoto. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30.
18. Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, and Xiaoling Wang. 2024. Length generalization of causal transformers without position encoding. arXiv preprint arXiv:2404.12224. [find](https://scholar.google.com/scholar?q=Jie+Wang%2C+Tao+Ji%2C+Yuanbin+Wu%2C+Hang+Yan%2C+Tao+Gui%2C+Qi+Zhang%2C+Xuanjing+Huang%2C+and+Xiaoling+Wang%2E+2024%2E+Length+generalization+of+causal+transformers+without+position+encoding%2E+arXiv+preprint+arXiv%3A2404%2E12)
19. Ruochen Xiong, William Fedus, and 1 others. 2023. Effective long-range modeling with rotary position embeddings. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ruochen+Xiong%2C+William+Fedus%2C+and+1+others%2E+2023%2E+Effective+long%2Drange+modeling+with+rotary+position+embeddings%2E+In+NeurIPS%2E)
20. Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. 2025. Mitigate position bias in llms via scaling a single hidden states channel. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6092–6111. [find](https://scholar.google.com/scholar?q=Yijiong+Yu%2C+Huiqiang+Jiang%2C+Xufang+Luo%2C+Qianhui+Wu%2C+Chin%2DYew+Lin%2C+Dongsheng+Li%2C+Yuqing+Yang%2C+Yongfeng+Huang%2C+and+Lili+Qiu%2E+2025%2E+Mitigate+position+bias+in+llms+via+scaling+a+single+hidden+states+chann)
21. Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. 2023. Length-based curriculum learning for efficient pre-training of language models. New Generation Computing, 41(1):109–134. [find](https://scholar.google.com/scholar?q=Koichi+Nagatsuka%2C+Clifford+Broni%2DBediako%2C+and+Masayasu+Atsumi%2E+2023%2E+Length%2Dbased+curriculum+learning+for+efficient+pre%2Dtraining+of+language+models%2E+New+Generation+Computing%2C+41%281%29%3A109%E2%80%93134%2E)
22. Ken M Nakanishi. 2025. Scalable-softmax is superior for attention. arXiv preprint arXiv:2501.19399. — [arXiv:2501.19399](https://arxiv.org/abs/2501.19399) [find](https://scholar.google.com/scholar?q=Ken+M+Nakanishi%2E+2025%2E+Scalable%2Dsoftmax+is+superior+for+attention%2E+arXiv+preprint+arXiv%3A2501%2E19399%2E+%E2%80%94+%5BarXiv%3A2501%2E19399%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E19399%29)
23. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
24. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2023%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2E)
25. Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting for in-context learning. In Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Alexander+Peysakhovich+and+Adam+Lerer%2E+2023%2E+Attention+sorting+for+in%2Dcontext+learning%2E+In+Advances+in+Neural+Information+Processing+Systems%2E)
26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
27. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 2, pages 464–468.
28. Zhen Sun, Yujie Zhang, Jian Zhang, Qiang Liu, Zhiqiang Wang, and 1 others. 2021. [RoFormer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). arXiv preprint arXiv:2104.09864.
29. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.

*Updated 2 November 2026*