---
title: "Mitigating Posterior Salience Attenuation in Long-Context LLMs"
description: "The authors analyze this from the decoding side and describe Posterior Salience Attenuation… 81.80 F1. 20 references, cited by 5 papers here."
authors: ["Zikai Xiao", "Ziyang Wang", "Wen Ma", "Yan Zhang", "Wei Shen", "Yan Wang", "Luqi Gong", "Zuozhu Liu"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2025.acl-short.58/"
doi: "10.18653/v1/2025.acl-short.58"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiao-2025-mitigating-posterior-salience-attenuation-long/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiao-2025-mitigating-posterior-salience-attenuation-long/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

# Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding

**Authors:** [Zikai Xiao](https://md.research.elamur.ai/free/en/authors/zikai-xiao/), [Ziyang Wang](https://md.research.elamur.ai/free/en/authors/ziyang-wang/), Wen Ma, [Yan Zhang](https://md.research.elamur.ai/free/en/authors/yan-zhang/), [Wei Shen](https://md.research.elamur.ai/free/en/authors/wei-shen/), [Yan Wang](https://md.research.elamur.ai/free/en/authors/yan-wang/), Luqi Gong and [Zuozhu Liu](https://md.research.elamur.ai/free/en/authors/zuozhu-liu/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-short.58/) · [PDF](https://aclanthology.org/2025.acl-short.58.pdf) · [DOI 10.18653/v1/2025.acl-short.58](https://doi.org/10.18653/v1/2025.acl-short.58) · [OpenAlex](https://openalex.org/W4412889510)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/xiao-2025-mitigating-posterior-salience-attenuation-long/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Long-context LLMs often perform worse as inputs grow, even when the input fits inside their nominal context window. The authors analyze this from the decoding side and describe Posterior Salience Attenuation (PSA). As context length increases, the gold token's rank-based salience score drops. Yet the correct answer usually stays near the top of the vocabulary ranking. Error analysis suggests models over-favor "proximal" tokens near the query.

To address this, they propose Positional Contrastive Decoding (PCD), a training-free method. It builds a local-aware attention variant by over-rotating RoPE's lower-frequency components, using a reduced base and a smooth transition across frequency blocks. Its logits are then subtracted from the standard long-aware logits, restricted to the top-ranked candidate tokens. A single-layer simulation indicates this slows long-range attention decay.

Experiments use Llama3-8B variants on RULER, InfiniteBench and LongBench. PCD mostly beats decoding baselines and calibration baselines, though a few settings show small drops. This makes it a cheap inference-time option for better use of long contexts.

## Key points
- **Motivating observation:** in a key-value retrieval task, the gold token typically ranked within the top 8 candidates despite a vocabulary of about 128k. This motivates boosting already-high-ranked answers rather than retraining.
- **Models, benchmarks and baselines:**
  - Models: Llama3-8B-8k plus Gradient AI's long-context Llama3-8B-262k and Llama3-8B-1048k.
  - Benchmarks: RULER, InfiniteBench and LongBench (LongBench tested on the 262k model only).
  - Baselines: greedy, beam search, DoLa, MsPoE, Segment-Reranking and Rephrasing.
- **Main gains on the 262k model:**
  - InfiniteBench KV retrieval at 8k improved from 72.0% to 79.0%.
  - RULER Variable Tracking at 4k improved from 74.02 to 81.80 F1.
  - LongBench average reached 26.87, versus 25.98 for the base model and 26.27 for MsPoE.
- **Hyperparameter ablation (InfiniteBench):**
  - Best contrast coefficient: β = 2.5.
  - Best base ratio: B/B′ = 10⁴.
  - Transition coefficient α (about 0.1–0.2) and top-γ (about 20–30) were stable across settings.
- **Limitations:**
  - PCD does not enlarge the attention window.
  - Gains on short-text tasks are small.
  - Effectiveness may depend on the positional encoding scheme.
  - Results were not uniformly positive, e.g., a slight drop for the 8k model at 8k on KV retrieval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Science and Technology of China](https://md.research.elamur.ai/free/en/universities/university-of-science-and-technology-of-china/) — University, Hefei, China · 108 works · 112,698 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 172 works · 12,988 citations
- [Zhejiang Lab](https://md.research.elamur.ai/free/en/labs/zhejiang-lab/) — Research institute, Hangzhou, China · 7 works · 1,596 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, Yan Wang, Luqi Gong and Zuozhu Liu. 2025. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xiao2025mitigating,
  title = {{Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding}},
  author = {Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, Yan Wang, Luqi Gong and Zuozhu Liu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.58},
  url = {https://doi.org/10.18653/v1/2025.acl-short.58}
}
```

- [← previous ME2-BERT: Are Events and Emotions what you need for Moral Foundation Prediction?](https://md.research.elamur.ai/free/en/papers/zangari-2025-me2-bert-are-events-emotions/)
- [next → MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering](https://md.research.elamur.ai/free/en/papers/bahaj-2025-mizanqa-benchmarking-large-language-models/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Zexiong+Ma%2C+Zeqi+Lin%2C+Nanning+Zheng%2C+and+Jian%2DGuang+Lou%2E+2024%2E+Make+your+llm+fully+utilize+the+context%2E+arXiv+preprint+arXiv%3A2404%2E16811%2E)
2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+2024%2E+LongBench%3A+A+bilingual%2C+multi%2Dtask)
3. Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, et al. 2025. CoT-Kinetics: A theoretical modeling assessing lrm reasoning process. arXiv preprint arXiv:2505.13408. — [arXiv:2505.13408](https://arxiv.org/abs/2505.13408) [find](https://scholar.google.com/scholar?q=Jinhe+Bi%2C+Danqi+Yan%2C+Yifan+Wang%2C+Wenke+Huang%2C+Haokun+Chen%2C+Guancheng+Wan%2C+Mang+Ye%2C+Xun+Xiao%2C+Hinrich+Schuetze%2C+Volker+Tresp%2C+et+al%2E+2025%2E+CoT%2DKinetics%3A+A+theoretical+modeling+assessing+lrm+reasoning+p)
4. Ta-Chung Chi, Ting-Han Fan, Peter J Ramadge, and Alexander Rudnicky. 2022. Kerple: Kernelized relative positional embedding for length extrapolation. Advances in Neural Information Processing Systems, 35:8386–8399. [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Peter+J+Ramadge%2C+and+Alexander+Rudnicky%2E+2022%2E+Kerple%3A+Kernelized+relative+positional+embedding+for+length+extrapolation%2E+Advances+in+Neural+Information+Processing+Systems%2C)
5. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. [Dola: Decoding by contrasting layers improves factuality in large language models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/). In The Twelfth International Conference on Learning Representations.
6. TUNING ON SYNTHETIC DATA. From artificial needles to real haystacks: Im-proving retrieval capabilities in llms by fine-tuning on synthetic data. [find](https://scholar.google.com/scholar?q=TUNING+ON+SYNTHETIC+DATA%2E+From+artificial+needles+to+real+haystacks%3A+Im%2Dproving+retrieval+capabilities+in+llms+by+fine%2Dtuning+on+synthetic+data%2E)
7. Amanda Dsouza, Christopher Glaze, Changho Shin, and Frederic Sala. 2024. Evaluating language model context windows: A" working memory" test and inference-time correction. arXiv preprint arXiv:2407.03651. — [arXiv:2407.03651](https://arxiv.org/abs/2407.03651) [find](https://scholar.google.com/scholar?q=Amanda+Dsouza%2C+Christopher+Glaze%2C+Changho+Shin%2C+and+Frederic+Sala%2E+2024%2E+Evaluating+language+model+context+windows%3A+A%22+working+memory%22+test+and+inference%2Dtime+correction%2E+arXiv+preprint+arXiv%3A2407%2E036)
8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
9. Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. 2024. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models. arXiv preprint arXiv:2408.10151. [find](https://scholar.google.com/scholar?q=Amey+Hengle%2C+Prasoon+Bajpai%2C+Soham+Dan%2C+and+Tanmoy+Chakraborty%2E+2024%2E+Multilingual+needle+in+a+haystack%3A+Investigating+long%2Dcontext+behavior+of+multilingual+large+language+models%2E+arXiv+preprint+arXiv)
10. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024a. [RULER: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) In First Conference on Language Modeling.
11. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024b. Found in the middle: Calibrating positional attention bias improves long context utilization. arXiv preprint arXiv:2406.16008. [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Yung%2DSung+Chuang%2C+Chun%2DLiang+Li%2C+Zifeng+Wang%2C+Long+T+Le%2C+Abhishek+Kumar%2C+James+Glass%2C+Alexander+Ratner%2C+Chen%2DYu+Lee%2C+Ranjay+Krishna%2C+et+al%2E+2024b%2E+Found+in+the+middle%3A+Calibrating+posi)
12. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
13. Taiming Lu, Muhan Gao, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into llm long-context failures: When transformers know but don’t tell. arXiv preprint arXiv:2406.14673. [find](https://scholar.google.com/scholar?q=Taiming+Lu%2C+Muhan+Gao%2C+Kuai+Yu%2C+Adam+Byerly%2C+and+Daniel+Khashabi%2E+2024%2E+Insights+into+llm+long%2Dcontext+failures%3A+When+transformers+know+but+don%E2%80%99t+tell%2E+arXiv+preprint+arXiv%3A2406%2E14673%2E)
14. Alexander Peysakhovich and Adam Lerer. 2023. Attention sorting combats recency bias in long context language models. arXiv preprint arXiv:2310.01427. — [arXiv:2310.01427](https://arxiv.org/abs/2310.01427) [find](https://scholar.google.com/scholar?q=Alexander+Peysakhovich+and+Adam+Lerer%2E+2023%2E+Attention+sorting+combats+recency+bias+in+long+context+language+models%2E+arXiv+preprint+arXiv%3A2310%2E01427%2E+%E2%80%94+%5BarXiv%3A2310%2E01427%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E01)
15. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+International+Conference+on+Learning+Representations%2E)
16. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
17. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. 2024. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+2024%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+Advances+in+Neural+Informat)
18. Proof. For each 2D subspace V j = R 2 , let (1) (2) (q j , q j ) = R θ j (m)(W q x m ) j and similarly for k j . Using the inner product invariance under rotation: Yijiong Yu. 2023. " paraphrasing the original text" makes high accuracy long-context qa. arXiv preprint arXiv:2312.11193. [find](https://scholar.google.com/scholar?q=Proof%2E+For+each+2D+subspace+V+j+%3D+R+2+%2C+let+%281%29+%282%29+%28q+j+%2C+q+j+%29+%3D+R+%CE%B8+j+%28m%29%28W+q+x+m+%29+j+and+similarly+for+k+j+%2E+Using+the+inner+product+invariance+under+rotation%3A+Yijiong+Yu%2E+2023%2E+%22+paraphrasing+the)
19. S j (k) = ⟨R θ j (m)(W q x m ) j , R θ j (n)(W k x n ) j ⟩ = ⟨(W q x m ) j , R θ j (k) ⊤ (W k x n ) j ⟩ Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024a. Infinite bench: Extending long context evaluation beyond 100k to… [find](https://scholar.google.com/scholar?q=S+j+%28k%29+%3D+%E2%9F%A8R+%CE%B8+j+%28m%29%28W+q+x+m+%29+j+%2C+R+%CE%B8+j+%28n%29%28W+k+x+n+%29+j+%E2%9F%A9+%3D+%E2%9F%A8%28W+q+x+m+%29+j+%2C+R+%CE%B8+j+%28k%29+%E2%8A%A4+%28W+k+x+n+%29+j+%E2%9F%A9+Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Sh)
20. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024b. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. arXiv preprint arXiv:2403.04797. — [arXiv:2403.04797](https://arxiv.org/abs/2403.04797) [find](https://scholar.google.com/scholar?q=Zhenyu+Zhang%2C+Runjin+Chen%2C+Shiwei+Liu%2C+Zhewei+Yao%2C+Olatunji+Ruwase%2C+Beidi+Chen%2C+Xiaoxia+Wu%2C+and+Zhangyang+Wang%2E+2024b%2E+Found+in+the+middle%3A+How+language+models+use+long+contexts+better+via+plug%2Dand%2Dpl)

*Updated 31 October 2026*