---
title: "LADM — paper by Chen et al. (2025) — summary and ideas"
description: "They then propose LADM, a filtering framework. 60 references, cited by 9 papers here. Among them: 3."
authors: ["Jianghao Chen", "Junhong Wu", "Yangyifan Xu", "Jiajun Zhang"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.154/"
doi: "10.18653/v1/2025.acl-long.154"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2025-ladm-long-context-training-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2025-ladm-long-context-training-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/) / LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

# LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs

**Authors:** [Jianghao Chen](https://md.research.elamur.ai/free/en/authors/jianghao-chen/), [Junhong Wu](https://md.research.elamur.ai/free/en/authors/junhong-wu/), [Yangyifan Xu](https://md.research.elamur.ai/free/en/authors/yangyifan-xu/) and [Jiajun Zhang](https://md.research.elamur.ai/free/en/authors/jiajun-zhang/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.154/) · [PDF](https://aclanthology.org/2025.acl-long.154.pdf) · [DOI 10.18653/v1/2025.acl-long.154](https://doi.org/10.18653/v1/2025.acl-long.154) · [OpenAlex](https://openalex.org/W4412886816)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/chen-2025-ladm-long-context-training-data/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Extending LLMs to long inputs usually relies on continual pre-training with long documents. However, there has been no reliable way to judge whether a long sample actually contains meaningful long-range connections or is just loosely joined text. The authors first show that this matters. Llama2-7B trained on 32K sequences built from concatenated shorter pieces does worse on Needle-in-the-Haystack retrieval than the same model trained on genuinely long samples. They then propose LADM, a filtering framework. It runs candidate texts through a small long-context model and reads its attention patterns to estimate how much later parts of a text rely on earlier ones. The scores are computed in three stages: span pairs, then each span, then the whole sample. The highest-scoring samples are kept for continual training. Using about 1B training tokens from a multi-domain corpus, LADM beats competing selection methods on LongBench and matches random sampling with half the data. This makes long-context training cheaper and more targeted.

## Key points
- **Scorer model and scores:**
  - The "Long Attention Calculator" is TinyLlama-1.1B-v1.1 trained on random 32K-token sequences.
  - It produces a Pairwise Focus Score (PFS) for each pair of spans, based on accumulated attention.
  - These combine into an Aggregated Focus Score (AFS) per span.
  - The AFS values are then combined into a sample-level Contextual Dependency Score (CDS), weighted by attention spread (standard deviation) and by distance between spans.
- **Main results:**
  - LADM gives an average gain of 2.16% on LongBench across four models of different sizes and architectures.
  - In the Llama2-7B ablation, removing the std weights lowers the LongBench average from 37.83 to 36.88. Removing the length weights lowers it to 36.49.
- **Efficiency:**
  - With span settings d_AFS = d_CDS = 4, scoring costs about 2.46 s per sample, similar to the ProLong baseline.
  - CDS rankings stay consistent across hyper-parameter settings (Pearson correlation > 0.7).
- **Domain analysis:**
  - Within the Pile, PhilPapers, ArXiv, PubMed Central, FreeLaw and PG-19 receive high CDS.
  - Wikipedia and Ubuntu IRC score low.
  - ProLong, by contrast, rates PG-19 lowest.
- **Limitations and code:**
  - Experiments cover only 32K context and models up to 13B parameters.
  - Code is available at github.com/ZNLP/LADM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 70 works · 14,251 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 147 works · 8,480 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Jianghao Chen, Junhong Wu, Yangyifan Xu and Jiajun Zhang. 2025. LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2025ladm,
  title = {{LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs}},
  author = {Jianghao Chen, Junhong Wu, Yangyifan Xu and Jiajun Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.154},
  url = {https://doi.org/10.18653/v1/2025.acl-long.154}
}
```

- [← previous Improving Multilingual Retrieval-Augmented Language Models through Dialectic Reasoning Argumentations](https://md.research.elamur.ai/free/en/papers/ranaldi-2025-improving-multilingual-retrieval-augmented-language/)
- [next → LaMP-QA: A Benchmark for Personalized Long-form Question Answering](https://md.research.elamur.ai/free/en/papers/salemi-2025-lamp-qa-benchmark-personalized-long/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/7/)

## References

60 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024a. Training-free long-context scaling of large language models. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+2024a%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+In+Forty%2Dfirst+International+Conference+o)
3. Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024b. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811. [find](https://scholar.google.com/scholar?q=Shengnan+An%2C+Zexiong+Ma%2C+Zeqi+Lin%2C+Nanning+Zheng%2C+and+Jian%2DGuang+Lou%2E+2024b%2E+Make+your+llm+fully+utilize+the+context%2E+arXiv+preprint+arXiv%3A2404%2E16811%2E)
4. Zhangir Azerbayev, Edward Ayers, and Bartosz Piotrowski. 2022. Proof-pile. [find](https://scholar.google.com/scholar?q=Zhangir+Azerbayev%2C+Edward+Ayers%2C+and+Bartosz+Piotrowski%2E+2022%2E+Proof%2Dpile%2E)
5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
6. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. [Longwriter: Unleashing 10,000+ word generation from long context llms](https://md.research.elamur.ai/free/en/papers/bai-2024-longwriter-unleashing-10-000/). arXiv preprint arXiv:2408.07055. bloc97. 2023. Ntk-aware scaled rope allows llama models to have extended (8k+) context…
7. Guanzheng Chen, Xin Li, Michael Shieh, and Lidong Bing. 2025a. LongPO: Long context self-evolution of large language models through short-to-long preference optimization. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guanzheng+Chen%2C+Xin+Li%2C+Michael+Shieh%2C+and+Lidong+Bing%2E+2025a%2E+LongPO%3A+Long+context+self%2Devolution+of+large+language+models+through+short%2Dto%2Dlong+preference+optimization%2E+In+The+Thirteenth+Internation)
8. Jianghao Chen, Zhenlin Wei, Zhenjiang Ren, Ziyong Li, and Jiajun Zhang. 2025b. LrΘ2 bench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems. arXiv preprint arXiv:2502.17848. — [arXiv:2502.17848](https://arxiv.org/abs/2502.17848) [find](https://scholar.google.com/scholar?q=Jianghao+Chen%2C+Zhenlin+Wei%2C+Zhenjiang+Ren%2C+Ziyong+Li%2C+and+Jiajun+Zhang%2E+2025b%2E+Lr%CE%982+bench%3A+Evaluating+long%2Dchain+reflective+reasoning+capabilities+of+large+language+models+via+constraint+satisfaction+)
9. Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. 2024a. Long context is not long at all: A prospector of long-dependency data for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Longze+Chen%2C+Ziqiang+Liu%2C+Wanwei+He%2C+Yinhe+Zheng%2C+Hao+Sun%2C+Yunshui+Li%2C+Run+Luo%2C+and+Min+Yang%2E+2024a%2E+Long+context+is+not+long+at+all%3A+A+prospector+of+long%2Ddependency+data+for+large+language+models%2E+In)
10. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2E)
11. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024b. [LongloRA: Efficient fine-tuning of long-context large language models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/). In The Twelfth International Conference on Learning Representations.
12. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
13. Tri Dao. 2024. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In International Conference on Learning Representations (ICLR).
14. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang…
15. Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. Fewer truncations improve language modeling. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Hantian+Ding%2C+Zijian+Wang%2C+Giovanni+Paolini%2C+Varun+Kumar%2C+Anoop+Deoras%2C+Dan+Roth%2C+and+Stefano+Soatto%2E+2024%2E+Fewer+truncations+improve+language+modeling%2E+In+Forty%2Dfirst+International+Conference+on+Mach)
16. Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. [Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084.
17. Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2024. What is wrong with perplexity for long-context language modeling? arXiv preprint arXiv:2410.23771. — [arXiv:2410.23771](https://arxiv.org/abs/2410.23771) [find](https://scholar.google.com/scholar?q=Lizhe+Fang%2C+Yifei+Wang%2C+Zhaoyang+Liu%2C+Chenheng+Zhang%2C+Stefanie+Jegelka%2C+Jinyang+Gao%2C+Bolin+Ding%2C+and+Yisen+Wang%2E+2024%2E+What+is+wrong+with+perplexity+for+long%2Dcontext+language+modeling%3F+arXiv+preprint+)
18. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024b. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Aviya+Skowron%2C+Lintang+Sutawika%2C+Eric+Tang%2C+Anish+Thite%2C+Ben+Wang%2C+Kevin+Wang%2C+and+Andy+Zou%2E+2024b%2E+A+framework+for+few%2Dshot+language+model+evaluation%2E)
19. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
20. Boyu Guan, Yining Zhang, Yang Zhao, and Chengqing Zong. 2025. [TriFine: A large-scale dataset of vision-audio-subtitle for tri-modal machine translation and benchmark with fine-grained annotated tags](https://md.research.elamur.ai/free/en/papers/guan-2025-trifine-large-scale-dataset-vision/). In Proceedings of the 31st International Conference on Computational Linguistics, Abu Dhabi, UAE. As…
21. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. [Lm-infinite: Zero-shot extreme length generalization for large language models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T…
22. Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Proceedings of the 62nd A… [find](https://scholar.google.com/scholar?q=Junqing+He%2C+Kunhao+Pan%2C+Xiaoqun+Dong%2C+Zhuoyang+Song%2C+LiuYiBo+LiuYiBo%2C+Qianguosun+Qianguosun%2C+Yuxin+Liang%2C+Hao+Wang%2C+Enming+Zhang%2C+and+Jiaxing+Zhang%2E+2024%2E+Never+lost+in+the+middle%3A+Mastering+long%2Dcont)
23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
24. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. [Data engineering for scaling language models to 128k context](https://md.research.elamur.ai/free/en/papers/fu-2024-data-engineering-scaling-language-models/). In Forty-first International Conference on Machine Learning.
25. Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al. 2024. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computation… [find](https://scholar.google.com/scholar?q=Cheng%2DYu+Hsieh%2C+Yung%2DSung+Chuang%2C+Chun%2DLiang+Li%2C+Zifeng+Wang%2C+Long+Le%2C+Abhishek+Kumar%2C+James+Glass%2C+Alexander+Ratner%2C+Chen%2DYu+Lee%2C+Ranjay+Krishna%2C+et+al%2E+2024%2E+Found+in+the+middle%3A+Calibrating+positio)
26. Chaochen Gao, Xing Wu, Qi Fu, and Songlin Hu. 2024a. Quest: Query-centric data synthesis approach for long-context scaling of large language model. arXiv preprint arXiv:2405.19846. — [arXiv:2405.19846](https://arxiv.org/abs/2405.19846) [find](https://scholar.google.com/scholar?q=Chaochen+Gao%2C+Xing+Wu%2C+Qi+Fu%2C+and+Songlin+Hu%2E+2024a%2E+Quest%3A+Query%2Dcentric+data+synthesis+approach+for+long%2Dcontext+scaling+of+large+language+model%2E+arXiv+preprint+arXiv%3A2405%2E19846%2E+%E2%80%94+%5BarXiv%3A2405%2E19846)
27. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). Preprint, arXiv:2106.09685.
28. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
29. Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. 2024. Can perplexity reflect large language model’s ability in long text understanding? In The Second Tiny Papers Track at ICLR 2024. [find](https://scholar.google.com/scholar?q=Yutong+Hu%2C+Quzhe+Huang%2C+Mingxu+Tao%2C+Chen+Zhang%2C+and+Yansong+Feng%2E+2024%2E+Can+perplexity+reflect+large+language+model%E2%80%99s+ability+in+long+text+understanding%3F+In+The+Second+Tiny+Papers+Track+at+ICLR+2024%2E)
30. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Luyang Huang, Shuyang Cao, Nikolaus Parulian, H… — [Efficient Attentions for Long Document Summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/)
31. Pu Jian, Donglei Yu, and Jiajun Zhang. 2024. Large language models know what is key visual entity: An llm-assisted multimodal retrieval for vqa. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10939–10956. [find](https://scholar.google.com/scholar?q=Pu+Jian%2C+Donglei+Yu%2C+and+Jiajun+Zhang%2E+2024%2E+Large+language+models+know+what+is+key+visual+entity%3A+An+llm%2Dassisted+multimodal+retrieval+for+vqa%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+Meth)
32. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
33. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. LLM maybe longLM: Selfextend LLM context window without tuning. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Hongye+Jin%2C+Xiaotian+Han%2C+Jingfeng+Yang%2C+Zhimeng+Jiang%2C+Zirui+Liu%2C+Chia%2DYuan+Chang%2C+Huiyuan+Chen%2C+and+Xia+Hu%2E+2024%2E+LLM+maybe+longLM%3A+Selfextend+LLM+context+window+without+tuning%2E+In+Forty%2Dfirst+Inter)
34. International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 16. IEEE. [find](https://scholar.google.com/scholar?q=International+Conference+for+High+Performance+Computing%2C+Networking%2C+Storage+and+Analysis%2C+pages+1%E2%80%93+16%2E+IEEE%2E)
35. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
36. Konrad Staniszewski, Szymon Tworkowski, Sebastian Jaszczur, Henryk Michalewski, Łukasz Kuciński, and Piotr Miłoś. 2023. Structured packing in llm training improves long context utilization. arXiv preprint arXiv:2312.17296. — [arXiv:2312.17296](https://arxiv.org/abs/2312.17296) [find](https://scholar.google.com/scholar?q=Konrad+Staniszewski%2C+Szymon+Tworkowski%2C+Sebastian+Jaszczur%2C+Henryk+Michalewski%2C+%C5%81ukasz+Kucin%CC%81ski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+2023%2E+Structured+packing+in+llm+training+improves+long+context+utilization%2E+arXiv+pr)
37. Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. [https://github.com/gkamradt/](https://github.com/gkamradt/) LLMTest_NeedleInAHaystack. [↗](https://github.com/gkamradt/)
38. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
39. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328.
40. Wei Sun, Qianlong Du, Fuwei Cui, and Jiajun Zhang. 2025a. [An efficient and precise training data construction framework for process-supervised reward model in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/sun-2025-efficient-precise-training-data-construction/). arXiv preprint arXiv:2503.02382.
41. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252.
42. Wei Sun, Wen Yang, Pu Jian, Qianlong Du, Fuwei Cui, Shuo Ren, and Jiajun Zhang. 2025b. Ktae: A model-free algorithm to key-tokens advantage estimation in mathematical reasoning. arXiv preprint arXiv:2505.16826. — [arXiv:2505.16826](https://arxiv.org/abs/2505.16826) [find](https://scholar.google.com/scholar?q=Wei+Sun%2C+Wen+Yang%2C+Pu+Jian%2C+Qianlong+Du%2C+Fuwei+Cui%2C+Shuo+Ren%2C+and+Jiajun+Zhang%2E+2025b%2E+Ktae%3A+A+model%2Dfree+algorithm+to+key%2Dtokens+advantage+estimation+in+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2)
43. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
44. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint+arXiv%3A1711%2E05101%2E)
45. Sarthak Mittal, Sharath Chandra Raparthy, Irina Rish, Yoshua Bengio, and Guillaume Lajoie. 2022. Compositional attention: Disentangling search and retrieval. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sarthak+Mittal%2C+Sharath+Chandra+Raparthy%2C+Irina+Rish%2C+Yoshua+Bengio%2C+and+Guillaume+Lajoie%2E+2022%2E+Compositional+attention%3A+Disentangling+search+and+retrieval%2E+In+International+Conference+on+Learning+Re)
46. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2024%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+In+The+Twelfth+International+Conference+on+Learning+Representat)
47. Chau Pham, Simeng Sun, and Mohit Iyyer. 2024. [Suri: Multi-constraint instruction following in long-form text generation](https://md.research.elamur.ai/free/en/papers/pham-2024-suri-multi-constraint-instruction-following/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1722–1753.
48. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. [Zero: Memory optimizations toward training trillion parameter models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). In SC20: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bh…
49. Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. 2024. Longvlm: Efficient long video understanding via large language models. In European Conference on Computer Vision, pages 453–470. Springer. [find](https://scholar.google.com/scholar?q=Yuetian+Weng%2C+Mingfei+Han%2C+Haoyu+He%2C+Xiaojun+Chang%2C+and+Bohan+Zhuang%2E+2024%2E+Longvlm%3A+Efficient+long+video+understanding+via+large+language+models%2E+In+European+Conference+on+Computer+Vision%2C+pages+453%E2%80%93)
50. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
51. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2024. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). arXiv preprint arXiv:2404.15574.
52. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2024%2E+Efficient+streaming+language+models+with+attention+sinks%2E+In+The+Twelfth+International+Conference+on+Learning+Representations)
53. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039.
54. Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. 2024. From artificial needles to real haystacks: Improving retrieval capabilities in llms by finetuning on synthetic data. arXiv preprint arXiv:2406.19292. — [arXiv:2406.19292](https://arxiv.org/abs/2406.19292) [find](https://scholar.google.com/scholar?q=Zheyang+Xiong%2C+Vasilis+Papageorgiou%2C+Kangwook+Lee%2C+and+Dimitris+Papailiopoulos%2E+2024%2E+From+artificial+needles+to+real+haystacks%3A+Improving+retrieval+capabilities+in+llms+by+finetuning+on+synthetic+dat)
55. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. [Retrieval meets long context large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/). arXiv preprint arXiv:2310.03025.
56. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. [Demystifying long chain-of-thought reasoning in llms](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/). arXiv preprint arXiv:2502.03373.
57. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
58. Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. 2024. A simple llm framework for long-range video question-answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21715–21737. [find](https://scholar.google.com/scholar?q=Ce+Zhang%2C+Taixi+Lu%2C+Md+Mohaiminul+Islam%2C+Ziyang+Wang%2C+Shoubin+Yu%2C+Mohit+Bansal%2C+and+Gedas+Bertasius%2E+2024%2E+A+simple+llm+framework+for+long%2Drange+video+question%2Danswering%2E+In+Proceedings+of+the+2024+Co)
59. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021. [Qmsum: A new benchmark for query-based multi-domain meeting summarization](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/). In Proceedings of the 2021 Conference of the North American Chapter of the Association fo…
60. Shiyi Zhu, Jing Ye, Wei Jiang, Siqiao Xue, Qi Zhang, Yifan Wu, and Jianguo Li. 2024. Coca: Fusing position embedding with collinear constrained attention in transformers for long context window extending. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol… [find](https://scholar.google.com/scholar?q=Shiyi+Zhu%2C+Jing+Ye%2C+Wei+Jiang%2C+Siqiao+Xue%2C+Qi+Zhang%2C+Yifan+Wu%2C+and+Jianguo+Li%2E+2024%2E+Coca%3A+Fusing+position+embedding+with+collinear+constrained+attention+in+transformers+for+long+context+window+extend)

*Updated 30 October 2026*