---
title: "Leave No Document Behind — paper by Wang et al. (2024)"
description: "Most current long-context benchmarks make inputs longer by padding them with unrelated filler text. 40 references, cited by 57 papers here."
authors: ["Minzheng Wang", "Longze Chen", "Fu Cheng", "Shengyi Liao", "Xinghua Zhang", "Bingli Wu", "Haiyang Yu", "Nan Xu", "Lei Zhang", "Run Luo", "Yunshui Li", "Min Yang", "Fei Huang", "Yongbin Li"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.322/"
doi: "10.18653/v1/2024.emnlp-main.322"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

# Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

**Authors:** Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, [Xinghua Zhang](https://md.research.elamur.ai/free/en/authors/xinghua-zhang/), Bingli Wu, [Haiyang Yu](https://md.research.elamur.ai/free/en/authors/haiyang-yu/), [Nan Xu](https://md.research.elamur.ai/free/en/authors/nan-xu/), [Lei Zhang](https://md.research.elamur.ai/free/en/authors/lei-zhang/), Run Luo, [Yunshui Li](https://md.research.elamur.ai/free/en/authors/yunshui-li/), [Min Yang](https://md.research.elamur.ai/free/en/authors/min-yang/), [Fei Huang](https://md.research.elamur.ai/free/en/authors/fei-huang/) and [Yongbin Li](https://md.research.elamur.ai/free/en/authors/yongbin-li/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.322/) · [PDF](https://aclanthology.org/2024.emnlp-main.322.pdf) · [DOI 10.18653/v1/2024.emnlp-main.322](https://doi.org/10.18653/v1/2024.emnlp-main.322) · [OpenAlex](https://openalex.org/W4404783613)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most current long-context benchmarks make inputs longer by padding them with unrelated filler text. In those setups, the evidence for an answer usually sits in one place, so a model can skip most of the input and still answer correctly. Wang et al. address this with Loong, a multi-document QA benchmark where every document in an instance contributes to the answer. Missing any single one makes a correct answer impossible. The authors gathered fresh documents, almost all from 2024, to reduce contamination risk. The documents cover financial reports, legal cases and academic papers in English and Chinese, and every question was newly annotated and checked by domain experts. They tested six strong LLMs, including GPT-4o and Gemini-Pro1.5, and none performed satisfactorily. Models' usable context also proved shorter than their advertised windows. Adding RAG generally lowered scores, because the evidence is spread across documents and retrieval breaks up that information. This makes Loong a realistic and hard-to-shortcut test of genuine long-context understanding.

## Key points
- **Task types:** Loong defines four tasks, each with its own test set: Spotlight Locating (250 instances), Comparison (300), Clustering (641) and Chain of Reasoning (409), with average inputs of roughly 104K–119K tokens.
- **Instance structure:** Each instance contains about 11 documents on average. Inputs are grouped into length sets of 10K–50K, 50K–100K, 100K–200K and >200K, which allows analysis by context length.
- **Effective window size:** Models handle far less context in practice than their claimed windows suggest. The authors argue that models must be trained on sequences longer than the length they are meant to process. Gemini, trained with a 1000K context, was the least sensitive to input length.
- **RAG setup and results:** RAG was tested on GPT-4o and Qwen2-72B-Instruct, using OpenAI and BGE (bge-m3) embeddings, top-k values of 5/10/30/50, and 1024-token chunks.
  - RAG helped only on sparse-evidence tasks such as Spotlight Locating, and only on ultra-long inputs at high top-k.
  - It hurt the stronger long-context model the most.
- **Availability:** Code and data are released on GitHub (MozerWang/Loong and AlibabaResearch/DAMO-ConvAI).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 64 works · 14,113 citations
- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 68 works · 22,753 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 20 works · 13,875 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 32 works · 7,615 citations
- [Shenzhen Institutes of Advanced Technology](https://md.research.elamur.ai/free/en/labs/shenzhen-institutes-of-advanced-technology/) — Research institute, Shenzhen, China · 12 works · 2,431 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 242 works

## Cite

Cite as: Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang and Yongbin Li. 2024. Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2024leave,
  title = {{Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA}},
  author = {Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang and Yongbin Li},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.322},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.322}
}
```

- [← previous Calibrating LLM-Based Evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/)
- [next → LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

40 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). In Proceedings of ACL, pages 14388–14411.
2. AI Anthropic. 2024a. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+2024a%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Claude%2D3+Model+Card%2E)
3. AI Anthropic. 2024b. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+2024b%2E+Claude+3%2E5+sonnet+model+card+addendum%2E+Claude%2D3%2E5+Model+Card%2E)
4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
5. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of ACL, pages 3119–3137. bloc97. 2023. Ntk… [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+2024%2E+LongBench%3A+A+bilingual%2C+multi%2Dtask)
6. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of ICML, pages 2206– 2240. [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+2022%2E+Improving+)
7. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024a. [Benchmarking large language models in retrieval-augmented generation](https://md.research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/). In Proceedings of AAAI, pages 17754–17762.
8. Longze Chen, Ziqiang Liu, Wanwei He, Yinhe Zheng, Hao Sun, Yunshui Li, Run Luo, and Min Yang. 2024b. Long context is not long at all: A prospector of long-dependency data for large language models. In Proceedings of ACL, pages 8222–8234. [find](https://scholar.google.com/scholar?q=Longze+Chen%2C+Ziqiang+Liu%2C+Wanwei+He%2C+Yinhe+Zheng%2C+Hao+Sun%2C+Yunshui+Li%2C+Run+Luo%2C+and+Min+Yang%2E+2024b%2E+Long+context+is+not+long+at+all%3A+A+prospector+of+long%2Ddependency+data+for+large+language+models%2E+In)
9. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+2023%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2E)
10. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2024. [BAMBOO: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). In Proceedings of LREC-COLING, pages 2086–2099.
11. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [Glm: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of ACL, pages 320–335.
12. Shahriar Golchin and Mihai Surdeanu. 2023. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). arXiv preprint arXiv:2308.08493.
13. Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. [LM-infinite: Zero-shot extreme length generalization for large language models](https://md.research.elamur.ai/free/en/papers/han-2024-lm-infinite-zero-shot-extreme/). In Proceedings of NAACL, pages 3991–4008.
14. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. [RULER: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) In Proceedings of COLM.
15. Greg Kamradt. 2023. Needle in a haystack - pressure testing llms. [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack). [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
16. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A "novel" challenge for long-context language models. arXiv preprint arXiv: 2406.16264. [find](https://scholar.google.com/scholar?q=Marzena+Karpinska%2C+Katherine+Thai%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2024%2E+One+thousand+and+one+pairs%3A+A+%22novel%22+challenge+for+long%2Dcontext+language+models%2E+arXiv+preprint+arXiv%3A+2406%2E16264%2E)
17. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024. [LooGLE: Can long-context language models understand long contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) In Proceedings of ACL, pages 16304–16333.
18. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024. [Calibrating LLM-based evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/). In Proceedings of LREC-COLING, pages 2638–2656.
19. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
20. Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint arXiv:2308.10882. — [arXiv:2308.10882](https://arxiv.org/abs/2308.10882) [find](https://scholar.google.com/scholar?q=Arka+Pal%2C+Deep+Karkhanis%2C+Manley+Roberts%2C+Samuel+Dooley%2C+Arvind+Sundararajan%2C+and+Siddartha+Naidu%2E+2023%2E+Giraffe%3A+Adventures+in+expanding+context+lengths+in+llms%2E+arXiv+preprint+arXiv%3A2308%2E10882%2E+%E2%80%94+%5Ba)
21. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2024%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+In+Proceedings+of+ICLR%2E)
22. Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+Smith%2C+and+Mike+Lewis%2E+2022%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+In+Proceedings+of+ICLR%2E)
23. Zexuan Qiu, Jingjing Li, Shijue Huang, Wanjun Zhong, and Irwin King. 2024. Clongeval: A chinese benchmark for evaluating long-context large language models. arXiv preprint arXiv:2403.03514. [find](https://scholar.google.com/scholar?q=Zexuan+Qiu%2C+Jingjing+Li%2C+Shijue+Huang%2C+Wanjun+Zhong%2C+and+Irwin+King%2E+2024%2E+Clongeval%3A+A+chinese+benchmark+for+evaluating+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E03514%2E)
24. Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. [Parallel context windows for large language models](https://md.research.elamur.ai/free/en/papers/ratner-2023-parallel-context-windows-large-language/). In Proceedings of ACL, pages 6383–6402.
25. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
26. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
27. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2024. [REPLUG: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/). In Proceedings of NAACL, pages 8371–8384.
28. Mingyang Song, Mao Zheng, and Xuan Luo. 2024. Counting-stars: A simple, efficient, and reasonable strategy for evaluating long-context large language models. arXiv preprint arXiv:2403.11802. — [arXiv:2403.11802](https://arxiv.org/abs/2403.11802) [find](https://scholar.google.com/scholar?q=Mingyang+Song%2C+Mao+Zheng%2C+and+Xuan+Luo%2E+2024%2E+Counting%2Dstars%3A+A+simple%2C+efficient%2C+and+reasonable+strategy+for+evaluating+long%2Dcontext+large+language+models%2E+arXiv+preprint+arXiv%3A2403%2E11802%2E+%E2%80%94+%5BarXiv%3A)
29. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
30. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2023. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). In Proceedings of ACL, pages 14590– 14604.
31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of NeurIPs, page 30.
32. Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. 2024. Evaluating open-qa evaluation. In Proceedings of NeurIPs, page 36. [find](https://scholar.google.com/scholar?q=Cunxiang+Wang%2C+Sirui+Cheng%2C+Qipeng+Guo%2C+Yuanhao+Yue%2C+Bowen+Ding%2C+Zhikun+Xu%2C+Yidong+Wang%2C+Xiangkun+Hu%2C+Zheng+Zhang%2C+and+Yue+Zhang%2E+2024%2E+Evaluating+open%2Dqa+evaluation%2E+In+Proceedings+of+NeurIPs%2C+page+3)
33. Kevin Wu, Eric Wu, and James Zou. 2024. How faithful are rag models? quantifying the tug-of-war between rag and llms’ internal prior. arXiv preprint arXiv:2404.10198. [find](https://scholar.google.com/scholar?q=Kevin+Wu%2C+Eric+Wu%2C+and+James+Zou%2E+2024%2E+How+faithful+are+rag+models%3F+quantifying+the+tug%2Dof%2Dwar+between+rag+and+llms%E2%80%99+internal+prior%2E+arXiv+preprint+arXiv%3A2404%2E10198%2E)
34. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. [find](https://scholar.google.com/scholar?q=Guangxuan+Xiao%2C+Yuandong+Tian%2C+Beidi+Chen%2C+Song+Han%2C+and+Mike+Lewis%2E+2023%2E+Efficient+streaming+language+models+with+attention+sinks%2E+arXiv+preprint+arXiv%3A2309%2E17453%2E)
35. [Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.…](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
36. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024. [Retrieval meets long context large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/). In Proceedings of ICLR.
37. Lei Zhang, Yunshui Li, Ziqiang Liu, Jiaxi Yang, Junhao Liu, Longze Chen, Run Luo, and Min Yang. 2024a. Marathon: A race through the realm of long context with large language models. In Proceedings of ACL, pages 5201–5217. [find](https://scholar.google.com/scholar?q=Lei+Zhang%2C+Yunshui+Li%2C+Ziqiang+Liu%2C+Jiaxi+Yang%2C+Junhao+Liu%2C+Longze+Chen%2C+Run+Luo%2C+and+Min+Yang%2E+2024a%2E+Marathon%3A+A+race+through+the+realm+of+long+context+with+large+language+models%2E+In+Proceedings+of+)
38. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. [Wider and deeper llm networks are fairer llm evaluators](https://md.research.elamur.ai/free/en/papers/zhang-2023-wider-deeper-llm-networks-fairer/). arXiv preprint arXiv:2308.01862.
39. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024b. ∞Bench: Extending long context evaluation beyond 100K tokens. In Proceedings of ACL, pages 15262– 15277. [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Hao%2C+Xu+Han%2C+Zhen+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2024b%2E+%E2%88%9EBench%3A+Extending+long+context+evaluation+beyond+100K+toke)
40. Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. PoSE: Efficient context window extension of LLMs via positional skip-wise training. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Dawei+Zhu%2C+Nan+Yang%2C+Liang+Wang%2C+Yifan+Song%2C+Wenhao+Wu%2C+Furu+Wei%2C+and+Sujian+Li%2E+2024%2E+PoSE%3A+Efficient+context+window+extension+of+LLMs+via+positional+skip%2Dwise+training%2E+In+Proceedings+of+ICLR%2E)

*Updated 24 October 2026*