---
title: "LooGLE: Can Long-Context Language Models — Li 2023"
description: "Existing long-context benchmarks tend to use texts much shorter than current LLM context windows."
authors: ["Jiaqi Li", "Mengmeng Wang", "Zilong Zheng", "Muhan Zhang"]
year: 2023
venue: "arXiv:2311.04939"
source: "https://arxiv.org/abs/2311.04939"
doi: "10.48550/arxiv.2311.04939"
arxiv: "2311.04939"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / LooGLE: Can Long-Context Language Models Understand Long Contexts?

# LooGLE: Can Long-Context Language Models Understand Long Contexts?

**Authors:** Jiaqi Li, Mengmeng Wang, [Zilong Zheng](https://md.research.elamur.ai/free/en/authors/zilong-zheng/), Muhan Zhang · **Year:** 2023 · **Venue:** arXiv:2311.04939 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.04939) · [PDF](https://arxiv.org/pdf/2311.04939) · [DOI 10.48550/arxiv.2311.04939](https://doi.org/10.48550/arxiv.2311.04939) · [arXiv 2311.04939](https://arxiv.org/abs/2311.04939) · [OpenAlex](https://openalex.org/W4402671258)

**Citations:** 54 citations (openalex-snapshot · 2026-10-14) · [108](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Existing long-context benchmarks tend to use texts much shorter than current LLM context windows. They also recycle older documents that models may have seen during pre-training, and they mostly test whether a model can find an answer in a single passage. LooGLE addresses these gaps with 776 documents published after 2022 (arXiv papers, Wikipedia articles, movie and TV scripts), each over 10k words and averaging about 19.3k words. Its 6,448 questions cover short dependency tasks (QA, cloze) and long dependency tasks that need evidence scattered across a whole document: summarization plus four QA types (multiple information retrieval, timeline reordering, computation, and comprehension/reasoning). Annotators wrote 1,101 long dependency QA pairs by hand and cross-checked them. Across eight LLMs, commercial systems beat open-source ones. All models did well on short dependency tasks but dropped sharply on long dependency ones. Prompting strategies, retrieval and context-window extension did not close this gap. LooGLE therefore offers a fresh, leakage-resistant test of real long-document comprehension.

## Key points
- Task counts by source: arXiv papers (Jan 2022–Apr 2023, 516 docs, avg 16,988 words) supply 516 summarization items. Wikipedia (avg 17,604 words) supplies 1,951 short QA and 459 long QA. Scripts (avg 28,483 words) supply 2,880 cloze and 642 long QA items.
- LooGLE's average length is 19,367 words, versus 10,392 for ZeroSCROLLS, 8,120 for LongBench and 8,008 for L-Eval. Among these four, only LooGLE combines manually labeled long dependency tasks with timeline-reorder and computation questions.
- In-context learning and chain-of-thought prompting gave only small gains. Retrieval helped short QA considerably, while architecture or positional-encoding methods for extending context did little for long-context understanding.
- In the qualitative examples, LLaMA2-7B-32K often produced empty, incoherent or garbled answers.
- Books were excluded for licensing reasons. The collection pipeline is open, allowing yearly refreshes, and the evaluation code is at github.com/bigai-nlco/LooGLE.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

108 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 102 not yet.

- [LongAlign: A Recipe for Long Context Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://md.research.elamur.ai/free/en/papers/bai-2024-longbench-v2-deeper-understanding/)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://md.research.elamur.ai/free/en/papers/levy-2024-same-task-more-tokens-impact/)
- [Long-context LLMs Struggle with Long In-context Learning](https://md.research.elamur.ai/free/en/papers/li-2024-long-context-llms-struggle-long/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023loogle,
  title = {{LooGLE: Can Long-Context Language Models Understand Long Contexts?}},
  author = {Li, Jiaqi and Wang, Mengmeng and Zheng, Zilong and Zhang, Muhan},
  year = {2023},
  doi = {10.48550/arxiv.2311.04939},
  eprint = {2311.04939},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.04939}
}
```

- [← previous LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/)
- [next → Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-looking-needle-haystack-comprehensive-study/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

59 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023. 2, 3 [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Shansan+Gong%2C+Ming+Zhong%2C+Mukai+Li%2C+Jun+Zhang%2C+Lingpeng+Kong%2C+and+Xipeng+Qiu%2E+L%2Deval%3A+Instituting+standardized+evaluation+for+long+context+language+models%2C+2023%2E+2%2C+3) [our card](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/)
2. Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. Extractive opinion summarization in quantized transformer spaces, 2020. 2 [find](https://scholar.google.com/scholar?q=Stefanos+Angelidis%2C+Reinald+Kim+Amplayo%2C+Yoshihiko+Suhara%2C+Xiaolan+Wang%2C+and+Mirella+Lapata%2E+Extractive+opinion+summarization+in+quantized+transformer+spaces%2C+2020%2E+2)
3. Arian Askari, Suzan Verberne, Amin Abolghasemi, Wessel Kraaij, and Gabriella Pasi. Retrieval for extremely long queries and documents with rprs: a highly efficient and effective transformer-based re-ranker, 2023. 7 [find](https://scholar.google.com/scholar?q=Arian+Askari%2C+Suzan+Verberne%2C+Amin+Abolghasemi%2C+Wessel+Kraaij%2C+and+Gabriella+Pasi%2E+Retrieval+for+extremely+long+queries+and+documents+with+rprs%3A+a+highly+efficient+and+effective+transformer%2Dbased+re%2Dr)
4. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2023. 2, 3 [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Hongchang+Lyu%2C+Jiankai+Tang%2C+Zhidian+Huang%2C+Zhengxiao+Du%2C+Xiao+Liu%2C+Aohan+Zeng%2C+Lei+Hou%2C+Yuxiao+Dong%2C+Jie+Tang%2C+and+Juanzi+Li%2E+Longbench%3A+A+bilingual%2C+multitask+benchm) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/)
5. Arkadii Bessonov, Alexey Staroverov, Huzhenyu Zhang, Alexey K. Kovalev, Dmitry Yudin, and Aleksandr I. Panov. Recurrent memory decision transformer, 2023. 3 [find](https://scholar.google.com/scholar?q=Arkadii+Bessonov%2C+Alexey+Staroverov%2C+Huzhenyu+Zhang%2C+Alexey+K%2E+Kovalev%2C+Dmitry+Yudin%2C+and+Aleksandr+I%2E+Panov%2E+Recurrent+memory+decision+transformer%2C+2023%2E+3)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1 [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
7. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Recurrent memory transformer, 2022. 3 [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yuri+Kuratov%2C+and+Mikhail+S%2E+Burtsev%2E+Recurrent+memory+transformer%2C+2022%2E+3)
8. Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Scaling transformer to 1m tokens and beyond with rmt, 2023. 2 [find](https://scholar.google.com/scholar?q=Aydar+Bulatov%2C+Yuri+Kuratov%2C+and+Mikhail+S%2E+Burtsev%2E+Scaling+transformer+to+1m+tokens+and+beyond+with+rmt%2C+2023%2E+2)
9. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023a. 7 [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2C+2023a%2E+7)
10. Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. How robust is gpt-3.5 to predecessors? a comprehensive study on language understanding tasks, 2023b. 7 [find](https://scholar.google.com/scholar?q=Xuanting+Chen%2C+Junjie+Ye%2C+Can+Zu%2C+Nuo+Xu%2C+Rui+Zheng%2C+Minlong+Peng%2C+Jie+Zhou%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+How+robust+is+gpt%2D3%2E5+to+predecessors%3F+a+comprehensive+study+on+language+understandi)
11. Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models, 2023c. 3 [find](https://scholar.google.com/scholar?q=Yukang+Chen%2C+Shengju+Qian%2C+Haotian+Tang%2C+Xin+Lai%2C+Zhijian+Liu%2C+Song+Han%2C+and+Jiaya+Jia%2E+Longlora%3A+Efficient+fine%2Dtuning+of+long%2Dcontext+large+language+models%2C+2023c%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/chen-2023-longlora-efficient-fine-tuning-long/)
12. Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, and Peter J. Ramadge. Dissecting transformer length extrapolation via the lens of receptive field analysis, 2023. 2 [find](https://scholar.google.com/scholar?q=Ta%2DChung+Chi%2C+Ting%2DHan+Fan%2C+Alexander+I%2E+Rudnicky%2C+and+Peter+J%2E+Ramadge%2E+Dissecting+transformer+length+extrapolation+via+the+lens+of+receptive+field+analysis%2C+2023%2E+2)
13. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents, 2018. 2 [find](https://scholar.google.com/scholar?q=Arman+Cohan%2C+Franck+Dernoncourt%2C+Doo+Soon+Kim%2C+Trung+Bui%2C+Seokhwan+Kim%2C+Walter+Chang%2C+and+Nazli+Goharian%2E+A+discourse%2Daware+attention+model+for+abstractive+summarization+of+long+documents%2C+2018%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/)
14. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. 2, 7 [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+Re%CC%81%2E+Flashattention%3A+Fast+and+memory%2Defficient+exact+attention+with+io%2Dawareness%2C+2022%2E+2%2C+7)
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. 6 [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2C+2019%2E+6)
16. Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023. 2 [find](https://scholar.google.com/scholar?q=Jiayu+Ding%2C+Shuming+Ma%2C+Li+Dong%2C+Xingxing+Zhang%2C+Shaohan+Huang%2C+Wenhui+Wang%2C+Nanning+Zheng%2C+and+Furu+Wei%2E+Longnet%3A+Scaling+transformers+to+1%2C000%2C000%2C000+tokens%2C+2023%2E+2)
17. Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey on long text modeling with transformers, 2023. 3 [find](https://scholar.google.com/scholar?q=Zican+Dong%2C+Tianyi+Tang%2C+Lunyi+Li%2C+and+Wayne+Xin+Zhao%2E+A+survey+on+long+text+modeling+with+transformers%2C+2023%2E+3)
18. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320–335, 2022.… [find](https://scholar.google.com/scholar?q=Zhengxiao+Du%2C+Yujie+Qian%2C+Xiao+Liu%2C+Ming+Ding%2C+Jiezhong+Qiu%2C+Zhilin+Yang%2C+and+Jie+Tang%2E+Glm%3A+General+language+model+pretraining+with+autoregressive+blank+infilling%2E+In+Proceedings+of+the+60th+Annual+M) [our card](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/)
19. Matthias Engelbach, Dennis Klau, Felix Scheerer, Jens Drawehn, and Maximilien Kintz. Fine-tuning and aligning question answering models for complex information extraction tasks, 2023. 8 [find](https://scholar.google.com/scholar?q=Matthias+Engelbach%2C+Dennis+Klau%2C+Felix+Scheerer%2C+Jens+Drawehn%2C+and+Maximilien+Kintz%2E+Fine%2Dtuning+and+aligning+question+answering+models+for+complex+information+extraction+tasks%2C+2023%2E+8)
20. Alexios Gidiotis and Grigorios Tsoumakas. A divide-and-conquer approach to the summarization of long documents, 2020. 3 [find](https://scholar.google.com/scholar?q=Alexios+Gidiotis+and+Grigorios+Tsoumakas%2E+A+divide%2Dand%2Dconquer+approach+to+the+summarization+of+long+documents%2C+2020%2E+3)
21. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization, 2021. 2 [find](https://scholar.google.com/scholar?q=Luyang+Huang%2C+Shuyang+Cao%2C+Nikolaus+Parulian%2C+Heng+Ji%2C+and+Lu+Wang%2E+Efficient+attentions+for+long+document+summarization%2C+2021%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/)
22. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022. 3 [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Patrick+Lewis%2C+Maria+Lomeli%2C+Lucas+Hosseini%2C+Fabio+Petroni%2C+Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Armand+Joulin%2C+Sebastian+Riedel%2C+and+Edouard+Grave%2E+Atlas%3A+Few%2Dshot+learning+with+retrieval+a)
23. NamHyeok Kim and Chanjun Park. Inter-annotator agreement in the wild: Uncovering its emerging roles and considerations in real-world scenarios, 2023. 6 [find](https://scholar.google.com/scholar?q=NamHyeok+Kim+and+Chanjun+Park%2E+Inter%2Dannotator+agreement+in+the+wild%3A+Uncovering+its+emerging+roles+and+considerations+in+real%2Dworld+scenarios%2C+2023%2E+6)
24. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. 10 [find](https://scholar.google.com/scholar?q=Takeshi+Kojima%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo%2C+and+Yusuke+Iwasawa%2E+Large+language+models+are+zero%2Dshot+reasoners%2C+2023%2E+10) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
25. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023. 11 [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Kellie+Lu%2C+Thomas+Mesnard%2C+Colton+Bishop%2C+Victor+Carbune%2C+and+Abhinav+Rastogi%2E+Rlaif%3A+Scaling+reinforcement+learning+from+human+feedback+with+ai+feedback%2C)
26. Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers, 2023a. 3 [find](https://scholar.google.com/scholar?q=Shanda+Li%2C+Chong+You%2C+Guru+Guruganesh%2C+Joshua+Ainslie%2C+Santiago+Ontanon%2C+Manzil+Zaheer%2C+Sumit+Sanghai%2C+Yiming+Yang%2C+Sanjiv+Kumar%2C+and+Srinadh+Bhojanapalli%2E+Functional+interpolation+for+relative+positi)
27. Yucheng Li. Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering, 2023. 3 [find](https://scholar.google.com/scholar?q=Yucheng+Li%2E+Unlocking+context+constraints+of+llms%3A+Enhancing+context+efficiency+of+llms+with+self%2Dinformation%2Dbased+content+filtering%2C+2023%2E+3)
28. Zihao Li, Samuel Belkadi, Nicolo Micheletti, Lifeng Han, Matthew Shardlow, and Goran Nenadic. Large language models and control mechanisms improve text readability of biomedical abstracts, 2023b. 7 [find](https://scholar.google.com/scholar?q=Zihao+Li%2C+Samuel+Belkadi%2C+Nicolo+Micheletti%2C+Lifeng+Han%2C+Matthew+Shardlow%2C+and+Goran+Nenadic%2E+Large+language+models+and+control+mechanisms+improve+text+readability+of+biomedical+abstracts%2C+2023b%2E+7)
29. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023a. 7 [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Kevin+Lin%2C+John+Hewitt%2C+Ashwin+Paranjape%2C+Michele+Bevilacqua%2C+Fabio+Petroni%2C+and+Percy+Liang%2E+Lost+in+the+middle%3A+How+language+models+use+long+contexts%2C+2023a%2E+7) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
30. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. Calibrating llm-based evaluator, 2023b. 8 [find](https://scholar.google.com/scholar?q=Yuxuan+Liu%2C+Tianchi+Yang%2C+Shaohan+Huang%2C+Zihan+Zhang%2C+Haizhen+Huang%2C+Furu+Wei%2C+Weiwei+Deng%2C+Feng+Sun%2C+and+Qi+Zhang%2E+Calibrating+llm%2Dbased+evaluator%2C+2023b%2E+8)
31. Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. Is sparse attention more interpretable?, 2021. 3 [find](https://scholar.google.com/scholar?q=Clara+Meister%2C+Stefan+Lazov%2C+Isabelle+Augenstein%2C+and+Ryan+Cotterell%2E+Is+sparse+attention+more+interpretable%3F%2C+2021%2E+3)
32. Usmi Mukherjee and Mohammad Masudur Rahman. Employing deep learning and structured information retrieval to answer clarification questions on bug reports, 2023. 7 [find](https://scholar.google.com/scholar?q=Usmi+Mukherjee+and+Mohammad+Masudur+Rahman%2E+Employing+deep+learning+and+structured+information+retrieval+to+answer+clarification+questions+on+bug+reports%2C+2023%2E+7)
33. OpenAI. Gpt-4 technical report, 2023. 1 [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E+1) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller%2C+)
35. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito,… [find](https://scholar.google.com/scholar?q=Bo+Peng%2C+Eric+Alcaide%2C+Quentin+Anthony%2C+Alon+Albalak%2C+Samuel+Arcadinho%2C+Huanqi+Cao%2C+Xin+Cheng%2C+Michael+Chung%2C+Matteo+Grella%2C+Kranthi+Kiran+GV%2C+Xuzheng+He%2C+Haowen+Hou%2C+Przemyslaw+Kazienko%2C+Jan+Kocon%2C+J)
36. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmented language models, 2023. 3 [find](https://scholar.google.com/scholar?q=Ori+Ram%2C+Yoav+Levine%2C+Itay+Dalmedigos%2C+Dor+Muhlgay%2C+Amnon+Shashua%2C+Kevin+Leyton%2DBrown%2C+and+Yoav+Shoham%2E+In%2Dcontext+retrieval%2Daugmented+language+models%2C+2023%2E+3)
37. Arya Roy. Recent trends in named entity recognition (ner), 2021. 6 [find](https://scholar.google.com/scholar?q=Arya+Roy%2E+Recent+trends+in+named+entity+recognition+%28ner%29%2C+2021%2E+6)
38. Tomáš Koˇ ciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, TBD:TBD, 2018. URL https://TBD. 2 [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Ko%CB%87+cisky%CC%81%2C+Jonathan+Schwarz%2C+Phil+Blunsom%2C+Chris+Dyer%2C+Karl+Moritz+Hermann%2C+Ga%CC%81bor+Melis%2C+and+Edward+Grefenstette%2E+The+NarrativeQA+reading+comprehension+challenge%2E+Transactions+of+the+Associa) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
39. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. Scrolls: Standardized comparison over long language sequences, 2022. 2, 3 [find](https://scholar.google.com/scholar?q=Uri+Shaham%2C+Elad+Segal%2C+Maor+Ivgi%2C+Avia+Efrat%2C+Ori+Yoran%2C+Adi+Haviv%2C+Ankit+Gupta%2C+Wenhan+Xiong%2C+Mor+Geva%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+Scrolls%3A+Standardized+comparison+over+long+language+sequences%2C) [our card](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/)
40. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding, 2023. 2, 3 [find](https://scholar.google.com/scholar?q=Uri+Shaham%2C+Maor+Ivgi%2C+Avia+Efrat%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+Zeroscrolls%3A+A+zero%2Dshot+benchmark+for+long+text+understanding%2C+2023%2E+2%2C+3) [our card](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/)
41. Eva Sharma, Chen Li, and Lu Wang. Bigpatent: A large-scale dataset for abstractive and coherent summarization, 2019. 2 [find](https://scholar.google.com/scholar?q=Eva+Sharma%2C+Chen+Li%2C+and+Lu+Wang%2E+Bigpatent%3A+A+large%2Dscale+dataset+for+abstractive+and+coherent+summarization%2C+2019%2E+2)
42. Roshan Sharma, Suyoun Kim, Daniel Lazar, Trang Le, Akshat Shrivastava, Kwanghoon Ahn, Piyush Kansal, Leda Sari, Ozlem Kalinli, and Michael Seltzer. Augmenting text for spoken language understanding with large language models, 2023. 8 [find](https://scholar.google.com/scholar?q=Roshan+Sharma%2C+Suyoun+Kim%2C+Daniel+Lazar%2C+Trang+Le%2C+Akshat+Shrivastava%2C+Kwanghoon+Ahn%2C+Piyush+Kansal%2C+Leda+Sari%2C+Ozlem+Kalinli%2C+and+Michael+Seltzer%2E+Augmenting+text+for+spoken+language+understanding+wi)
43. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding, 2020. 10 [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+Mpnet%3A+Masked+and+permuted+pre%2Dtraining+for+language+understanding%2C+2020%2E+10) [our card](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/)
44. Gaurav Suri, Lily R. Slater, Ali Ziaee, and Morgan Nguyen. Do large language models show decision heuristics similar to humans? a case study using gpt-3.5, 2023. 8 [find](https://scholar.google.com/scholar?q=Gaurav+Suri%2C+Lily+R%2E+Slater%2C+Ali+Ziaee%2C+and+Morgan+Nguyen%2E+Do+large+language+models+show+decision+heuristics+similar+to+humans%3F+a+case+study+using+gpt%2D3%2E5%2C+2023%2E+8)
45. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers, 2020. 3 [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Samira+Abnar%2C+Yikang+Shen%2C+Dara+Bahri%2C+Philip+Pham%2C+Jinfeng+Rao%2C+Liu+Yang%2C+Sebastian+Ruder%2C+and+Donald+Metzler%2E+Long+range+arena%3A+A+benchmark+for+efficient+transformers%2C+2020) [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/)
46. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022. 3 [find](https://scholar.google.com/scholar?q=Yi+Tay%2C+Mostafa+Dehghani%2C+Dara+Bahri%2C+and+Donald+Metzler%2E+Efficient+transformers%3A+A+survey%2C+2022%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)
47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 7 [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edo)
48. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition, 2022. 2 [find](https://scholar.google.com/scholar?q=Harsh+Trivedi%2C+Niranjan+Balasubramanian%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+Musique%3A+Multihop+questions+via+single%2Dhop+question+composition%2C+2022%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/trivedi-2022-musique-multihop-questions-single-hop/)
49. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. Focused transformer: Contrastive training for context scaling, 2023. 7 [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2C+2023%2E+7)
50. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. Squality: Building a long-document summarization dataset the hard way, 2022. 2 [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Richard+Yuanzhe+Pang%2C+Angelica+Chen%2C+Jason+Phang%2C+and+Samuel+R%2E+Bowman%2E+Squality%3A+Building+a+long%2Ddocument+summarization+dataset+the+hard+way%2C+2022%2E+2)
51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. 11 [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2C+2023%2E+11) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
52. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021. 3 [find](https://scholar.google.com/scholar?q=Jeff+Wu%2C+Long+Ouyang%2C+Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Ryan+Lowe%2C+Jan+Leike%2C+and+Paul+Christiano%2E+Recursively+summarizing+books+with+human+feedback%2C+2021%2E+3) [our card](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/)
53. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Jingyu+Liu%2C+Igor+Molybog%2C+Hejia+Zhang%2C+Prajjwal+Bhargava%2C+Rui+Hou%2C+Louis+Martin%2C+Rashi+Rungta%2C+Karthik+Abinav+Sankararaman%2C+Barlas+Oguz%2C+Madian+Khabsa%2C+Han+Fang%2C+Yashar+Mehdad%2C+Sharan+Na) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
54. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models, 2023. 7 [find](https://scholar.google.com/scholar?q=Peng+Xu%2C+Wei+Ping%2C+Xianchao+Wu%2C+Lawrence+McAfee%2C+Chen+Zhu%2C+Zihan+Liu%2C+Sandeep+Subramanian%2C+Evelina+Bakhturina%2C+Mohammad+Shoeybi%2C+and+Bryan+Catanzaro%2E+Retrieval+meets+long+context+large+language+models) [our card](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/)
55. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. 2 [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W%2E+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2C+2018) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
56. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models, 2023. 7 [find](https://scholar.google.com/scholar?q=Junjie+Ye%2C+Xuanting+Chen%2C+Nuo+Xu%2C+Can+Zu%2C+Zekai+Shao%2C+Shichun+Liu%2C+Yuhan+Cui%2C+Zeyang+Zhou%2C+Chao+Gong%2C+Yang+Shen%2C+Jie+Zhou%2C+Siming+Chen%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+A+comprehensive+capabilit)
57. Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training a gpt4-style language model with multimodal inputs?, 2023. 1 [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Hanbo+Zhang%2C+Jiani+Zheng%2C+Jiangnan+Xia%2C+Guoqiang+Wei%2C+Yang+Wei%2C+Yuchen+Zhang%2C+and+Tao+Kong%2E+What+matters+in+training+a+gpt4%2Dstyle+language+model+with+multimodal+inputs%3F%2C+2023%2E+1)
58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 8 [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Das%2Da%2Djud)
59. Yucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao, Guodong Long, Can Xu, and Daxin Jiang. Fine-grained distillation for long document retrieval, 2022. 3 [find](https://scholar.google.com/scholar?q=Yucheng+Zhou%2C+Tao+Shen%2C+Xiubo+Geng%2C+Chongyang+Tao%2C+Guodong+Long%2C+Can+Xu%2C+and+Daxin+Jiang%2E+Fine%2Dgrained+distillation+for+long+document+retrieval%2C+2022%2E+3)