---
title: "Summary of a Haystack — paper by Laban et al. (2024)"
description: "The authors propose SummHay, a benchmark built from synthetic document collections (\"Haystacks\")… 58 references, cited by 43 papers here."
authors: ["Philippe Laban", "Alexander Fabbri", "Caiming Xiong", "Chien-Sheng Wu"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.552/"
doi: "10.18653/v1/2024.emnlp-main.552"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/laban-2024-summary-haystack-challenge-long-context/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/laban-2024-summary-haystack-challenge-long-context/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems

# Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems

**Authors:** [Philippe Laban](https://md.research.elamur.ai/free/en/authors/philippe-laban/), [Alexander Fabbri](https://md.research.elamur.ai/free/en/authors/alexander-fabbri/), [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/) and [Chien-Sheng Wu](https://md.research.elamur.ai/free/en/authors/chien-sheng-wu/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.552/) · [PDF](https://aclanthology.org/2024.emnlp-main.552.pdf) · [DOI 10.18653/v1/2024.emnlp-main.552](https://doi.org/10.18653/v1/2024.emnlp-main.552) · [OpenAlex](https://openalex.org/W4404783159)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/laban-2024-summary-haystack-challenge-long-context/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Long-context LLMs and RAG pipelines can now take in enormous inputs, but simple probes such as Needle-in-a-Haystack are too easy to separate strong systems. Traditional summarization evaluation also depends on costly reference summaries and weak automatic metrics. The authors propose SummHay, a benchmark built from synthetic document collections ("Haystacks") in two domains, conversations and news. In each Haystack, insights grouped by subtopic are deliberately planted and repeated across specific documents. Because the location of every insight is known, a system's query-focused summary can be scored automatically. Scoring has two parts: Coverage (whether the expected insights appear) and Citation (whether the correct source documents are referenced). The two are combined into a Joint Score.

The authors evaluated 14 LLMs and 50 RAG systems. Every system falls well short of an estimated human score of 56%, even when given oracle knowledge of which documents are relevant. The benchmark therefore offers a demanding and reproducible way to compare long-context models and retrieval pipelines on reasoning, aggregation and attribution.

## Key points
- **Dataset size:** 10 Haystacks, each with about 100 documents (roughly 100k tokens) and around 10 queries, giving 92 SummHay tasks. Code and data are available at github.com/salesforce/summary-of-a-haystack.
- **Evaluation reliability:** Manual evaluation among annotators correlated at 0.77. LLM-based scoring correlated at 0.71 while costing about 50 times less.
- **Long-context results:** When run without a retriever, neither GPT-4o nor Claude 3 Opus reaches 20% on the Joint Score. Oracle-relevance setups still trail the human estimate by more than 10 points.
- **RAG findings:** RAG pipelines, which receive up to 15k retrieved tokens, tend to cite more accurately but cover fewer insights. Stronger components such as Cohere's Rerank3 improve end-to-end results.
- **Position bias:** A positional bias experiment reproduces the "lost in the middle" effect. Limitations include independently generated documents and the exclusion of open models such as Llama-3 because of context-length limits.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 56 works · 17,867 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Philippe Laban, Alexander Fabbri, Caiming Xiong and Chien-Sheng Wu. 2024. Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{laban2024summary,
  title = {{Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems}},
  author = {Philippe Laban, Alexander Fabbri, Caiming Xiong and Chien-Sheng Wu},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.552},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.552}
}
```

- [← previous Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://md.research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/)
- [next → A Thorough Examination of Decoding Methods in the Era of LLMs](https://md.research.elamur.ai/free/en/papers/shi-2024-thorough-examination-decoding-methods-era/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

58 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+2024%2E+Llama+3+model+card%2E)
2. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). arXiv preprint arXiv:2307.11088.
3. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508.
4. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150.
5. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
6. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2023. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785. [find](https://scholar.google.com/scholar?q=Yapei+Chang%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2023%2E+Booookscore%3A+A+systematic+exploration+of+book%2Dlength+summarization+in+the+era+of+llms%2E+arXiv+preprint+arXiv%3A2310%2E00785%2E)
7. Hailin Chen, Fangkai Jiao, Xingxuan Li, Chengwei Qin, Mathieu Ravaut, Ruochen Zhao, Caiming Xiong, and Shafiq Joty. 2023a. Chatgpt’s one-year anniversary: Are open-source large language models catching up? arXiv preprint arXiv:2311.16989. — [arXiv:2311.16989](https://arxiv.org/abs/2311.16989) [find](https://scholar.google.com/scholar?q=Hailin+Chen%2C+Fangkai+Jiao%2C+Xingxuan+Li%2C+Chengwei+Qin%2C+Mathieu+Ravaut%2C+Ruochen+Zhao%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2023a%2E+Chatgpt%E2%80%99s+one%2Dyear+anniversary%3A+Are+open%2Dsource+large+language+models+catchin)
8. Hung-Ting Chen, Fangyuan Xu, Shane A Arora, and Eunsol Choi. 2023b. [Understanding retrieval augmentation for long-form question answering](https://md.research.elamur.ai/free/en/papers/chen-2023-understanding-retrieval-augmentation-long-form/). arXiv preprint arXiv:2310.12150.
9. Yanran Chen and Steffen Eger. 2023. [Menli: Robust evaluation metrics from natural language inference](https://md.research.elamur.ai/free/en/papers/chen-2023-menli-robust-evaluation-metrics-natural/). Transactions of the Association for Computational Linguistics, 11:804–825.
10. Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur P Parikh. 2023. Seahorse: A multilingual, multifaceted dataset for summarization evaluation. arXiv preprint arXiv:2305.13194. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Shruti+Rijhwani%2C+Sebastian+Gehrmann%2C+Joshua+Maynez%2C+Roee+Aharoni%2C+Vitaly+Nikolaev%2C+Thibault+Sellam%2C+Aditya+Siddhant%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+2023%2E+Seahorse%3A+A+multilingual%2C+)
11. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2023. [Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). arXiv preprint arXiv:2309.13345.
12. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021a. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021a%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+fo)
13. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021b. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021b%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+f)
14. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
15. Mingqi Gao and Xiaojun Wan. 2022. DialSummEval: Revisiting summarization evaluation for dialogues. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5693–5709, Seattle, United States. Association f… [find](https://scholar.google.com/scholar?q=Mingqi+Gao+and+Xiaojun+Wan%2E+2022%2E+DialSummEval%3A+Revisiting+summarization+evaluation+for+dialogues%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Computatio)
16. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
17. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. [Retrieval augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In International conference on machine learning, pages 3929–3938. PMLR.
18. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NIPS, pages 1693–1701.
19. Nan Hu, Jiaoyan Chen, Yike Wu, Guilin Qi, Sheng Bi, Tongtong Wu, and Jeff Z Pan. 2024. Benchmarking large language models in complex question answering attribution using knowledge graphs. arXiv preprint arXiv:2401.14640. — [arXiv:2401.14640](https://arxiv.org/abs/2401.14640) [find](https://scholar.google.com/scholar?q=Nan+Hu%2C+Jiaoyan+Chen%2C+Yike+Wu%2C+Guilin+Qi%2C+Sheng+Bi%2C+Tongtong+Wu%2C+and+Jeff+Z+Pan%2E+2024%2E+Benchmarking+large+language+models+in+complex+question+answering+attribution+using+knowledge+graphs%2E+arXiv+prepri)
20. Kung-Hsiang Huang, Philippe Laban, Alexander R Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint… [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Philippe+Laban%2C+Alexander+R+Fabbri%2C+Prafulla+Kumar+Choubey%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Chien%2DSheng+Wu%2E+2023%2E+Embrace+divergence+for+richer+insights%3A+A+multi%2Ddocument+summarizat)
21. Cohere Inc. 2024. Introducing rerank 3: The next generation of search relevance. Accessed: 2024-06-10. [find](https://scholar.google.com/scholar?q=Cohere+Inc%2E+2024%2E+Introducing+rerank+3%3A+The+next+generation+of+search+relevance%2E+Accessed%3A+2024%2D06%2D10%2E)
22. Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307.16883. — [arXiv:2307.16883](https://arxiv.org/abs/2307.16883) [find](https://scholar.google.com/scholar?q=Ehsan+Kamalloo%2C+Aref+Jafari%2C+Xinyu+Zhang%2C+Nandan+Thakur%2C+and+Jimmy+Lin%2E+2023%2E+Hagrid%3A+A+human%2Dllm+collaborative+dataset+for+generative+information%2Dseeking+with+attribution%2E+arXiv+preprint+arXiv%3A2307%2E1)
23. Gregory Kamradt. 2023. Needleinahaystack. [find](https://scholar.google.com/scholar?q=Gregory+Kamradt%2E+2023%2E+Needleinahaystack%2E)
24. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261. — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) [find](https://scholar.google.com/scholar?q=Yekyung+Kim%2C+Yapei+Chang%2C+Marzena+Karpinska%2C+Aparna+Garimella%2C+Varun+Manjunatha%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2024%2E+Fables%3A+Evaluating+faithfulness+and+content+selection+in+book%2Dlength+summa)
25. Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. [Longeval: Guidelines for human evaluation of faithfulness in long-form summarization](https://md.research.elamur.ai/free/en/papers/krishna-2023-longeval-guidelines-human-evaluation-faithfulness/). arXiv preprint arXiv:2301.13298.
26. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. In search of needles in a 10m haystack: Recurrent memory finds what llms miss. arXiv preprint arXiv:2402.10790. — [arXiv:2402.10790](https://arxiv.org/abs/2402.10790) [find](https://scholar.google.com/scholar?q=Yuri+Kuratov%2C+Aydar+Bulatov%2C+Petr+Anokhin%2C+Dmitry+Sorokin%2C+Artyom+Sorokin%2C+and+Mikhail+Burtsev%2E+2024%2E+In+search+of+needles+in+a+10m+haystack%3A+Recurrent+memory+finds+what+llms+miss%2E+arXiv+preprint+arXi)
27. Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. 2023. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310.19240. [find](https://scholar.google.com/scholar?q=Wai%2DChung+Kwan%2C+Xingshan+Zeng%2C+Yufei+Wang%2C+Yusen+Sun%2C+Liangyou+Li%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Kam%2DFai+Wong%2E+2023%2E+M4le%3A+A+multi%2Dability+multi%2Drange+multi%2Dtask+multi%2Ddomain+long%2Dcontext+evaluation+benc)
28. Philippe Laban, Andrew Hsi, John Canny, and Marti A Hearst. 2020. The summary loop: Learning to write abstractive summaries without examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5135–5150. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Andrew+Hsi%2C+John+Canny%2C+and+Marti+A+Hearst%2E+2020%2E+The+summary+loop%3A+Learning+to+write+abstractive+summaries+without+examples%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associati)
29. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022a. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N+Bennett%2C+and+Marti+A+Hearst%2E+2022a%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Transactions+of+the+Association+for+Compu)
30. Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs’ka, Xiang Chen, and Caiming Xiong. 2022b. Discord questions: A computational approach to diversity analysis in news coverage. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5180–5194. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Chien%2DSheng+Wu%2C+Lidiya+Murakhovs%E2%80%99ka%2C+Xiang+Chen%2C+and+Caiming+Xiong%2E+2022b%2E+Discord+questions%3A+A+computational+approach+to+diversity+analysis+in+news+coverage%2E+In+Findings+of+the+Associ)
31. LangChain. 2024. Multi-needle in a haystack. [find](https://scholar.google.com/scholar?q=LangChain%2E+2024%2E+Multi%2Dneedle+in+a+haystack%2E)
32. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
33. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33:9459–9474.
34. Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aiguo Wu, and Min Zhang. 2023a. A survey of large language models attribution. arXiv preprint arXiv:2311.03731. — [arXiv:2311.03731](https://arxiv.org/abs/2311.03731) [find](https://scholar.google.com/scholar?q=Dongfang+Li%2C+Zetian+Sun%2C+Xinshuo+Hu%2C+Zhenyu+Liu%2C+Ziyang+Chen%2C+Baotian+Hu%2C+Aiguo+Wu%2C+and+Min+Zhang%2E+2023a%2E+A+survey+of+large+language+models+attribution%2E+arXiv+preprint+arXiv%3A2311%2E03731%2E+%E2%80%94+%5BarXiv%3A2311%2E)
35. Xinze Li, Yixin Cao, Liangming Pan, Yubo Ma, and Aixin Sun. 2023b. Towards verifiable generation: A benchmark for knowledge-aware language model attribution. arXiv preprint arXiv:2310.05634. [find](https://scholar.google.com/scholar?q=Xinze+Li%2C+Yixin+Cao%2C+Liangming+Pan%2C+Yubo+Ma%2C+and+Aixin+Sun%2E+2023b%2E+Towards+verifiable+generation%3A+A+benchmark+for+knowledge%2Daware+language+model+attribution%2E+arXiv+preprint+arXiv%3A2310%2E05634%2E)
36. Yifei Li, Xiang Yue, Zeyi Liao, and Huan Sun. 2024. Attributionbench: How hard is automatic attribution evaluation? arXiv preprint arXiv:2402.15089. [find](https://scholar.google.com/scholar?q=Yifei+Li%2C+Xiang+Yue%2C+Zeyi+Liao%2C+and+Huan+Sun%2E+2024%2E+Attributionbench%3A+How+hard+is+automatic+attribution+evaluation%3F+arXiv+preprint+arXiv%3A2402%2E15089%2E)
37. Yixin Liu, Alexander R Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, et al. 2022. [Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/). arXiv preprint arXiv:2212.07981.
38. Yixin Liu, Alexander R Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023. Towards interpretable and efficient automatic reference-based summarization evaluation. arXiv preprint arXiv:2303.03608. [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Alexander+R+Fabbri%2C+Yilun+Zhao%2C+Pengfei+Liu%2C+Shafiq+Joty%2C+Chien%2DSheng+Wu%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+2023%2E+Towards+interpretable+and+efficient+automatic+reference%2Dbased+summarizatio)
39. Daniel Machlab and Rick Battle. 2024. Llm in-context recall is prompt dependent. arXiv preprint arXiv:2404.08865. — [arXiv:2404.08865](https://arxiv.org/abs/2404.08865) [find](https://scholar.google.com/scholar?q=Daniel+Machlab+and+Rick+Battle%2E+2024%2E+Llm+in%2Dcontext+recall+is+prompt+dependent%2E+arXiv+preprint+arXiv%3A2404%2E08865%2E+%E2%80%94+%5BarXiv%3A2404%2E08865%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E08865%29)
40. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022. [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/). arXiv preprint arXiv:2203.11147.
41. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251.
42. Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, and Piji Li. 2024. Xl bench: A benchmark for extremely long context understanding with long-range dependencies. arXiv preprint arXiv:2404.05446. — [arXiv:2404.05446](https://arxiv.org/abs/2404.05446) [find](https://scholar.google.com/scholar?q=Xuanfan+Ni%2C+Hengyi+Cai%2C+Xiaochi+Wei%2C+Shuaiqiang+Wang%2C+Dawei+Yin%2C+and+Piji+Li%2E+2024%2E+Xl+bench%3A+A+benchmark+for+extremely+long+context+understanding+with+long%2Drange+dependencies%2E+arXiv+preprint+arXiv%3A24)
43. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140):1–67.
44. Mathieu Ravaut, Shafiq Joty, Aixin Sun, and Nancy F Chen. 2023. [On context utilization in summarization with large language models](https://md.research.elamur.ai/free/en/papers/ravaut-2024-context-utilization-summarization-large-language/). arXiv e-prints, pages arXiv–2310.
45. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.…
46. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
47. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). arXiv preprint arXiv:2305.14196.
48. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532. — [arXiv:2404.18532](https://arxiv.org/abs/2404.18532) [find](https://scholar.google.com/scholar?q=Dingjie+Song%2C+Shunian+Chen%2C+Guiming+Hardy+Chen%2C+Fei+Yu%2C+Xiang+Wan%2C+and+Benyou+Wang%2E+2024%2E+Milebench%3A+Benchmarking+mllms+in+long+context%2E+arXiv+preprint+arXiv%3A2404%2E18532%2E+%E2%80%94+%5BarXiv%3A2404%2E18532%5D%28https%3A%2F%2Fa)
49. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063.
50. Liyan Tang, Philippe Laban, and Greg Durrett. 2024. [Minicheck: Efficient fact-checking of llms on grounding documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/). arXiv preprint arXiv:2404.10774.
51. Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for query-focused summarization. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1455–1468, Seattle, United States. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Jesse+Vig%2C+Alexander+Fabbri%2C+Wojciech+Kryscinski%2C+Chien%2DSheng+Wu%2C+and+Wenhao+Liu%2E+2022%2E+Exploring+neural+models+for+query%2Dfocused+summarization%2E+In+Findings+of+the+Association+for+Computational+Lingui)
52. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771.
53. Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023. Less is more for long document summary evaluation by llms. arXiv preprint arXiv:2309.07382. [find](https://scholar.google.com/scholar?q=Yunshu+Wu%2C+Hayate+Iso%2C+Pouya+Pezeshkpour%2C+Nikita+Bhutani%2C+and+Estevam+Hruschka%2E+2023%2E+Less+is+more+for+long+document+summary+evaluation+by+llms%2E+arXiv+preprint+arXiv%3A2309%2E07382%2E)
54. Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Boshi+Wang%2C+Ziru+Chen%2C+Kai+Zhang%2C+Yu+Su%2C+and+Huan+Sun%2E+2023%2E+Automatic+evaluation+of+attribution+by+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E06311%2E)
55. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [Alignscore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348.
56. Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024a. Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks. arXiv preprint arXiv:2402.17630. [find](https://scholar.google.com/scholar?q=Huajian+Zhang%2C+Yumo+Xu%2C+and+Laura+Perez%2DBeltrachini%2E+2024a%2E+Fine%2Dgrained+natural+language+inference+based+faithfulness+evaluation+for+diverse+summarisation+tasks%2E+arXiv+preprint+arXiv%3A2402%2E17630%2E)
57. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. 2024b. Infinitybench: Extending long context evaluation beyond 100k tokens. arXiv:2402.13718. arXiv preprint Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma… — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) [find](https://scholar.google.com/scholar?q=Xinrong+Zhang%2C+Yingfa+Chen%2C+Shengding+Hu%2C+Zihang+Xu%2C+Junhao+Chen%2C+Moo+Khai+Hao%2C+Xu+Han%2C+Zhen+Leng+Thai%2C+Shuo+Wang%2C+Zhiyuan+Liu%2C+et+al%2E+2024b%2E+Infinitybench%3A+Extending+long+context+evaluation+beyond+10)
58. Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. [Longembed: Extending embedding models for long context retrieval](https://md.research.elamur.ai/free/en/papers/zhu-2024-longembed-extending-embedding-models-long/). arXiv preprint arXiv:2404.12096.

*Updated 31 October 2026*