---
title: "On Context Utilization in Summarization with Large Language"
description: "The authors run a large study with 6 LLMs (Flan-UL2, Llama-2-7B/13B, Xgen-7B, Mistral-7B… 65 references, cited by 36 papers here."
authors: ["Mathieu Ravaut", "Aixin Sun", "Nancy Chen", "Shafiq Joty"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.153/"
doi: "10.18653/v1/2024.acl-long.153"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ravaut-2024-context-utilization-summarization-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ravaut-2024-context-utilization-summarization-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / On Context Utilization in Summarization with Large Language Models

# On Context Utilization in Summarization with Large Language Models

**Authors:** Mathieu Ravaut, [Aixin Sun](https://md.research.elamur.ai/free/en/authors/aixin-sun/), [Nancy Chen](https://md.research.elamur.ai/free/en/authors/nancy-chen/) and [Shafiq Joty](https://md.research.elamur.ai/free/en/authors/shafiq-joty/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.153/) · [PDF](https://aclanthology.org/2024.acl-long.153.pdf) · [DOI 10.18653/v1/2024.acl-long.153](https://doi.org/10.18653/v1/2024.acl-long.153) · [OpenAlex](https://openalex.org/W4402671958)

**Citations:** 31 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/ravaut-2024-context-utilization-summarization-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Recent LLMs can take in very long inputs, but question-answering studies have shown that they mostly attend to the start and end of the prompt and neglect the middle. This paper asks whether the same position bias affects abstractive summarization, where important facts are spread through the source and are usually reworded, so they are harder to trace. The authors run a large study with 6 LLMs (Flan-UL2, Llama-2-7B/13B, Xgen-7B, Mistral-7B, GPT-3.5) on 10 datasets of standard and long input length, scored with 5 automatic metrics. They locate the origin of summary content by matching summary bigrams and aligned sentences back to positions in the source. LLM outputs rely on document openings much more than reference summaries do. On some datasets a U-shaped pattern appears, which confirms a "middle curse." To measure the effect directly, they build MiddleSum, an evaluation set whose key content sits mid-context. On it they test hierarchical and incremental summarization. Both help, especially on scientific papers, but neither fully fixes the problem. The work questions how much summarization gains from ever-longer context windows.

## Key points
- **Datasets:** 5 standard-length datasets (under 2k tokens: CNN/DM, XSum, Reddit-TIFU, SAMSum, Multi-XScience) and 5 long-input datasets (Arxiv, PubMed, GovReport, SummScreenFD, Multi-News). Test sets are capped at 1,000 samples, or 300 when GPT-3.5 is used.
- **Metrics:** ROUGE-2, BERTScore, A3CU, SummaC, and GPT-3.5 acting as a 1–5 Likert-scale judge.
- **Lead bias:** Summary bigrams were mapped onto 20 equal source bins. The resulting distributions differ significantly from those of the references (Kolmogorov-Smirnov p \< 0.001) in all but 4 of the 55 dataset–model setups.
- **U-shape:** In the sentence-alignment analysis (10 bins), the first and last 10% of the context are over-represented. A clear U-shape appears for all LLMs on PubMed and SummScreenFD.
- **Model results:** Flan-UL2 leads on standard-length data and GPT-3.5 leads on long inputs. Code and the MiddleSum data are available at github.com/ntunlp/MiddleSum.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 156 works · 25,644 citations
- [Institute for Infocomm Research](https://md.research.elamur.ai/free/en/labs/institute-for-infocomm-research/) — Research institute, Singapore, Singapore · 43 works · 14,598 citations
- [Agency for Science, Technology and Research](https://md.research.elamur.ai/free/en/government/agency-for-science-technology-and-research/) — Government body, Singapore, Singapore · 30 works · 14,235 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works

## Cite

Cite as: Mathieu Ravaut, Aixin Sun, Nancy Chen and Shafiq Joty. 2024. On Context Utilization in Summarization with Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ravaut2024context,
  title = {{On Context Utilization in Summarization with Large Language Models}},
  author = {Mathieu Ravaut, Aixin Sun, Nancy Chen and Shafiq Joty},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.153},
  url = {https://doi.org/10.18653/v1/2024.acl-long.153}
}
```

- [← previous OffsetBias: Leveraging Debiased Data for Tuning Evaluators](https://md.research.elamur.ai/free/en/papers/park-2024-offsetbias-leveraging-debiased-data-tuning/)
- [next → OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://md.research.elamur.ai/free/en/papers/wu-2024-os-atlas-foundation-action-model/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

65 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1074%E2%80%931084%2C+Florence%2C+Italy%2E+Association+for+Computational+Linguistics%2E)
2. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. [Hierarchical neural story generation](https://md.research.elamur.ai/free/en/papers/fan-2018-hierarchical-neural-story-generation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
3. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
4. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
5. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356. [find](https://scholar.google.com/scholar?q=Tanya+Goyal%2C+Junyi+Jessy+Li%2C+and+Greg+Durrett%2E+2022%2E+News+summarization+and+evaluation+in+the+era+of+gpt%2D3%2E+arXiv+preprint+arXiv%3A2209%2E12356%2E)
6. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28.
7. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online.…
8. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023. Multi-dimensional evaluation of text summarization with in-context learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8487–8495,… [find](https://scholar.google.com/scholar?q=Sameer+Jain%2C+Vaishakh+Keshava%2C+Swarnashree+Mysore+Sathyendra%2C+Patrick+Fernandes%2C+Pengfei+Liu%2C+Graham+Neubig%2C+and+Chunting+Zhou%2E+2023%2E+Multi%2Ddimensional+evaluation+of+text+summarization+with+in%2Dcontext)
9. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023a%2E+Mis)
10. Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023c. [Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/). arXiv preprint arXiv:2310.06839.
11. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. [Abstractive summarization of Reddit posts with multi-level memory networks](https://md.research.elamur.ai/free/en/papers/kim-2019-abstractive-summarization-reddit-posts-multi/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short…
12. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N%2E+Bennett%2C+and+Marti+A%2E+Hearst%2E+2022%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Transactions+of+the+Association+for+Comp)
13. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
14. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
15. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–11…
16. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). arXiv preprint arXiv:2307.03172.
17. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
18. Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023b. [Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/). In Proceedings of the 61st Annual Meeting of the…
19. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023b. [Llmlingua: Compressing prompts for accelerated inference of large language models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/). arXiv preprint arXiv:2310.05736.
20. Yixin Liu, Alexander Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023c. Towards interpretable and efficient automatic reference-based summarization evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi… [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Alexander+Fabbri%2C+Yilun+Zhao%2C+Pengfei+Liu%2C+Shafiq+Joty%2C+Chien%2DSheng+Wu%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+2023c%2E+Towards+interpretable+and+efficient+automatic+reference%2Dbased+summarization)
21. Yixin Liu, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023d. [On learning to summarize with large language models as references](https://md.research.elamur.ai/free/en/papers/liu-2024-learning-summarize-large-language-models/). arXiv preprint arXiv:2305.14239.
22. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022b. [BRIO: Bringing order to abstractive summarization](https://md.research.elamur.ai/free/en/papers/liu-2022-brio-bringing-order-abstractive-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890–2903, Dublin, Ireland. Association for Computational Lin…
23. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). arXiv preprint arXiv:2104.08786.
24. Yao Lu, Yue Dong, and Laurent Charlin. 2020. [Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles](https://md.research.elamur.ai/free/en/papers/lu-2020-multi-xscience-large-scale-dataset/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8068–8074, Online. Association for Computationa…
25. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/).
26. Frank J Massey Jr. 1951. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46(253):68–78. [find](https://scholar.google.com/scholar?q=Frank+J+Massey+Jr%2E+1951%2E+The+kolmogorov%2Dsmirnov+test+for+goodness+of+fit%2E+Journal+of+the+American+statistical+Association%2C+46%28253%29%3A68%E2%80%9378%2E)
27. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
28. Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, et al. 2023. Xgen-7b technical report. arXiv preprint arXiv:2309.03450. — [arXiv:2309.03450](https://arxiv.org/abs/2309.03450) [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Tian+Xie%2C+Hiroaki+Hayashi%2C+Bo+Pang%2C+Congying+Xia%2C+Chen+Xing%2C+Jesse+Vig%2C+Semih+Yavuz%2C+Philippe+Laban%2C+Ben+Krause%2C+et+al%2E+2023%2E+Xgen%2D7b+technical+report%2E+arXiv+preprint+arXiv%3A2309%2E03450%2E+%E2%80%94)
29. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
30. Bo Pang, Erik Nijkamp, Wojciech Kryściński, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2022. Long document summarization with top-down and bottom-up inference. arXiv preprint arXiv:2203.07586. [find](https://scholar.google.com/scholar?q=Bo+Pang%2C+Erik+Nijkamp%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Silvio+Savarese%2C+Yingbo+Zhou%2C+and+Caiming+Xiong%2E+2022%2E+Long+document+summarization+with+top%2Ddown+and+bottom%2Dup+inference%2E+arXiv+preprint+arXiv%3A2203%2E07586%2E)
31. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
32. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+2023%2E+Yarn%3A+Efficient+context+window+extension+of+large+language+models%2E+arXiv+preprint+arXiv%3A2309%2E00071%2E)
33. Pouya Pezeshkpour and Estevam Hruschka. 2023. [Large language models sensitivity to the order of options in multiple-choice questions](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/). arXiv preprint arXiv:2308.11483.
34. Ofir Press, Noah A Smith, and Mike Lewis. 2021. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A+Smith%2C+and+Mike+Lewis%2E+2021%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2E+arXiv+preprint+arXiv%3A2108%2E12409%2E)
35. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. arXiv preprint arXiv:2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) [find](https://scholar.google.com/scholar?q=Xiao+Pu%2C+Mingqi+Gao%2C+and+Xiaojun+Wan%2E+2023%2E+Summarization+is+%28almost%29+dead%2E+arXiv+preprint+arXiv%3A2309%2E09558%2E+%E2%80%94+%5BarXiv%3A2309%2E09558%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E09558%29)
36. Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022. Towards summary candidates fusion. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8488–8504, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mathieu+Ravaut%2C+Shafiq+Joty%2C+and+Nancy+Chen%2E+2022%2E+Towards+summary+candidates+fusion%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+8488%E2%80%938504%2C+Abu+Dh)
37. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
38. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
39. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
40. Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. 2022. [SCROLLS: Standardized CompaRison over long language sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pa…
41. Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing. 2023a. Are large language models good evaluators for abstractive summarization? arXiv preprint arXiv:2305.13091. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Yang+You%2C+and+Lidong+Bing%2E+2023a%2E+Are+large+language+models+good+evaluators+for+abstractive+summarization%3F+arXiv+preprint+arXiv%3A2305%2E13091%2E)
42. Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. 2023b. Positional description matters for transformers arithmetic. arXiv preprint arXiv:2311.14737. [find](https://scholar.google.com/scholar?q=Ruoqi+Shen%2C+S%C3%A9bastien+Bubeck%2C+Ronen+Eldan%2C+Yin+Tat+Lee%2C+Yuanzhi+Li%2C+and+Yi+Zhang%2E+2023b%2E+Positional+description+matters+for+transformers+arithmetic%2E+arXiv+preprint+arXiv%3A2311%2E14737%2E)
43. Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. 2021. [Do long-range language models actually use long-range context?](https://md.research.elamur.ai/free/en/papers/sun-2021-do-long-range-language-models/) In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807– 822, Online and Punta Cana, Dominican Republic. Associatio…
44. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. [A length-extrapolatable transformer](https://md.research.elamur.ai/free/en/papers/sun-2023-length-extrapolatable-transformer/). arXiv preprint arXiv:2212.10554.
45. Raphael Tang, Xinyu Zhang, Xueguang Ma, Jimmy Lin, and Ferhan Ture. 2023. Found in the middle: Permutation self-consistency improves listwise ranking in large language models. arXiv preprint arXiv:2310.07712. [find](https://scholar.google.com/scholar?q=Raphael+Tang%2C+Xinyu+Zhang%2C+Xueguang+Ma%2C+Jimmy+Lin%2C+and+Ferhan+Ture%2E+2023%2E+Found+in+the+middle%3A+Permutation+self%2Dconsistency+improves+listwise+ranking+in+large+language+models%2E+arXiv+preprint+arXiv%3A231)
46. Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, et al. 2022. [Ul2: Unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In The Eleventh International Conference on Learning Representations.
47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
48. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
49. Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Toronto%2C+Canada%2E+Association+for+Computational+Linguistics%2E)
50. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652.
51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
52. Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. [PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization](https://md.research.elamur.ai/free/en/papers/xiao-2022-primera-pyramid-based-masked-sentence/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5245–5263, Dublin, Ireland.…
53. Jiawen Xie, Qi Su, Shaoting Zhang, and Xiaofan Zhang. 2023. Alleviating exposure bias via multi-level contrastive learning and deviation simulation in abstractive summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9732–9747, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Jiawen+Xie%2C+Qi+Su%2C+Shaoting+Zhang%2C+and+Xiaofan+Zhang%2E+2023%2E+Alleviating+exposure+bias+via+multi%2Dlevel+contrastive+learning+and+deviation+simulation+in+abstractive+summarization%2E+In+Findings+of+the+Ass)
54. Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Scott Yih. 2023. Adapting pre-trained text-to-text models for long text sequences. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5566–5578, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Anchit+Gupta%2C+Shubham+Toshniwal%2C+Yashar+Mehdad%2C+and+Scott+Yih%2E+2023%2E+Adapting+pre%2Dtrained+text%2Dto%2Dtext+models+for+long+text+sequences%2E+In+Findings+of+the+Association+for+Computational+Li)
55. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Summit: Iterative text summarization via chatgpt. arXiv preprint arXiv:2305.14835. [find](https://scholar.google.com/scholar?q=Haopeng+Zhang%2C+Xiao+Liu%2C+and+Jiawei+Zhang%2E+2023a%2E+Summit%3A+Iterative+text+summarization+via+chatgpt%2E+arXiv+preprint+arXiv%3A2305%2E14835%2E)
56. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
57. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
58. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
59. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023b. [Benchmarking large language models for news summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). arXiv preprint arXiv:2301.13848.
60. Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023c. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8640–8665, Chao… [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Zhuosheng+Zhang%2C+and+Rui+Wang%2E+2023c%2E+Element%2Daware+summarization+with+large+language+models%3A+Expert%2Daligned+evaluation+and+chain%2Dof%2Dthought+method%2E+In+Proceedings+of+the+61st+Annual+Meet)
61. Summarize the above text in [n] sentences. [find](https://scholar.google.com/scholar?q=Summarize+the+above+text+in+%5Bn%5D+sentences%2E)
62. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2022b. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.00045. — [arXiv:2210.00045](https://arxiv.org/abs/2210.00045) [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Misha+Khalman%2C+Rishabh+Joshi%2C+Shashi+Narayan%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+2022b%2E+Calibrating+sequence+likelihood+improves+conditional+language+generation%2E+arXiv+preprint+arXiv%3A2210%2E0004)
63. Summary: [summary generated by the LLM to evaluate] [find](https://scholar.google.com/scholar?q=Summary%3A+%5Bsummary+generated+by+the+LLM+to+evaluate%5D)
64. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. arXiv e-prints, pages arXiv–2309. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2023%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932309%2E)
65. When evaluating for the specific aspect of informativeness, the first paragraph becomes: Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. [Neural document summarization by jointly learning to score and select sentences](https://md.research.elamur.ai/free/en/papers/zhou-2018-neural-document-summarization-jointly-learning/). In Proceedings of the 56th Annual Meeting of the…

*Updated 29 October 2026*