---
title: "Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization (2023)"
description: "Earlier work on large language models for summarization mostly examined generic news summaries. 17 references, cited by 57 papers here."
authors: ["Xianjun Yang", "Yan Li", "Xinlu Zhang", "Haifeng Chen", "Wei Cheng"]
year: 2023
venue: "arXiv:2302.08081"
source: "https://arxiv.org/abs/2302.08081"
doi: "10.48550/arxiv.2302.08081"
arxiv: "2302.08081"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

# Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

**Authors:** Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng · **Year:** 2023 · **Venue:** arXiv:2302.08081 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.08081) · [PDF](https://arxiv.org/pdf/2302.08081) · [DOI 10.48550/arxiv.2302.08081](https://doi.org/10.48550/arxiv.2302.08081) · [arXiv 2302.08081](https://arxiv.org/abs/2302.08081) · [OpenAlex](https://openalex.org/W4321276803)

**Citations:** 84 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/yang-2023-exploring-limits-chatgpt-query-aspect/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Earlier work on large language models for summarization mostly examined generic news summaries. This paper asks how ChatGPT performs in the harder, user-driven setting where a summary must answer a specific query or cover a given aspect. The authors tested it on four public benchmarks from different domains: QMSum (meetings) and SQuALITY (stories) for query-based summarization, and CovidET (Reddit posts) and NEWTS (news) for aspect-based summarization. No API was available, so 100 randomly sampled test instances per dataset were entered by hand into the ChatGPT web interface with simple task-specific prompts. The setup was zero-shot everywhere except CovidET, which received one in-context example. Outputs were scored with Rouge against published fine-tuned systems and profiled with coverage, density, compression and unique n-gram statistics. On average, ChatGPT scored close to the fine-tuned models and beat them on NEWTS. This suggests prompting alone can be a competitive, annotation-free baseline for controllable summarization. The authors note that Rouge may undervalue ChatGPT and that human evaluation is still pending.

## Key points
- **Datasets and input lengths:** QMSum averages about 9,070 input tokens (2,505 for gold spans), SQuALITY 6,052, NEWTS 602 and CovidET 192. Long inputs were constrained by ChatGPT's limit of roughly 5,000 tokens.
- **Average Rouge vs. fine-tuning:** ChatGPT scored R-1 30.94 vs. 32.90, R-2 8.96 vs. 9.34 and R-L 19.22 vs. 23.61.
- **Best and worst cases:**
  - With gold meeting spans, QMSum(Golden) beat fine-tuning on R-1 (36.83 vs. 36.06) and R-2 (12.78 vs. 11.36) but trailed on R-L (24.23 vs. 31.27).
  - On NEWTS, ChatGPT outperformed fine-tuning on every Rouge variant.
  - CovidET was weakest (R-1 20.81 vs. 26.19), which the authors attribute to its very short inputs and one-sentence targets.
- **Summary style:** ChatGPT summaries consistently had lower compression ratios than the references, meaning they were longer. Coverage and density showed no clear difference.
- **Proposed future directions:**
  - pairing ChatGPT with a lighter retrieval model such as LED for long documents;
  - detecting generated summaries that may be nonfactual or biased;
  - systematic prompt engineering and multi-turn self-correction.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 35 works · 9,287 citations
- [NEC](https://md.research.elamur.ai/free/en/companies/nec/) — Company, Irving, United States · 2 works · 526 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng. 2023. Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization. arXiv:2302.08081.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2023exploring,
  title = {{Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization}},
  author = {Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen and Wei Cheng},
  year = {2023},
  doi = {10.48550/arxiv.2302.08081},
  eprint = {2302.08081},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.08081}
}
```

- [← previous Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://md.research.elamur.ai/free/en/papers/xu-2023-large-language-models-really-good/)
- [next → Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://md.research.elamur.ai/free/en/papers/chung-2023-increasing-diversity-while-maintaining-accuracy/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

17 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Eickhoff. 2022. NEWTS: A corpus for news topic-focused summarization. In Findings of the Association for Computational Linguistics: ACL 2022, pages 493–503, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Seyed+Ali+Bahrainian%2C+Sheridan+Feucht%2C+and+Carsten+Eickhoff%2E+2022%2E+NEWTS%3A+A+corpus+for+news+topic%2Dfocused+summarization%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+4)
2. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. [our card](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bogdan+Gliwa%2C+Iwona+Mochol%2C+Maciej+Biesek%2C+and+Aleksander+Wawer%2E+2019%2E+SAMSum+corpus%3A+A+human%2Dannotated+dialogue+dataset+for+abstractive+summarization%2E+In+Proceedings+of+the+2nd+Workshop+on+New+Fronti)
6. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356. [find](https://scholar.google.com/scholar?q=Tanya+Goyal%2C+Junyi+Jessy+Li%2C+and+Greg+Durrett%2E+2022%2E+News+summarization+and+evaluation+in+the+era+of+gpt%2D3%2E+arXiv+preprint+arXiv%3A2209%2E12356%2E)
7. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28. [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
9. Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020. Aquamuse: Automatically generating datasets for query-based multi-document summarization. arXiv preprint arXiv:2010.12694. — [arXiv:2010.12694](https://arxiv.org/abs/2010.12694) [find](https://scholar.google.com/scholar?q=Sayali+Kulkarni%2C+Sheide+Chammas%2C+Wan+Zhu%2C+Fei+Sha%2C+and+Eugene+Ie%2E+2020%2E+Aquamuse%3A+Automatically+generating+datasets+for+query%2Dbased+multi%2Ddocument+summarization%2E+arXiv+preprint+arXiv%3A2010%2E12694%2E+%E2%80%94+%5Bar)
10. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
11. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+arXiv+preprint+arXiv%3A1808%2E0874)
12. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)
13. Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R Bowman. 2022. Squality: Building a long-document summarization dataset the hard way. arXiv preprint arXiv:2205.11465. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Richard+Yuanzhe+Pang%2C+Angelica+Chen%2C+Jason+Phang%2C+and+Samuel+R+Bowman%2E+2022%2E+Squality%3A+Building+a+long%2Ddocument+summarization+dataset+the+hard+way%2E+arXiv+preprint+arXiv%3A2205%2E11465%2E)
14. Xianjun Yang, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Xiaoman Pan, Linda Petzold, and Dong Yu. 2022. Oasum: Large-scale open domain aspect-based summarization. arXiv preprint arXiv:2212.09233. [find](https://scholar.google.com/scholar?q=Xianjun+Yang%2C+Kaiqiang+Song%2C+Sangwoo+Cho%2C+Xiaoyang+Wang%2C+Xiaoman+Pan%2C+Linda+Petzold%2C+and+Dong+Yu%2E+2022%2E+Oasum%3A+Large%2Dscale+open+domain+aspect%2Dbased+summarization%2E+arXiv+preprint+arXiv%3A2212%2E09233%2E)
15. Hongli Zhan, Tiberiu Sosea, Cornelia Caragea, and Junyi Jessy Li. 2022. Why do you feel this way? summarizing triggers of emotions in social media posts. arXiv preprint arXiv:2210.12531. [find](https://scholar.google.com/scholar?q=Hongli+Zhan%2C+Tiberiu+Sosea%2C+Cornelia+Caragea%2C+and+Junyi+Jessy+Li%2E+2022%2E+Why+do+you+feel+this+way%3F+summarizing+triggers+of+emotions+in+social+media+posts%2E+arXiv+preprint+arXiv%3A2210%2E12531%2E)
16. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/)
17. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chapte… [our card](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/)

*Updated 24 October 2026*