---
title: "On Learning to Summarize with Large Language Models as"
description: "Reference summaries in popular news datasets such as CNN/DailyMail are often rated below LLM… 71 references, cited by 55 papers here."
authors: ["Yixin Liu", "Kejian Shi", "Katherine He", "Longtian Ye", "Alexander Fabbri", "Pengfei Liu", "Dragomir Radev", "Arman Cohan"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.478/"
doi: "10.18653/v1/2024.naacl-long.478"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-learning-summarize-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-learning-summarize-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / On Learning to Summarize with Large Language Models as References

# On Learning to Summarize with Large Language Models as References

**Authors:** [Yixin Liu](https://md.research.elamur.ai/free/en/authors/yixin-liu/), [Kejian Shi](https://md.research.elamur.ai/free/en/authors/kejian-shi/), Katherine He, Longtian Ye, [Alexander Fabbri](https://md.research.elamur.ai/free/en/authors/alexander-fabbri/), [Pengfei Liu](https://md.research.elamur.ai/free/en/authors/pengfei-liu/), [Dragomir Radev](https://md.research.elamur.ai/free/en/authors/dragomir-radev/) and [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.478/) · [PDF](https://aclanthology.org/2024.naacl-long.478.pdf) · [DOI 10.18653/v1/2024.naacl-long.478](https://doi.org/10.18653/v1/2024.naacl-long.478) · [OpenAlex](https://openalex.org/W4401042394)

**Citations:** 39 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/liu-2024-learning-summarize-large-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Reference summaries in popular news datasets such as CNN/DailyMail are often rated below LLM outputs by human judges. This caps how good models trained on them can become. The authors ask whether a small summarizer, BART (about 350M parameters), improves when LLMs serve as the gold standard. They use LLMs in two roles. First, LLMs write target summaries for ordinary MLE fine-tuning, a form of sequence-level distillation. Second, LLMs score or rank candidate summaries, and these rankings drive BRIO-style contrastive training. For scoring they use GPTScore, based on LLM log-probabilities, and GPTRank, a G-Eval extension in which the LLM explains its reasoning and then ranks several candidates.

Experiments on CNNDM articles, in both low- and high-resource regimes, show two things:
- LLM-written targets beat the original references.
- Contrastive learning beats plain MLE.

The best BART checkpoint surpasses GPT-3.5 when GPT-4 is the judge. However, expert human evaluation reveals subtle gaps that LLM evaluators miss. This warns against relying on LLMs alone to evaluate systems of similar quality.

## Key points
- **Training setup:** BART-large is warmed up on roughly 10K GPT-3.5 summaries (gpt-3.5-turbo-0301). It is then trained further with MLE or contrastive learning on about 100–1000 examples, using 8 candidates per example. LLM API cost is kept comparable across the two methods.
- **GPTScore results** (GPT3D3 as reference; higher is better): BRIO.GPT3D3 scores −0.318, compared with −0.420 for MLE-trained BART.GPT3D3 and −0.498 for GPT-3.5.
- **Human evaluation:** experts judged the best model equal to or better than GPT-3.5 in 50% of cases. The originally fine-tuned BART reached this in only 4% of cases.
- **Meta-analysis:** LLM-based evaluators correlate well on benchmarks like SummEval. They correlate poorly with humans, however, when the compared systems are close in quality.
- **Resources and limitations:** scripts, training data and outputs are released at github.com/yixinL7/SumLLM. Evaluation used 100 sampled test articles, and factual consistency was not assessed separately.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 37 works · 15,019 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 63 works · 15,047 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 54 works · 11,964 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 159 works

## Cite

Cite as: Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev and Arman Cohan. 2024. On Learning to Summarize with Large Language Models as References. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2024learning,
  title = {{On Learning to Summarize with Large Language Models as References}},
  author = {Yixin Liu, Kejian Shi, Katherine He, Longtian Ye, Alexander Fabbri, Pengfei Liu, Dragomir Radev and Arman Cohan},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.478},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.478}
}
```

- [← previous Moral Foundations of Large Language Models](https://md.research.elamur.ai/free/en/papers/abdulhai-2024-moral-foundations-large-language-models/)
- [next → Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

71 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. [An actor-critic algorithm for sequence prediction](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/). In International Conference on Learning Representations.
2. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1171–1179, Cambridge, MA, USA. MIT Pre… [find](https://scholar.google.com/scholar?q=Samy+Bengio%2C+Oriol+Vinyals%2C+Navdeep+Jaitly%2C+and+Noam+Shazeer%2E+2015%2E+Scheduled+sampling+for+sequence+prediction+with+recurrent+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Ne)
3. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Shuyang Cao and Lu Wang. 2021. [CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Co…
5. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. [Exploring the use of large language models for reference-free text quality evaluation: An empirical study](https://md.research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/). In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), pages 361–374, Nusa Dua, Bali. Ass…
6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
7. [Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Ya…](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
8. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. [Is GPT-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/) In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for…
9. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). In Thirty-seventh Conference on Neural Information Processing Systems.
10. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. [Classical structured prediction losses for sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/edunov-2018-classical-structured-prediction-losses-sequence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
11. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+fo)
12. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTScore: Evaluate as you desire. ArXiv, abs/2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+GPTScore%3A+Evaluate+as+you+desire%2E+ArXiv%2C+abs%2F2302%2E04166%2E)
13. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. ArXiv, abs/2304.02554. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E+ArXiv%2C+abs%2F2304%2E02554%2E)
14. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Tosin+Adewumi%2C+Karmanya+Aggarwal%2C+Pawan+Sasanka+Ammanamanchi%2C+Anuoluwapo+Aremu%2C+Antoine+Bosselut%2C+Khyathi+Raghavi+Chandu%2C+Miruna%2DAdriana+Clinciu%2C+Dipanjan+Das%2C+Kaustubh+Dhole%2C+Wany)
15. Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, D… [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Abhik+Bhattacharjee%2C+Abinaya+Mahendiran%2C+Alex+Wang%2C+Alexandros+Papangelis%2C+Aman+Madaan%2C+Angelina+McMillan%2DMajor%2C+Anna+Shvets%2C+Ashish+Upadhyay%2C+Bingsheng+Yao%2C+Bryan+Wilie%2C+Chandra+B)
16. Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. [TrueTeacher: Learning factual consistency evaluation with large language models](https://md.research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2053–2070, Singapore. Association for…
17. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). ArXiv, abs/2303.15056.
18. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. ArXiv, abs/2209.12356. [find](https://scholar.google.com/scholar?q=Tanya+Goyal%2C+Junyi+Jessy+Li%2C+and+Greg+Durrett%2E+2022%2E+News+summarization+and+evaluation+in+the+era+of+gpt%2D3%2E+ArXiv%2C+abs%2F2209%2E12356%2E)
19. Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2, CVPR ’06, page 1735–1742, USA. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=Raia+Hadsell%2C+Sumit+Chopra%2C+and+Yann+LeCun%2E+2006%2E+Dimensionality+reduction+by+learning+an+invariant+mapping%2E+In+Proceedings+of+the+2006+IEEE+Computer+Society+Conference+on+Computer+Vision+and+Pattern+)
20. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
21. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Findings of the Association for Computati…
22. Junmo Kang, Wei Xu, and Alan Ritter. 2023. Distill or annotate? cost-efficient fine-tuning of compact models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11100–11119, Toronto, Canada. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Junmo+Kang%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2023%2E+Distill+or+annotate%3F+cost%2Defficient+fine%2Dtuning+of+compact+models%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics)
23. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
24. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2011%2E+Computing+krippendorff%E2%80%99s+alpha%2Dreliability%2E)
25. Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward design with language models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Minae+Kwon%2C+Sang+Michael+Xie%2C+Kalesha+Bullard%2C+and+Dorsa+Sadigh%2E+2023%2E+Reward+design+with+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
26. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
27. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. [Deep reinforcement learning for dialogue generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202, Austin, Texas. Association for Computational Ling…
28. Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jingu Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. 2022. Explanations from large language models make small reasoners better. ArXiv, abs/2210.06726. [find](https://scholar.google.com/scholar?q=Shiyang+Li%2C+Jianshu+Chen%2C+Yelong+Shen%2C+Zhiyu+Chen%2C+Xinlu+Zhang%2C+Zekun+Li%2C+Hong+Wang%2C+Jingu+Qian%2C+Baolin+Peng%2C+Yi+Mao%2C+Wenhu+Chen%2C+and+Xifeng+Yan%2E+2022%2E+Explanations+from+large+language+models+make+sma)
29. Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura… [find](https://scholar.google.com/scholar?q=Siyao+Li%2C+Deren+Lei%2C+Pengda+Qin%2C+and+William+Yang+Wang%2E+2019%2E+Deep+reinforcement+learning+with+distributional+semantic+rewards+for+abstractive+summarization%2E+In+Proceedings+of+the+2019+Conference+on+E)
30. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hu… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
31. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comp)
32. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
33. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
34. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023a%2E+G%2Deval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+)
35. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association for… [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+Proceedings+of+The+20th+SIGNLL+Con)
36. Yixin Liu, Alexander R Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summarization. In Findings of the Association… [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Alexander+R+Fabbri%2C+Jiawen+Chen%2C+Yilun+Zhao%2C+Simeng+Han%2C+Shafiq+Joty%2C+Pengfei+Liu%2C+Dragomir+Radev%2C+Chien%2DSheng+Wu%2C+and+Arman+Cohan%2E+2024%2E+Benchmarking+generation+and+evaluation+capabilities)
37. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
38. Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq R. Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir R. Radev. 2023b. [Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/). In Proceedings of the 61th Annual…
39. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
40. Yixin Liu, Alexander R Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023c. Towards interpretable and efficient automatic reference-based summarization evaluation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin… [find](https://scholar.google.com/scholar?q=Yixin+Liu%2C+Alexander+R+Fabbri%2C+Yilun+Zhao%2C+Pengfei+Liu%2C+Shafiq+Joty%2C+Chien%2DSheng+Wu%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+2023c%2E+Towards+interpretable+and+efficient+automatic+reference%2Dbased+summarizati)
41. Yixin Liu and Pengfei Liu. 2021. [SimCLS: A simple framework for contrastive learning of abstractive summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-simcls-simple-framework-contrastive-learning/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pa…
42. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. [BRIO: Bringing order to abstractive summarization](https://md.research.elamur.ai/free/en/papers/liu-2022-brio-bringing-order-abstractive-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890–2903, Dublin, Ireland. Association for Computational Ling…
43. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [ChatGPT as a factual inconsistency evaluator for text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). ArXiv.
44. [ArXiv, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 20…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
45. Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021. [Contrastive learning for many-to-many multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/pan-2021-contrastive-learning-many-many-multilingual/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V…
46. Richard Yuanzhe Pang and He He. 2021. Text generation by learning from demonstrations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Richard+Yuanzhe+Pang+and+He+He%2E+2021%2E+Text+generation+by+learning+from+demonstrations%2E+In+International+Conference+on+Learning+Representations%2E)
47. Rebecca Passonneau. 2006. [Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation](https://md.research.elamur.ai/free/en/papers/passonneau-2006-measuring-agreement-set-valued-items/). In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
48. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). In International Conference on Learning Representations.
49. Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023. Summarization is (almost) dead. ArXiv, abs/2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) [find](https://scholar.google.com/scholar?q=Xiao+Pu%2C+Mingqi+Gao%2C+and+Xiaojun+Wan%2E+2023%2E+Summarization+is+%28almost%29+dead%2E+ArXiv%2C+abs%2F2309%2E09558%2E+%E2%80%94+%5BarXiv%3A2309%2E09558%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E09558%29)
50. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems.
51. Nazneen Rajani, Nathan Lambert, Sheon Han, Jean Wang, Osvald Nitski, Edward Beeching, and Lewis Tunstall. 2023. Can foundation models label data like humans? Hugging Face Blog. Https://huggingface.co/blog/llm-v-human-data. [find](https://scholar.google.com/scholar?q=Nazneen+Rajani%2C+Nathan+Lambert%2C+Sheon+Han%2C+Jean+Wang%2C+Osvald+Nitski%2C+Edward+Beeching%2C+and+Lewis+Tunstall%2E+2023%2E+Can+foundation+models+label+data+like+humans%3F+Hugging+Face+Blog%2E+Https%3A%2F%2Fhuggingface%2Eco%2F)
52. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. [Sequence level training with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
53. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. [Minimum risk training for neural machine translation](https://md.research.elamur.ai/free/en/papers/shen-2015-minimum-risk-training-neural-machine/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683–1692, Berlin, Germany. Associatio…
54. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. [Distilling reasoning capabilities into smaller language models](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, Toronto, Canada. Association for Computational Linguistics.
55. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, pages 3008–3021. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+2020%2E+Learning+to+summarize+with+human+feedback%2E+In+Advances+in+Neu)
56. Shichao Sun and Wenjie Li. 2021. Alleviating exposure bias via contrastive learning for abstractive text summarization. CoRR, abs/2108.11846. — [arXiv:2108.11846](https://arxiv.org/abs/2108.11846) [find](https://scholar.google.com/scholar?q=Shichao+Sun+and+Wenjie+Li%2E+2021%2E+Alleviating+exposure+bias+via+contrastive+learning+for+abstractive+text+summarization%2E+CoRR%2C+abs%2F2108%2E11846%2E+%E2%80%94+%5BarXiv%3A2108%2E11846%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2108%2E11846%29)
57. Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. [Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/). In Proceedings of the 61st Annual Meeting of the Association fo…
58. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
59. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is ChatGPT a good NLG evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Ling…
60. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
61. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. [Want to reduce labeling cost? GPT-3 can help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.
62. John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. [Beyond BLEU:training neural machine translation with semantic similarity](https://md.research.elamur.ai/free/en/papers/wieting-2019-beyond-bleu-training-neural-machine/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344– 4355, Florence, Italy. Association for Compu…
63. Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Comput., 1(2):270–280. [find](https://scholar.google.com/scholar?q=Ronald+J%2E+Williams+and+David+Zipser%2E+1989%2E+A+learning+algorithm+for+continually+running+fully+recurrent+neural+networks%2E+Neural+Comput%2E%2C+1%282%29%3A270%E2%80%93280%2E)
64. Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296–1306, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sam+Wiseman+and+Alexander+M%2E+Rush%2E+2016%2E+Sequence%2Dto%2Dsequence+learning+as+beam%2Dsearch+optimization%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+129)
65. Zonghan Yang, Yong Cheng, Yang Liu, and Maosong Sun. 2019. Reducing word omission errors in neural machine translation: A contrastive learning approach. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6191– 6196, Florence, Italy. Association for Comp… [find](https://scholar.google.com/scholar?q=Zonghan+Yang%2C+Yong+Cheng%2C+Yang+Liu%2C+and+Maosong+Sun%2E+2019%2E+Reducing+word+omission+errors+in+neural+machine+translation%3A+A+contrastive+learning+approach%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+th)
66. Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Hongyi+Yuan%2C+Zheng+Yuan%2C+Chuanqi+Tan%2C+Wei+Wang%2C+Songfang+Huang%2C+and+Fei+Huang%2E+2023%2E+RRHF%3A+Rank+responses+to+align+language+models+with+human+feedback%2E+In+Thirty%2Dseventh+Conference+on+Neural+Informati)
67. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2A%2C+Varsha+Kishore%2A%2C+Felix+Wu%2A%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
68. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. [Benchmarking Large Language Models for News Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 12:39–57.
69. Xingxing Zhang, Yiran Liu, Xun Wang, Pengcheng He, Yang Yu, Si-Qing Chen, Wayne Xiong, and Furu Wei. 2022. [Momentum calibration for text generation](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). ArXiv, abs/2212.04257. and Eq. 8 is changed to L̂ ctr (θ) = Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023a…
70. Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J Liu. 2023b. Calibrating sequence likelihood improves conditional language generation. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yao+Zhao%2C+Mikhail+Khalman%2C+Rishabh+Joshi%2C+Shashi+Narayan%2C+Mohammad+Saleh%2C+and+Peter+J+Liu%2E+2023b%2E+Calibrating+sequence+likelihood+improves+conditional+language+generation%2E+In+The+Eleventh+Internationa)
71. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+LLM%2Das%2Da%2D)

*Updated 24 October 2026*