---
title: "TrueTeacher — paper by Gekhman et al. (2023)"
description: "Checking whether a generated summary is faithful to its source document is usually done with NLI… 60 references, cited by 74 papers here."
authors: ["Zorik Gekhman", "Jonathan Herzig", "Roee Aharoni", "Chen Elkind", "Idan Szpektor"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.127/"
doi: "10.18653/v1/2023.emnlp-main.127"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models

# TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models

**Authors:** [Zorik Gekhman](https://md.research.elamur.ai/free/en/authors/zorik-gekhman/), [Jonathan Herzig](https://md.research.elamur.ai/free/en/authors/jonathan-herzig/), [Roee Aharoni](https://md.research.elamur.ai/free/en/authors/roee-aharoni/), Chen Elkind and [Idan Szpektor](https://md.research.elamur.ai/free/en/authors/idan-szpektor/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.127/) · [PDF](https://aclanthology.org/2023.emnlp-main.127.pdf) · [DOI 10.18653/v1/2023.emnlp-main.127](https://doi.org/10.18653/v1/2023.emnlp-main.127) · [OpenAlex](https://openalex.org/W4389519430)

**Citations:** 32 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Checking whether a generated summary is faithful to its source document is usually done with NLI classifiers. These classifiers struggle with document–summary pairs, and their synthetic training data is typically built by corrupting human-written summaries, which covers few error types and does not resemble real system output. TrueTeacher takes a different route. The authors fine-tuned T5 models of five sizes (small to 11B) on XSum and used them to summarize CNN/DailyMail articles. FLAN-PaLM 540B, prompted zero-shot, then labeled each summary as consistent or inconsistent. The result is about 1.4M labeled pairs, used to train a compact student classifier. Because no gold summaries are needed, any document collection can be used, and a multilingual LLM extends the method beyond English. On the summarization portion of the TRUE benchmark, adding this data to NLI training clearly beats the previous best model of similar size and even the much larger teacher. The method also holds up better under domain shift than perturbation-based approaches. Both the dataset and a model checkpoint are publicly released.

## Key points
- The dataset has roughly 1.4M examples: 907,899 consistent and 475,563 inconsistent. Smaller T5 summarizers produced far more inconsistent outputs than T5-11B.
- On TRUE (MNBM, FRANK, SummEval, QAGS-X, QAGS-C), ROC-AUC rose from 82.7 to 87.8 at comparable capacity. The student also outperforms FLAN-PaLM 540B, a teacher about 50× larger.
- Multilingual tests used mFACE (45 languages) with an mT5-XXL baseline trained on ANLI and XNLI:
  - average ROC-AUC was 71.6 for the baseline;
  - adding English-only synthetic data raised it to 73.8;
  - adding multilingual TrueTeacher data from WikiLingua (en, fr, es, de) raised it to 75.3, with gains in 35 of 45 languages.
- Filtering labels via LLM self-verification made the labels more accurate but did not improve the student model.
- Limitations include label noise, the heavy compute cost of a 540B labeler, and little exploration of low-resource languages in the synthetic data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations
- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 28 works · 4,941 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind and Idan Szpektor. 2023. TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gekhman2023trueteacher,
  title = {{TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models}},
  author = {Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind and Idan Szpektor},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.127},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.127}
}
```

- [← previous SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval](https://md.research.elamur.ai/free/en/papers/wang-2023-simlm-pre-training-representation-bottleneck/)
- [next → Complementary Explanations for Effective In-Context Learning](https://md.research.elamur.ai/free/en/papers/ye-2023-complementary-explanations-effective-context-learning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

60 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2022. Qameleon: Multilingual QA with only 5 examples. CoRR, abs/2211.08264. [find](https://scholar.google.com/scholar?q=Priyanka+Agrawal%2C+Chris+Alberti%2C+Fantine+Huot%2C+Joshua+Maynez%2C+Ji+Ma%2C+Sebastian+Ruder%2C+Kuzman+Ganchev%2C+Dipanjan+Das%2C+and+Mirella+Lapata%2E+2022%2E+Qameleon%3A+Multilingual+QA+with+only+5+examples%2E+CoRR%2C+abs%2F)
2. Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2022. mface: Multilingual summarization with factual consistency evaluation. CoRR, abs/2212.10622. [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Shashi+Narayan%2C+Joshua+Maynez%2C+Jonathan+Herzig%2C+Elizabeth+Clark%2C+and+Mirella+Lapata%2E+2022%2E+mface%3A+Multilingual+summarization+with+factual+consistency+evaluation%2E+CoRR%2C+abs%2F2212%2E10622%2E)
3. Rachith Aiyappa, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2023. Can we trust the evaluation on chatgpt? CoRR, abs/2303.12767. [find](https://scholar.google.com/scholar?q=Rachith+Aiyappa%2C+Jisun+An%2C+Haewoon+Kwak%2C+and+Yong%2DYeol+Ahn%2E+2023%2E+Can+we+trust+the+evaluation+on+chatgpt%3F+CoRR%2C+abs%2F2303%2E12767%2E)
4. Vidhisha Balachandran, Hannaneh Hajishirzi, William W. Cohen, and Yulia Tsvetkov. 2022. Correcting diverse factual errors in abstractive summarization via post-editing and language model infilling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022,… [find](https://scholar.google.com/scholar?q=Vidhisha+Balachandran%2C+Hannaneh+Hajishirzi%2C+William+W%2E+Cohen%2C+and+Yulia+Tsvetkov%2E+2022%2E+Correcting+diverse+factual+errors+in+abstractive+summarization+via+post%2Dediting+and+language+model+infilling%2E+In)
5. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. 2007. [Open information extraction from the web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/). In IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007, pages 2670–2676.
6. Yonatan Bitton, Shlomi Cohen-Ganor, Ido Hakimi, Yoad Lewenberg, Roee Aharoni, and Enav Weinreb. 2023. q2d: Turning questions into dialogs to teach models how to search. [find](https://scholar.google.com/scholar?q=Yonatan+Bitton%2C+Shlomi+Cohen%2DGanor%2C+Ido+Hakimi%2C+Yoad+Lewenberg%2C+Roee+Aharoni%2C+and+Enav+Weinreb%2E+2023%2E+q2d%3A+Turning+questions+into+dialogs+to+teach+models+how+to+search%2E)
7. Shiqi Chen, Siyang Gao, and Junxian He. 2023. Evaluating factual consistency of summaries with large language models. CoRR, abs/2305.14069. — [arXiv:2305.14069](https://arxiv.org/abs/2305.14069) [find](https://scholar.google.com/scholar?q=Shiqi+Chen%2C+Siyang+Gao%2C+and+Junxian+He%2E+2023%2E+Evaluating+factual+consistency+of+summaries+with+large+language+models%2E+CoRR%2C+abs%2F2305%2E14069%2E+%E2%80%94+%5BarXiv%3A2305%2E14069%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E14069%29)
8. [Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi…](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
9. [Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang…](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
10. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orlean…
11. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+R%2E+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+)
12. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. A… [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+2020%2E+Don%E2%80%99t+stop+pretraining%3A+Adapt+language+models+to+domains+and+tasks%2E+In+Proceedings+of+)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
14. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq R. Joty, and Boyang Li. 2022. [Is GPT-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/) CoRR, abs/2212.10450.
15. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2020%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+arXiv+preprint+arXiv%3A2007%2E12626%2E)
16. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019a. [Ranking generated summaries by correctness: An interesting but challenging application for natural language inference](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/). In Proceedings of the 57th Conference of the Association for Computational Linguisti…
17. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019b. [Ranking generated summaries by correctness: An interesting but challenging application for natural language inference](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/). In Proceedings of the 57th Annual Meeting of the Association for Computational Lingu…
18. Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 20… [find](https://scholar.google.com/scholar?q=Tahmid+Hasan%2C+Abhik+Bhattacharjee%2C+Md%2E+Saiful+Islam%2C+Kazi+Samin+Mubasshir%2C+Yuan%2DFang+Li%2C+Yong%2DBin+Kang%2C+M%2E+Sohel+Rahman%2C+and+Rifat+Shahriyar%2E+2021%2E+Xl%2Dsum%3A+Large%2Dscale+multilingual+abstractive+summari)
19. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-…
20. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
21. Ben Goodrich, Vinay Rao, Mohammad Saleh, and Peter J. Liu. 2019. Assessing the factual accuracy of generated text. CoRR, abs/1905.13322. — [arXiv:1905.13322](https://arxiv.org/abs/1905.13322) [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Mohammad+Saleh%2C+and+Peter+J%2E+Liu%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+CoRR%2C+abs%2F1905%2E13322%2E+%E2%80%94+%5BarXiv%3A1905%2E13322%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1905%2E13322%29)
22. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021. $qˆ2$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing… [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Leshem+Choshen%2C+Roee+Aharoni%2C+Ella+Neeman%2C+Idan+Szpektor%2C+and+Omri+Abend%2E+2021%2E+%24q%CB%862%24%3A+Evaluating+factual+consistency+in+knowledge%2Dgrounded+dialogues+via+question+generation+and+question+)
23. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of GPT-3. CoRR, abs/2209.12356. [find](https://scholar.google.com/scholar?q=Tanya+Goyal%2C+Junyi+Jessy+Li%2C+and+Greg+Durrett%2E+2022%2E+News+summarization+and+evaluation+in+the+era+of+GPT%2D3%2E+CoRR%2C+abs%2F2209%2E12356%2E)
24. Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. 2020. [BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual…
25. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/). In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003– 8017, Toronto, Canada. Association for Computational Linguistics.
26. Jie Huang and Kevin Chen-Chuan Chang. 2022. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). CoRR, abs/2212.10403.
27. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th A… [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+Proceedings+of+the+Thirty%2DSecond+AAAI+Conference+on+Artificial+Intellige)
28. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. CoRR, abs/2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+CoRR%2C+abs%2F2302%2E14520%2E)
29. Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2023. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Comput. Surv., 55(8):154:1–154:35. [find](https://scholar.google.com/scholar?q=Huan+Yee+Koh%2C+Jiaxin+Ju%2C+Ming+Liu%2C+and+Shirui+Pan%2E+2023%2E+An+empirical+survey+on+long+document+summarization%3A+Datasets%2C+models%2C+and+metrics%2E+ACM+Comput%2E+Surv%2E%2C+55%288%29%3A154%3A1%E2%80%93154%3A35%2E)
30. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In NeurIPS.
31. Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019. [Neural text summarization: A critical evaluation](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
32. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9332– 9346. Ass… [find](https://scholar.google.com/scholar?q=Wojciech+Kryscinski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2020%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Met)
33. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. Summac: Re-visiting nli-based models for inconsistency detection in summarization. Trans. Assoc. Comput. Linguistics, 10:163– 177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N%2E+Bennett%2C+and+Marti+A%2E+Hearst%2E+2022%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Trans%2E+Assoc%2E+Comput%2E+Linguistics%2C+10%3A16)
34. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. [WANLI: worker and AI collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6826–6847. Ass…
35. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using GPT-4 with better human alignment. CoRR, abs/2303.16634. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+NLG+evaluation+using+GPT%2D4+with+better+human+alignment%2E+CoRR%2C+abs%2F2303%2E16634%2E)
36. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for abstractive text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). CoRR, abs/2303.15621.
37. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoR… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Sean+Welleck%2C+Bodhisattwa+Prasad+Majumder%2C+Shashank+Gupta%2C)
38. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1906–1919. Association for Comput… [find](https://scholar.google.com/scholar?q=Joshua+Maynez%2C+Shashi+Narayan%2C+Bernd+Bohnet%2C+and+Ryan+T%2E+McDonald%2E+2020%2E+On+faithfulness+and+factuality+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+C)
39. Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021. Looking beyond sentence-level natural language inference for question answering and text summarization. In Proceedings of the 2021 Conference of the North American Chapter of the… [find](https://scholar.google.com/scholar?q=Anshuman+Mishra%2C+Dhruvesh+Patel%2C+Aparna+Vijayakumar%2C+Xiang+Lorraine+Li%2C+Pavan+Kapanipathi%2C+and+Kartik+Talamadupula%2E+2021%2E+Looking+beyond+sentence%2Dlevel+natural+language+inference+for+question+answerin)
40. Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen R. McKeown. 2020. [Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization](https://md.research.elamur.ai/free/en/papers/ladhak-2020-wikilingua-new-benchmark-dataset-cross/). CoRR, abs/2010.03093.
41. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -Novembe… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
42. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E)
43. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 4885–49… [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
44. OpenAI. 2022. [https://openai.com/blog/chatgpt/](https://openai.com/blog/chatgpt/). [↗](https://openai.com/blog/chatgpt/)
45. Chatgpt, Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Huma…
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
47. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021. [Measuring attribution in natural language generation models](https://md.research.elamur.ai/free/en/papers/rashkin-2023-measuring-attribution-natural-language-generation/). CoRR, abs/2112.12870.
48. Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages… [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Sihao+Chen%2C+Senaka+Buthpitiya%2C+Alex+Fabrikant%2C+and+Donald+Metzler%2E+2022%2E+Stretching+sentence%2Dpair+NLI+models+to+reason+over+long+documents+and+clusters%2E+In+Findings+of+the+Association+fo)
49. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. [Questeval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Eve…
50. Amir Soleimani, Christof Monz, and Marcel Worring. 2023. NonFactS: NonFactual summary generation for factuality evaluation in document summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6405–6419, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Amir+Soleimani%2C+Christof+Monz%2C+and+Marcel+Worring%2E+2023%2E+NonFactS%3A+NonFactual+summary+generation+for+factuality+evaluation+in+document+summarization%2E+In+Findings+of+the+Association+for+Computational+L)
51. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
52. Prasetya Utama, Joshua Bambrick, Nafise Sadat Moosavi, and Iryna Gurevych. 2022. Falsesum: Generating document-level NLI examples for recognizing factual inconsistency in summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Prasetya+Utama%2C+Joshua+Bambrick%2C+Nafise+Sadat+Moosavi%2C+and+Iryna+Gurevych%2E+2022%2E+Falsesum%3A+Generating+document%2Dlevel+NLI+examples+for+recognizing+factual+inconsistency+in+summarization%2E+In+Proceedings)
53. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5008–5020. Association for Computation… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
54. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good NLG evaluator? A preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). CoRR, abs/2303.04048.
55. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. [Want to reduce labeling cost? GPT-3 can help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/). In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 4195–4205. Association for C…
56. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. 2023. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). CoRR, abs/2212.09561.
57. Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. [Symbolic knowledge distillation: from general language models to commonsense models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). In Proceedings of the 2022 Conference of the North American Chapter of the Associa…
58. Wenhao Wu, Wei Li, Xinyan Xiao, Jiachen Liu, Sujian Li, and Yajuan Lv. 2023. Wecheck: Strong factual consistency checker via weakly supervised learning. Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, ACL 2023. [find](https://scholar.google.com/scholar?q=Wenhao+Wu%2C+Wei+Li%2C+Xinyan+Xiao%2C+Jiachen+Liu%2C+Sujian+Li%2C+and+Yajuan+Lv%2E+2023%2E+Wecheck%3A+Strong+factual+consistency+checker+via+weakly+supervised+learning%2E+Proceedings+of+the+61th+Annual+Meeting+of+the+A)
59. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
60. Wenpeng Yin, Dragomir R. Radev, and Caiming Xiong. 2021. [Docnli: A large-scale dataset for document-level natural language inference](https://md.research.elamur.ai/free/en/papers/yin-2021-docnli-large-scale-dataset-document/). In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 4913–4…

*Updated 24 October 2026*