--- title: "Discourse-Driven Evaluation — paper by Zhong and Litman (2025)" description: "Based on these findings, they propose StructScore, which does two things: It reweights sentence-level… 63 references, cited by 7 papers here." authors: ["Yang Zhong", "Diane Litman"] year: 2025 venue: "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)" source: "https://aclanthology.org/2025.naacl-long.103/" doi: "10.18653/v1/2025.naacl-long.103" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhong-2025-discourse-driven-evaluation-unveiling-factual/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2025-discourse-driven-evaluation-unveiling-factual/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization # Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization **Authors:** [Yang Zhong](https://md.research.elamur.ai/free/en/authors/yang-zhong/) and [Diane Litman](https://md.research.elamur.ai/free/en/authors/diane-litman/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2025](https://md.research.elamur.ai/free/en/venues/naacl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.naacl-long.103/) · [PDF](https://aclanthology.org/2025.naacl-long.103.pdf) · [DOI 10.18653/v1/2025.naacl-long.103](https://doi.org/10.18653/v1/2025.naacl-long.103) · [OpenAlex](https://openalex.org/W4411120463) **Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/zhong-2025-discourse-driven-evaluation-unveiling-factual/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Checking whether a generated summary stays faithful to its source is especially hard for long documents. NLI-based detectors like AlignScore split the source into fixed chunks, which can cut across sections and paragraphs. They also combine sentence scores by plain averaging or by taking the minimum, as if every summary sentence mattered equally. The authors use Rhetorical Structure Theory (RST) to study this problem. They parse summaries and sources with the DMRST parser and analyze sentence-level error annotations from DiverSumm. They find that summary sentences made of many EDUs are more error-prone, and that several discourse features correlate with factual errors. Based on these findings, they propose StructScore, which does two things: - It reweights sentence-level NLI scores using discourse information such as nuclearity. - It segments long sources into chunks that follow the discourse structure. StructScore improves several baseline models on benchmarks in science, government and legal text. The extra cost is small, coming only from running the parser. The work shows that linguistic structure helps long-document factuality evaluation and makes the scores easier to interpret. ## Key points - Analysis data: DiverSumm-sent (ArXiv and GovReport subsets of DiverSumm), with 293 document–summary pairs and 3138 summary sentences annotated at sentence level. - Evaluation benchmarks: AggreFact-FT SOTA split, DiverSumm, LongSciVerify, LongEval, and a new LegalSumm dataset. LegalSumm contains CanLII legal-opinion summaries with factuality labels from legal experts. - Discourse parsing uses the open-source DMRST parser (Liu et al., 2021). It runs on CPU or GPU and processes the full DiverSumm test set within minutes. - On LegalSumm, StructScore improves the baselines and beats strong LLM-based checkers such as BeSpoke-MC-7B by a wide margin. - Code and model outputs are released at github.com/cs329yangzhong/discourse-driven-summary-factuality-evaluation. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Pittsburgh](https://md.research.elamur.ai/free/en/universities/university-of-pittsburgh/) — University, Pittsburgh, United States · 59 works · 22,324 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (3): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 466 works · [📝 Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · 232 works ## Cite Cite as: Yang Zhong and Diane Litman. 2025. Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{zhong2025discourse, title = {{Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization}}, author = {Yang Zhong and Diane Litman}, booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, year = {2025}, doi = {10.18653/v1/2025.naacl-long.103}, url = {https://doi.org/10.18653/v1/2025.naacl-long.103} } ``` - [← previous DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning](https://md.research.elamur.ai/free/en/papers/wu-2025-dimo-gui-advancing-test-time/) - [next → Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly](https://md.research.elamur.ai/free/en/papers/hosseini-2025-efficient-solutions-intriguing-failure-llms/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) ## References 63 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Griffin Adams, Alex Fabbri, Faisal Ladhak, Noémie Elhadad, and Kathleen McKeown. 2023a. Generating EDU extracts for plan-guided summary re-ranking. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2680–2697, Toronto, Canada. As… [find](https://scholar.google.com/scholar?q=Griffin+Adams%2C+Alex+Fabbri%2C+Faisal+Ladhak%2C+No%C3%A9mie+Elhadad%2C+and+Kathleen+McKeown%2E+2023a%2E+Generating+EDU+extracts+for+plan%2Dguided+summary+re%2Dranking%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Ass) 2. Griffin Adams, Bichlien Nguyen, Jake Smith, Yingce Xia, Shufang Xie, Anna Ostropolets, Budhaditya Deb, Yuan-Jyue Chen, Tristan Naumann, and Noémie Elhadad. 2023b. What are the desired characteristics of calibration sets? identifying correlates on long form scientific summarization. In Proceedings of… [find](https://scholar.google.com/scholar?q=Griffin+Adams%2C+Bichlien+Nguyen%2C+Jake+Smith%2C+Yingce+Xia%2C+Shufang+Xie%2C+Anna+Ostropolets%2C+Budhaditya+Deb%2C+Yuan%2DJyue+Chen%2C+Tristan+Naumann%2C+and+No%C3%A9mie+Elhadad%2E+2023b%2E+What+are+the+desired+characteristics+) 3. Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv preprint arXiv:2004.05150. 4. Jennifer A. Bishop, Sophia Ananiadou, and Qianqian Xie. 2024. LongDocFACTScore: Evaluating the factuality of long document abstractive summarisation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 10… [find](https://scholar.google.com/scholar?q=Jennifer+A%2E+Bishop%2C+Sophia+Ananiadou%2C+and+Qianqian+Xie%2E+2024%2E+LongDocFACTScore%3A+Evaluating+the+factuality+of+long+document+abstractive+summarisation%2E+In+Proceedings+of+the+2024+Joint+International+Con) 5. Volume 2 (Short Papers), pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Volume+2+%28Short+Papers%29%2C+pages+615%E2%80%93621%2C+New+Orleans%2C+Louisiana%2E+Association+for+Computational+Linguistics%2E) 6. Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjia… — [InternLM2 Technical Report](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/) 7. Mohamed Elaraby and Diane Litman. 2022. ArgLegal-Summ: Improving abstractive summarization of legal documents with argument mining. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6187–6194, Gyeongju, Republic of Korea. International Committee on Computational… [find](https://scholar.google.com/scholar?q=Mohamed+Elaraby+and+Diane+Litman%2E+2022%2E+ArgLegal%2DSumm%3A+Improving+abstractive+summarization+of+legal+documents+with+argument+mining%2E+In+Proceedings+of+the+29th+International+Conference+on+Computational) 8. Meng Cao, Yue Dong, and Jackie Cheung. 2022. [Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization](https://md.research.elamur.ai/free/en/papers/cao-2022-hallucinated-but-factual-inspecting-factuality/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340–3354, Dublin, Ireland. Assoc… 9. Shuyang Cao and Lu Wang. 2021. [CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Co… 10. Lynn Carlson, Mary Ellen Okurowski, and Daniel Marcu. 2002. Rst discourse treebank. Technical report, Linguistic Data Consortium, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Lynn+Carlson%2C+Mary+Ellen+Okurowski%2C+and+Daniel+Marcu%2E+2002%2E+Rst+discourse+treebank%2E+Technical+report%2C+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2E) 11. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yapei+Chang%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2024%2E+Booookscore%3A+A+systematic+exploration+of+book%2Dlength+summarization+in+the+era+of+LLMs%2E+In+The+Twelfth+International+Conference+on+Learning+Rep) 12. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. [A discourse-aware attention model for abstractive summarization of long documents](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… 13. Mohamed Elaraby, Yang Zhong, and Diane Litman. 2023. Towards argument-aware abstractive summarization of long legal opinions with summary reranking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7601–7612, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mohamed+Elaraby%2C+Yang+Zhong%2C+and+Diane+Litman%2E+2023%2E+Towards+argument%2Daware+abstractive+summarization+of+long+legal+opinions+with+summary+reranking%2E+In+Findings+of+the+Association+for+Computational+Li) 14. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. [Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model](https://md.research.elamur.ai/free/en/papers/fabbri-2019-multi-news-large-scale-multi/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy.… 15. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. [QAFactEval: Improved QA-based factual consistency evaluation for summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… 16. John Glover, Federico Fancellu, Vasudevan Jagannathan, Matthew R. Gormley, and Thomas Schaaf. 2022. Revisiting text decomposition methods for NLI-based factuality scoring of summaries. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 97–105, Abu… [find](https://scholar.google.com/scholar?q=John+Glover%2C+Federico+Fancellu%2C+Vasudevan+Jagannathan%2C+Matthew+R%2E+Gormley%2C+and+Thomas+Schaaf%2E+2022%2E+Revisiting+text+decomposition+methods+for+NLI%2Dbased+factuality+scoring+of+summaries%2E+In+Proceedings+) 17. Tanya Goyal and Greg Durrett. 2021. [Annotating and modeling fine-grained factuality in summarization](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1449–1462, Online. Association for Computatio… 18. Freya Hewett, Roshan Prakash Rane, Nina Harlacher, and Manfred Stede. 2019. [The utility of discourse parsing features for predicting argumentation structure](https://md.research.elamur.ai/free/en/papers/hewett-2019-utility-discourse-parsing-features-predicting/). In Proceedings of the 6th Workshop on Argument Mining, pages 98–103, Florence, Italy. Association for Computational Linguistics. 19. Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5449–5474, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+5449%E2%80%935474%2C+Bangkok%2C+Thailand%2E+Association+for+Computational+Linguistics%2E) 20. Freya Hewett and Manfred Stede. 2022. Extractive summarisation for German-language data: A text-level approach with discourse features. In Proceedings of the 29th International Conference on Computational Linguistics, pages 756–765, Gyeongju, Republic of Korea. International Committee on Computation… [find](https://scholar.google.com/scholar?q=Freya+Hewett+and+Manfred+Stede%2E+2022%2E+Extractive+summarisation+for+German%2Dlanguage+data%3A+A+text%2Dlevel+approach+with+discourse+features%2E+In+Proceedings+of+the+29th+International+Conference+on+Computati) 21. Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022a. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Computing Surveys, 55:1 – 35. [find](https://scholar.google.com/scholar?q=Huan+Yee+Koh%2C+Jiaxin+Ju%2C+Ming+Liu%2C+and+Shirui+Pan%2E+2022a%2E+An+empirical+survey+on+long+document+summarization%3A+Datasets%2C+models%2C+and+metrics%2E+ACM+Computing+Surveys%2C+55%3A1+%E2%80%93+35%2E) 22. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. [TRUE: Re-evaluating factual consistency evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/). In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Co… 23. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1419–1436, Online.… 24. Patrick Huber, Wen Xiao, and Giuseppe Carenini. 2021. W-RST: Towards a weighted RST-style discourse framework. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)… [find](https://scholar.google.com/scholar?q=Patrick+Huber%2C+Wen+Xiao%2C+and+Giuseppe+Carenini%2E+2021%2E+W%2DRST%3A+Towards+a+weighted+RST%2Dstyle+discourse+framework%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistic) 25. Shafiq Joty, Francisco Guzmán, Lluís Màrquez, and Preslav Nakov. 2017. [Discourse structure in machine translation evaluation](https://md.research.elamur.ai/free/en/papers/joty-2017-discourse-structure-machine-translation-evaluation/). Computational Linguistics, 43(4):683–722. 26. Yuta Kikuchi, Tsutomu Hirao, Hiroya Takamura, Manabu Okumura, and Masaaki Nagata. 2014. Single document summarization based on nested tree structure. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 315–320, Baltimore, Marylan… [find](https://scholar.google.com/scholar?q=Yuta+Kikuchi%2C+Tsutomu+Hirao%2C+Hiroya+Takamura%2C+Manabu+Okumura%2C+and+Masaaki+Nagata%2E+2014%2E+Single+document+summarization+based+on+nested+tree+structure%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+A) 27. Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024a. Fables: Evaluating faithfulness and content selection in book-length summarization. First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Yekyung+Kim%2C+Yapei+Chang%2C+Marzena+Karpinska%2C+Aparna+Garimella%2C+Varun+Manjunatha%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2024a%2E+Fables%3A+Evaluating+faithfulness+and+content+selection+in+book%2Dlength+summ) 28. Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, and Dongyeop Kang. 2024b. Threads of subtlety: Detecting machine-generated texts through discourse motifs. In Proceedings of the 62nd Annual Huan Yee Koh, Jiaxin Ju, He Zhang, Ming Liu, and Shirui Pan. 2022b. How far are we from robust long abstra… [find](https://scholar.google.com/scholar?q=Zae+Myung+Kim%2C+Kwang+Lee%2C+Preston+Zhu%2C+Vipul+Raheja%2C+and+Dongyeop+Kang%2E+2024b%2E+Threads+of+subtlety%3A+Detecting+machine%2Dgenerated+texts+through+discourse+motifs%2E+In+Proceedings+of+the+62nd+Annual+Huan+Y) 29. Mathias Kraus and Stefan Feuerriegel. 2019. Sentiment analysis based on rhetorical structure theory:learning deep neural networks from discourse trees. Expert Systems with Applications, 118:65–79. [find](https://scholar.google.com/scholar?q=Mathias+Kraus+and+Stefan+Feuerriegel%2E+2019%2E+Sentiment+analysis+based+on+rhetorical+structure+theory%3Alearning+deep+neural+networks+from+discourse+trees%2E+Expert+Systems+with+Applications%2C+118%3A65%E2%80%9379%2E) 30. Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. [LongEval: Guidelines for human evaluation of faithfulness in long-form summarization](https://md.research.elamur.ai/free/en/papers/krishna-2023-longeval-guidelines-human-evaluation-faithfulness/). In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisti… 31. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. [Evaluating the factual consistency of abstractive text summarization](https://md.research.elamur.ai/free/en/papers/kryscinski-2020-evaluating-factual-consistency-abstractive-text/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational… 32. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N%2E+Bennett%2C+and+Marti+A%2E+Hearst%2E+2022%2E+SummaC%3A+Re%2DVisiting+NLI%2Dbased+Models+for+Inconsistency+Detection+in+Summarization%2E+Transactions+of+the+Association+for+Comp) 33. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173. 34. Yinhong Liu, Yixuan Su, Ehsan Shareghi, and Nigel Collier. 2024b. Unlocking structure measuring: Introducing PDD, an automatic metric for positional discourse coherence. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Yixuan+Su%2C+Ehsan+Shareghi%2C+and+Nigel+Collier%2E+2024b%2E+Unlocking+structure+measuring%3A+Introducing+PDD%2C+an+automatic+metric+for+positional+discourse+coherence%2E+In+Proceedings+of+the+2024+Con) 35. Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021. DMRST: A joint framework for document-level multilingual RST discourse segmentation and parsing. [find](https://scholar.google.com/scholar?q=Zhengyuan+Liu%2C+Ke+Shi%2C+and+Nancy+Chen%2E+2021%2E+DMRST%3A+A+joint+framework+for+document%2Dlevel+multilingual+RST+discourse+segmentation+and+parsing%2E) 36. In Proceedings of the 2nd Workshop on Computational Approaches to Discourse, pages 154–164, Punta Cana, Dominican Republic and Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2nd+Workshop+on+Computational+Approaches+to+Discourse%2C+pages+154%E2%80%93164%2C+Punta+Cana%2C+Dominican+Republic+and+Online%2E+Association+for+Computational+Linguistics%2E) 37. Annie Louis, Aravind Joshi, and Ani Nenkova. 2010. Discourse indicators for content selection in summarization. In Proceedings of the SIGDIAL 2010 Conference, pages 147–156, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Annie+Louis%2C+Aravind+Joshi%2C+and+Ani+Nenkova%2E+2010%2E+Discourse+indicators+for+content+selection+in+summarization%2E+In+Proceedings+of+the+SIGDIAL+2010+Conference%2C+pages+147%E2%80%93156%2C+Tokyo%2C+Japan%2E+Association+) 38. Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation. [find](https://scholar.google.com/scholar?q=Wen+Luo%2C+Tianshu+Shen%2C+Wei+Li%2C+Guangyue+Peng%2C+Richeng+Xuan%2C+Houfeng+Wang%2C+and+Xi+Yang%2E+2024%2E+Halludial%3A+A+large%2Dscale+benchmark+for+automatic+dialogue%2Dlevel+hallucination+evaluation%2E) 39. Aru Maekawa, Tsutomu Hirao, Hidetaka Kamigaito, and Manabu Okumura. 2024. [Can we obtain significant success in RST discourse parsing by using large language models?](https://md.research.elamur.ai/free/en/papers/maekawa-2024-can-we-obtain-significant-success/) In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),… 40. William C. Mann and Sandra A. Thompson. 1988. [Rhetorical structure theory: Toward a functional theory of text organization](https://md.research.elamur.ai/free/en/papers/mann-1988-rhetorical-structure-theory-functional/). Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. 41. Daniel Marcu. 1998. To build text summaries of high quality, nuclearity is not sufficient. Working Notes of the AAAI-98 Spring Symposium on Intelligent Text Summarization. [find](https://scholar.google.com/scholar?q=Daniel+Marcu%2E+1998%2E+To+build+text+summaries+of+high+quality%2C+nuclearity+is+not+sufficient%2E+Working+Notes+of+the+AAAI%2D98+Spring+Symposium+on+Intelligent+Text+Summarization%2E) 42. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. 43. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. 44. Kenji Ono, Kazuo Sumita, and Seiji Miike. 1994. [Abstract generation based on rhetorical structure extraction](https://md.research.elamur.ai/free/en/papers/ono-1994-abstract-generation-based-rhetorical-structure/). In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics, Kyoto, Japan. 45. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 46. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… 47. Andreas Peldszus and Manfred Stede. 2016. An annotated corpus of argumentative microtexts. In Argumentation and Reasoned Action - Proceedings of the 1st European Conference on Argumentation, volume 2, pages 801–816. [find](https://scholar.google.com/scholar?q=Andreas+Peldszus+and+Manfred+Stede%2E+2016%2E+An+annotated+corpus+of+argumentative+microtexts%2E+In+Argumentation+and+Reasoned+Action+%2D+Proceedings+of+the+1st+European+Conference+on+Argumentation%2C+volume+2%2C) 48. Dongqi Pu, Yifan Wang, and Vera Demberg. 2023. Incorporating distributions of discourse structure for long document abstractive summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5574–5590, Toronto, Canada. Associa… [find](https://scholar.google.com/scholar?q=Dongqi+Pu%2C+Yifan+Wang%2C+and+Vera+Demberg%2E+2023%2E+Incorporating+distributions+of+discourse+structure+for+long+document+abstractive+summarization%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Associat) 49. Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 394–412, Abu Dhabi, United Arab Emirates. Association… [find](https://scholar.google.com/scholar?q=Tal+Schuster%2C+Sihao+Chen%2C+Senaka+Buthpitiya%2C+Alex+Fabrikant%2C+and+Donald+Metzler%2E+2022%2E+Stretching+sentence%2Dpair+NLI+models+to+reason+over+long+documents+and+clusters%2E+In+Findings+of+the+Association+fo) 50. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. [QuestEval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online… 51. Alessandro Scirè, Karim Ghonim, and Roberto Navigli. 2024. FENICE: Factuality evaluation of summarization based on natural language inference and claim extraction. In Findings of the Association for Computational Linguistics ACL 2024, pages 14148–14161, Bangkok, Thailand and virtual meeting. Associa… [find](https://scholar.google.com/scholar?q=Alessandro+Scir%C3%A8%2C+Karim+Ghonim%2C+and+Roberto+Navigli%2E+2024%2E+FENICE%3A+Factuality+evaluation+of+summarization+based+on+natural+language+inference+and+claim+extraction%2E+In+Findings+of+the+Association+for+C) 52. Liyan Tang, Tanya Goyal, Alex Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin Rousseau, and Greg Durrett. 2023. [Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors](https://md.research.elamur.ai/free/en/papers/tang-2023-understanding-factual-errors-summarization-errors/). In Proceedings of the 61st Annual Meeting of the Association fo… 53. Liyan Tang, Philippe Laban, and Greg Durrett. 2024. [MiniCheck: Efficient fact-checking of LLMs on grounding documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8818–8847, Miami, Florida, USA. Association for Computational Linguistics. 54. Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2021. Predicting discourse trees from transformer-based neural summarizers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4139–4152, Online. Asso… [find](https://scholar.google.com/scholar?q=Wen+Xiao%2C+Patrick+Huber%2C+and+Giuseppe+Carenini%2E+2021%2E+Predicting+discourse+trees+from+transformer%2Dbased+neural+summarizers%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+A) 55. Huihui Xu, Jaromir Savelka, and Kevin D. Ashley. 2021. Accounting for sentence position and legal domain sentence embedding in learning to classify case sentences. In International Conference on Legal Knowledge and Information Systems. [find](https://scholar.google.com/scholar?q=Huihui+Xu%2C+Jaromir+Savelka%2C+and+Kevin+D%2E+Ashley%2E+2021%2E+Accounting+for+sentence+position+and+legal+domain+sentence+embedding+in+learning+to+classify+case+sentences%2E+In+International+Conference+on+Legal) 56. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. [Discourse-aware neural extractive text summarization](https://md.research.elamur.ai/free/en/papers/xu-2019-discourse-aware-neural-extractive-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021–5031, Online. Association for Computational Linguistics. 57. Joonho Yang, Seunghyun Yoon, ByeongJeong Kim, and Hwanhee Lee. 2024. FIZZ: Factual inconsistency detection by zoom-in summary and zoom-out document. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Miami, Florida, USA. Association for Computatio… [find](https://scholar.google.com/scholar?q=Joonho+Yang%2C+Seunghyun+Yoon%2C+ByeongJeong+Kim%2C+and+Hwanhee+Lee%2E+2024%2E+FIZZ%3A+Factual+inconsistency+detection+by+zoom%2Din+summary+and+zoom%2Dout+document%2E+In+Proceedings+of+the+2024+Conference+on+Empirical+) 58. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [AlignScore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association… 59. Huajian Zhang, Yumo Xu, and Laura Perez-Beltrachini. 2024a. Fine-grained natural language inference based faithfulness evaluation for diverse summarisation tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p… [find](https://scholar.google.com/scholar?q=Huajian+Zhang%2C+Yumo+Xu%2C+and+Laura+Perez%2DBeltrachini%2E+2024a%2E+Fine%2Dgrained+natural+language+inference+based+faithfulness+evaluation+for+diverse+summarisation+tasks%2E+In+Proceedings+of+the+18th+Conference) 60. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2024b. [How language model hallucinations can snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/). In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 59670–59684. PMLR. 61. Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. [QMSum: A new benchmark for query-based multi-domain meeting summarization](https://md.research.elamur.ai/free/en/papers/zhong-2021-qmsum-new-benchmark-query-based/). In Proceedings of the 2021 Conference of the North American Chapte… 62. Yang Zhong, Chao Jiang, Wei Xu, and Junyi Jessy Li. 2020. Discourse level factors for sentence deletion in text simplification. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9709–9716. [find](https://scholar.google.com/scholar?q=Yang+Zhong%2C+Chao+Jiang%2C+Wei+Xu%2C+and+Junyi+Jessy+Li%2E+2020%2E+Discourse+level+factors+for+sentence+deletion+in+text+simplification%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+34%2805%29%3A97) 63. Zhiying Zhu, Yiming Yang, and Zhiqing Sun. 2024. Halueval-wild: Evaluating hallucinations of language models in the wild. [find](https://scholar.google.com/scholar?q=Zhiying+Zhu%2C+Yiming+Yang%2C+and+Zhiqing+Sun%2E+2024%2E+Halueval%2Dwild%3A+Evaluating+hallucinations+of+language+models+in+the+wild%2E) *Updated 31 October 2026*