---
title: "Discursive Socratic Questioning — Miao 2024"
description: "High accuracy on discourse relation classification does not show that a language model actually grasps those relations in a reliable way."
authors: ["Yisong Miao", "Hongfu Liu", "Wenqiang Lei", "Nancy Chen", "Min-Yen Kan"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.341/"
doi: "10.18653/v1/2024.acl-long.341"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/miao-2024-discursive-socratic-questioning-evaluating-faithfulness/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/miao-2024-discursive-socratic-questioning-evaluating-faithfulness/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/2/) / Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations

# Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations

**Authors:** Yisong Miao, Hongfu Liu, [Wenqiang Lei](https://md.research.elamur.ai/free/authors/wenqiang-lei/), [Nancy Chen](https://md.research.elamur.ai/free/authors/nancy-chen/), [Min-Yen Kan](https://md.research.elamur.ai/free/authors/min-yen-kan/) · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.341/) · [PDF](https://aclanthology.org/2024.acl-long.341.pdf) · [DOI 10.18653/v1/2024.acl-long.341](https://doi.org/10.18653/v1/2024.acl-long.341) · [OpenAlex](https://openalex.org/W4402672043)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [15](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
High accuracy on discourse relation classification does not show that a language model actually grasps those relations in a reliable way. The authors propose DiSQ (Discursive Socratic Questioning), which tests a model with a series of yes/no questions instead of a single label prediction.

For each implicit relation in the PDTB-3.0 test sections, the method works in three steps:
- **Extraction:** event-like spans are pulled from both arguments using semantic role labeling.
- **Selection:** LLaMA2-13B, used with in-context learning, picks the span pairs that actually carry the relation.
- **Validation:** human experts check these choices.

From the selected pairs, three kinds of questions are generated. Targeted questions should be affirmed, counterfactual questions should be rejected, and converse rephrasings check whether answers stay logically consistent. These scores are combined into one measure.

Across open- and closed-source models, DiSQ proves hard: even the strongest GPT model reaches only 41% of the ideal score. The measure holds up under domain shift (TED-MDB) and paraphrased questions. It therefore gives a more trustworthy view of discourse comprehension than accuracy alone.

## Key points
- **Question bank:** 22,380 questions covering 11 Level-2 PDTB senses, with some Contingency subtypes merged. Counterfactual questions draw on 5 event relations unrelated to the given sense. Converse questions either mirror or flip the span order, depending on whether the relation is bi- or uni-directional.
- **Evidence selection:** LLaMA2-13B uses a JSON step-by-step reasoning template with one positive and one negative demonstration per relation. Its selections are then checked by human annotators.
- **Main results:**
  - The top GPT model reaches only about 41% of ideal performance on PDTB.
  - Large closed models answer targeted questions well but stumble on converse questions, a form of the "reversal curse".
  - Open-source models generally trail, except variants tuned on chat or code/math data.
- **Helpful signals:** explicit discourse connectives, paragraph context, and previous question–answer history all improve how faithfully models understand discourse.
- **Resources and limitations:** code and data are available at github.com/YisongMiao/DiSQ-Score. The evaluation is limited to English and PDTB-style annotation (PDTB and TED-MDB), covering news and public-speech genres.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

15 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 14 not yet.

- [DRInQ: Evaluating Conversational Implicature with Controlled Context Variation](https://md.research.elamur.ai/free/papers/arai-2026-drinq-evaluating-conversational-implicature-controlled/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{miao2024discursive,
  title = {{Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations}},
  author = {Miao, Yisong and Liu, Hongfu and Lei, Wenqiang and Chen, Nancy and Kan, Min-Yen},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.341},
  url = {https://doi.org/10.18653/v1/2024.acl-long.341}
}
```

- [← previous Discourse Relation Prediction and Discourse Parsing in Dialogues with Minimal Supervision](https://md.research.elamur.ai/free/papers/li-2024-discourse-relation-prediction-discourse-parsing/)
- [next → DistiLLM: Towards Streamlined Distillation for Large Language Models](https://md.research.elamur.ai/free/papers/ko-2024-distillm-towards-streamlined-distillation-large/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/2/)

## References

50 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahul Aralikatte, Matthew Lamm, Daniel Hardt, and Anders Søgaard. 2021. Ellipsis resolution as question answering: An evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 810–817, Online. Association for Comput… [find](https://scholar.google.com/scholar?q=Rahul+Aralikatte%2C+Matthew+Lamm%2C+Daniel+Hardt%2C+and+Anders+S%C3%B8gaard%2E+2021%2E+Ellipsis+resolution+as+question+answering%3A+An+evaluation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+A) [our card](https://md.research.elamur.ai/free/papers/eacl-2021-aralikatte-ellipsis-resolution-as-question-answering/)
2. Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. 2020. Interpretability and analysis in neural NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 1–5, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Sebastian+Gehrmann%2C+and+Ellie+Pavlick%2E+2020%2E+Interpretability+and+analysis+in+neural+NLP%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+)
3. Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Ko-rbak, and Owain Evans. 2024. The reversal curse: LLMs trained on “a is b” fail to learn “b is a”. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lukas+Berglund%2C+Meg+Tong%2C+Maximilian+Kaufmann%2C+Mikita+Balesni%2C+Asa+Cooper+Stickland%2C+Tomasz+Ko%2Drbak%2C+and+Owain+Evans%2E+2024%2E+The+reversal+curse%3A+LLMs+trained+on+%E2%80%9Ca+is+b%E2%80%9D+fail+to+learn+%E2%80%9Cb+is+a%E2%80%9D%2E+In+The+)
4. Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christop… [find](https://scholar.google.com/scholar?q=Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen+Krueger%2C+Tom+Henighan%2C+Rewon+Child%2C+Aditya+Ramesh%2C+Daniel+Ziegler%2C+Jeffrey+Wu%2C+Clemens+Winter%2C+C)
5. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Tim+Rockt%C3%A4schel%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2018%2E+e%2Dsnli%3A+Natural+language+inference+with+natural+language+explanations%2E+In+Advances+in+Neural+Information+Processing+Sys) [our card](https://md.research.elamur.ai/free/papers/camburu-2018-e-snli-natural-language-inference/)
6. Chunkit Chan, Cheng Jiayang, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2024. Exploring the potential of ChatGPT on sentence level relations: A focus on temporal, causal, and discourse relations. In Findings of the Association for Computational Linguistics: EACL 2024, pages 6… [find](https://scholar.google.com/scholar?q=Chunkit+Chan%2C+Cheng+Jiayang%2C+Weiqi+Wang%2C+Yuxin+Jiang%2C+Tianqing+Fang%2C+Xin+Liu%2C+and+Yangqiu+Song%2E+2024%2E+Exploring+the+potential+of+ChatGPT+on+sentence+level+relations%3A+A+focus+on+temporal%2C+causal%2C+and+d) [our card](https://md.research.elamur.ai/free/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
7. Chunkit Chan, Xin Liu, Jiayang Cheng, Zihan Li, Yangqiu Song, Ginny Wong, and Simon See. 2023. DiscoPrompt: Path prediction prompt tuning for implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: ACL 2023, pages 35–57, Toronto, Canada. Association for… [find](https://scholar.google.com/scholar?q=Chunkit+Chan%2C+Xin+Liu%2C+Jiayang+Cheng%2C+Zihan+Li%2C+Yangqiu+Song%2C+Ginny+Wong%2C+and+Simon+See%2E+2023%2E+DiscoPrompt%3A+Path+prediction+prompt+tuning+for+implicit+discourse+relation+recognition%2E+In+Findings+of+th) [our card](https://md.research.elamur.ai/free/papers/chan-2023-discoprompt-path-prediction-prompt-tuning/)
8. Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019. Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… [find](https://scholar.google.com/scholar?q=Mingda+Chen%2C+Zewei+Chu%2C+and+Kevin+Gimpel%2E+2019%2E+Evaluation+benchmarks+and+learning+criteria+for+discourse%2Daware+sentence+representations%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+)
9. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
10. Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, and Amir Zeldes. 2023. [find](https://scholar.google.com/scholar?q=Chlo%C3%A9+Braud%2C+Yang+Janet+Liu%2C+Eleni+Metheniti%2C+Philippe+Muller%2C+Laura+Rivi%C3%A8re%2C+Attapol+Rutherford%2C+and+Amir+Zeldes%2E+2023%2E)
11. The DISRPT 2023 shared task on elementary discourse unit segmentation, connective detection, and relation classification. In Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023), pages 1–21, Toronto, Canada. The Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+DISRPT+2023+shared+task+on+elementary+discourse+unit+segmentation%2C+connective+detection%2C+and+relation+classification%2E+In+Proceedings+of+the+3rd+Shared+Task+on+Discourse+Relation+Parsing+and+Treeba) [our card](https://md.research.elamur.ai/free/papers/braud-2023-disrpt-2023-shared-task-elementary/)
12. Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022a. QAFactEval: Improved QA-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p… [find](https://scholar.google.com/scholar?q=Alexander+Fabbri%2C+Chien%2DSheng+Wu%2C+Wenhao+Liu%2C+and+Caiming+Xiong%2E+2022a%2E+QAFactEval%3A+Improved+QA%2Dbased+factual+consistency+evaluation+for+summarization%2E+In+Proceedings+of+the+2022+Conference+of+the+Nor) [our card](https://md.research.elamur.ai/free/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for C… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+)
14. Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022b. Qafacteval: Improved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… [find](https://scholar.google.com/scholar?q=Alexander+Richard+Fabbri%2C+Chien%2DSheng+Wu%2C+Wenhao+Liu%2C+and+Caiming+Xiong%2E+2022b%2E+Qafacteval%3A+Improved+qa%2Dbased+factual+consistency+evaluation+for+summarization%2E+In+Proceedings+of+the+2022+Conference+of) [our card](https://md.research.elamur.ai/free/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
15. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne,… [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+Zettlemoyer%2E+2018%2E+AllenNLP%3A+A+deep+semantic+natural+language+processing) [our card](https://md.research.elamur.ai/free/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
16. MAK Halliday. 1976. Cohesion in english. Longman. [find](https://scholar.google.com/scholar?q=MAK+Halliday%2E+1976%2E+Cohesion+in+english%2E+Longman%2E)
17. Yufang Hou. 2020. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yufang+Hou%2E+2020%2E+Bridging+anaphora+resolution+as+question+answering%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1428%E2%80%931438%2C+Online%2E+Association+f) [our card](https://md.research.elamur.ai/free/papers/hou-2020-bridging-anaphora-resolution-as-question/)
18. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20349–20360. [find](https://scholar.google.com/scholar?q=Yushi+Hu%2C+Benlin+Liu%2C+Jungo+Kasai%2C+Yizhong+Wang%2C+Mari+Ostendorf%2C+Ranjay+Krishna%2C+and+Noah+A%2E+Smith%2E+2023%2E+Tifa%3A+Accurate+and+interpretable+text%2Dto%2Dimage+faithfulness+evaluation+with+question+answering)
19. Patrick Huber and Giuseppe Carenini. 2022. Towards understanding large-scale discourse structures in pre-trained and fine-tuned language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 23… [find](https://scholar.google.com/scholar?q=Patrick+Huber+and+Giuseppe+Carenini%2E+2022%2E+Towards+understanding+large%2Dscale+discourse+structures+in+pre%2Dtrained+and+fine%2Dtuned+language+models%2E+In+Proceedings+of+the+2022+Conference+of+the+North+Amer)
20. Feng Jiang, Yaxin Fan, Xiaomin Chu, Peifeng Li, and Qiaoming Zhu. 2021. Not just classification: Recognizing implicit discourse relation on joint modeling of classification and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2418–2431, Onl… [find](https://scholar.google.com/scholar?q=Feng+Jiang%2C+Yaxin+Fan%2C+Xiaomin+Chu%2C+Peifeng+Li%2C+and+Qiaoming+Zhu%2E+2021%2E+Not+just+classification%3A+Recognizing+implicit+discourse+relation+on+joint+modeling+of+classification+and+generation%2E+In+Proceedi) [our card](https://md.research.elamur.ai/free/papers/jiang-2021-not-just-classification-recognizing-implicit/)
21. Shafiq Joty, Muhammad Tasnim Mohiuddin, and Dat Tien Nguyen. 2018. Coherence modeling of asynchronous conversations: A neural entity grid approach. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 558–568, Melbourne, Australia.… [find](https://scholar.google.com/scholar?q=Shafiq+Joty%2C+Muhammad+Tasnim+Mohiuddin%2C+and+Dat+Tien+Nguyen%2E+2018%2E+Coherence+modeling+of+asynchronous+conversations%3A+A+neural+entity+grid+approach%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Ass)
22. Prathyusha Jwalapuram, Shafiq Joty, and Xiang Lin. 2022. Rethinking self-supervision objectives for generalizable coherence modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6044–6059, Dublin, Ireland. Association for… [find](https://scholar.google.com/scholar?q=Prathyusha+Jwalapuram%2C+Shafiq+Joty%2C+and+Xiang+Lin%2E+2022%2E+Rethinking+self%2Dsupervision+objectives+for+generalizable+coherence+modeling%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+C)
23. Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. 2020. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404–5414, Online. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Najoung+Kim%2C+Song+Feng%2C+Chulaka+Gunasekara%2C+and+Luis+Lastras%2E+2020%2E+Implicit+discourse+relation+classification%3A+We+need+to+talk+about+evaluation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Assoc) [our card](https://md.research.elamur.ai/free/papers/kim-2020-implicit-discourse-relation-classification-we/)
24. Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2022. Discourse comprehension: A question answering framework to represent sentence connections. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11752–11764, Abu… [find](https://scholar.google.com/scholar?q=Wei%2DJen+Ko%2C+Cutter+Dalton%2C+Mark+Simmons%2C+Eliza+Fisher%2C+Greg+Durrett%2C+and+Junyi+Jessy+Li%2E+2022%2E+Discourse+comprehension%3A+A+question+answering+framework+to+represent+sentence+connections%2E+In+Proceedings)
25. Wei-Jen Ko, Yating Wu, Cutter Dalton, Danan-jay Srinivas, Greg Durrett, and Junyi Jessy Li. 2023. Discourse analysis via questions and answers: Parsing dependency structures of questions under discussion. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11181– 11195, Tor… [find](https://scholar.google.com/scholar?q=Wei%2DJen+Ko%2C+Yating+Wu%2C+Cutter+Dalton%2C+Danan%2Djay+Srinivas%2C+Greg+Durrett%2C+and+Junyi+Jessy+Li%2E+2023%2E+Discourse+analysis+via+questions+and+answers%3A+Parsing+dependency+structures+of+questions+under+discuss) [our card](https://md.research.elamur.ai/free/papers/ko-2023-discourse-analysis-questions-answers-parsing/)
26. Murathan Kurfalı and Robert Östling. 2021. Let’s be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction. In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language, pages 1–10, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Murathan+Kurfal%C4%B1+and+Robert+%C3%96stling%2E+2021%2E+Let%E2%80%99s+be+explicit+about+that%3A+Distant+supervision+for+implicit+discourse+relation+classification+via+connective+prediction%2E+In+Proceedings+of+the+1st+Worksho) [our card](https://md.research.elamur.ai/free/papers/kurfal-2021-let-s-be-explicit-about/)
27. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+2021%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingu)
28. Wei Liu and Michael Strube. 2023. Annotation-inspired implicit discourse relation classification with auxiliary discourse connective generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15696–15712, Toronto, Canada. Ass… [find](https://scholar.google.com/scholar?q=Wei+Liu+and+Michael+Strube%2E+2023%2E+Annotation%2Dinspired+implicit+discourse+relation+classification+with+auxiliary+discourse+connective+generation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Associ) [our card](https://md.research.elamur.ai/free/papers/liu-2023-annotation-inspired-implicit-discourse-relation/)
29. Xin Liu, Jiefu Ou, Yangqiu Song, and Xin Jiang. 2021. On the importance of word and sentence representation learning in implicit discourse relation classification. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3830– 3… [find](https://scholar.google.com/scholar?q=Xin+Liu%2C+Jiefu+Ou%2C+Yangqiu+Song%2C+and+Xin+Jiang%2E+2021%2E+On+the+importance+of+word+and+sentence+representation+learning+in+implicit+discourse+relation+classification%2E+In+Proceedings+of+the+Twenty%2DNinth+I)
30. Yan Liu, Yu Liu, Xiaokang Chen, Pin-Yu Chen, Daoguang Zan, Min-Yen Kan, and Tsung-Yi Ho. 2024. The devil is in the neurons: Interpreting and mitigating social biases in language models. In The Twelfth International Conference on Learning Representations. pages 8–14, New Orleans, Louisiana. Associati… [find](https://scholar.google.com/scholar?q=Yan+Liu%2C+Yu+Liu%2C+Xiaokang+Chen%2C+Pin%2DYu+Chen%2C+Daoguang+Zan%2C+Min%2DYen+Kan%2C+and+Tsung%2DYi+Ho%2E+2024%2E+The+devil+is+in+the+neurons%3A+Interpreting+and+mitigating+social+biases+in+language+models%2E+In+The+Twelfth)
31. William C Mann and Sandra A Thompson. 1987. Rhetorical structure theory: A theory of text organization. University of Southern California, Information Sciences Institute Los Angeles. [find](https://scholar.google.com/scholar?q=William+C+Mann+and+Sandra+A+Thompson%2E+1987%2E+Rhetorical+structure+theory%3A+A+theory+of+text+organization%2E+University+of+Southern+California%2C+Information+Sciences+Institute+Los+Angeles%2E)
32. Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 645–654, Gothenburg, Sweden. As… [find](https://scholar.google.com/scholar?q=Attapol+Rutherford+and+Nianwen+Xue%2E+2014%2E+Discovering+implicit+discourse+relations+through+brown+cluster+pair+representation+and+coreference+patterns%2E+In+Proceedings+of+the+14th+Conference+of+the+Euro) [our card](https://md.research.elamur.ai/free/papers/rutherford-2014-discovering-implicit-discourse-relations-through/)
33. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+35%2C+pages+17359%E2%80%9317372%2E+)
34. Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. ArXiv, abs/2401.07078. [find](https://scholar.google.com/scholar?q=Settaluri+Lakshmi+Sravanthi%2C+Meet+Doshi%2C+Tankala+Pavan+Kalyan%2C+Rudra+Murthy%2C+Pushpak+Bhattacharyya%2C+and+Raj+Dabre%2E+2024%2E+Pub%3A+A+pragmatics+understanding+benchmark+for+assessing+llms%E2%80%99+pragmatics+capabi) [our card](https://md.research.elamur.ai/free/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)
35. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 683–691, S… [find](https://scholar.google.com/scholar?q=Emily+Pitler%2C+Annie+Louis%2C+and+Ani+Nenkova%2E+2009%2E+Automatic+sense+prediction+for+implicit+discourse+relations+in+text%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+) [our card](https://md.research.elamur.ai/free/papers/disc-2009-pitler-automatic-sense-prediction-implicit-discourse/)
36. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4593%E2%80%934) [our card](https://md.research.elamur.ai/free/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
37. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu… [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+Proceedings+of+the+Seventh+Joint+Conference+)
38. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Asso… [find](https://scholar.google.com/scholar?q=Rashmi+Prasad%2C+Nikhil+Dinesh%2C+Alan+Lee%2C+Eleni+Milt%2Dsakaki%2C+Livio+Robaldo%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2008%2E+The+Penn+Discourse+TreeBank+2%2E0%2E+In+Proceedings+of+the+Sixth+International+Conference+) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
39. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819, Online. Association for C… [find](https://scholar.google.com/scholar?q=Valentina+Pyatkin%2C+Ayal+Klein%2C+Reut+Tsarfaty%2C+and+Ido+Dagan%2E+2020%2E+QADiscourse+%2D+Discourse+Relations+as+QA+Pairs%3A+Representation%2C+Crowdsourcing+and+Baselines%2E+In+Proceedings+of+the+2020+Conference+on+) [our card](https://md.research.elamur.ai/free/papers/pyatkin-2020-qadiscourse-discourse-relations-as-qa/)
40. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+R%2E+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Daniel+M%2E+Bikel%2C+Lukas+Blecher%2C+Cristian+)
41. Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 97–101, San Diego, California.… [find](https://scholar.google.com/scholar?q=Marco+Ribeiro%2C+Sameer+Singh%2C+and+Carlos+Guestrin%2E+2016%2E+%E2%80%9Cwhy+should+I+trust+you%3F%E2%80%9D%3A+Explaining+the+predictions+of+any+classifier%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+) [our card](https://md.research.elamur.ai/free/papers/ribeiro-2016-why-should-i-trust-explaining/)
42. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Yizhong+Wang%2C+Sujian+Li%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+Do+NLP+models+know+numbers%3F+probing+numeracy+in+embeddings%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+N) [our card](https://md.research.elamur.ai/free/papers/wallace-2019-do-nlp-models-know-numbers/)
43. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Chenxu Wa… [find](https://scholar.google.com/scholar?q=Rachel+Rudinger%2C+Jason+Naradowsky%2C+Brian+Leonard%2C+and+Benjamin+Van+Durme%2E+2018%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Associ) [our card](https://md.research.elamur.ai/free/papers/rudinger-2018-gender-bias-coreference-resolution/)
44. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania, 35:108. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2C+Rashmi+Prasad%2C+Alan+Lee%2C+and+Aravind+Joshi%2E+2019%2E+The+penn+discourse+treebank+3%2E0+annotation+manual%2E+Philadelphia%2C+University+of+Pennsylvania%2C+35%3A108%2E)
45. Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. 2023. QUDeval: The evaluation of questions under discussion discourse parsing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5344–5363, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yating+Wu%2C+Ritika+Mangla%2C+Greg+Durrett%2C+and+Junyi+Jessy+Li%2E+2023%2E+QUDeval%3A+The+evaluation+of+questions+under+discussion+discourse+parsing%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in) [our card](https://md.research.elamur.ai/free/papers/wu-2023-qudeval-evaluation-questions-under-discussion/)
46. Wei Xiang, Zhenglin Wang, Lu Dai, and Bang Wang. 2022. ConnPrompt: Connective-cloze prompt learning for implicit discourse relation recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 902–911, Gyeongju, Republic of Korea. International Committee on Co… [find](https://scholar.google.com/scholar?q=Wei+Xiang%2C+Zhenglin+Wang%2C+Lu+Dai%2C+and+Bang+Wang%2E+2022%2E+ConnPrompt%3A+Connective%2Dcloze+prompt+learning+for+implicit+discourse+relation+recognition%2E+In+Proceedings+of+the+29th+International+Conference+on+) [our card](https://md.research.elamur.ai/free/papers/xiang-2022-connprompt-connective-cloze-prompt-learning/)
47. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. ArXiv, abs/2304.12244. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+2023%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2E+ArXiv%2C+abs%2F2304%2E122) [our card](https://md.research.elamur.ai/free/papers/xu-2023-wizardlm-empowering-large-pre-trained/)
48. Deniz Zeyrek, Amália Mendes, and Murathan Kurfalı. 2018. Multilingual extension of PDTB-style annotation: The case of TED multilingual discourse bank. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resource… [find](https://scholar.google.com/scholar?q=Deniz+Zeyrek%2C+Am%C3%A1lia+Mendes%2C+and+Murathan+Kurfal%C4%B1%2E+2018%2E+Multilingual+extension+of+PDTB%2Dstyle+annotation%3A+The+case+of+TED+multilingual+discourse+bank%2E+In+Proceedings+of+the+Eleventh+International+Conf)
49. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. [find](https://scholar.google.com/scholar?q=Zihao+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E+In+Proceedings+of+the+38th+International+Conference+on+M) [our card](https://md.research.elamur.ai/free/papers/zhao-2021-calibrate-before-use-improving-few/)
50. Hao Zhou, Man Lan, Yuanbin Wu, Yuefeng Chen, and Meirong Ma. 2022. Prompt-based connective prediction method for fine-grained implicit discourse relation recognition. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3848–3858, Abu Dhabi, United Arab Emirates. Associati… [find](https://scholar.google.com/scholar?q=Hao+Zhou%2C+Man+Lan%2C+Yuanbin+Wu%2C+Yuefeng+Chen%2C+and+Meirong+Ma%2E+2022%2E+Prompt%2Dbased+connective+prediction+method+for+fine%2Dgrained+implicit+discourse+relation+recognition%2E+In+Findings+of+the+Association+fo) [our card](https://md.research.elamur.ai/free/papers/zhou-2022-prompt-based-connective-prediction-method/)