---
title: "Molweni — paper by Li et al. (2020) — summary and ideas"
description: "Most machine reading comprehension (MRC) benchmarks use well-edited prose. 54.7% EM. 28 references, cited by 62 papers here."
authors: ["Jiaqi Li", "Ming Liu", "Min-Yen Kan", "Zihao Zheng", "Zekun Wang", "Wenqiang Lei", "Ting Liu", "Bing Qin"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.238/"
doi: "10.18653/v1/2020.coling-main.238"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2020-molweni-challenge-multiparty-dialogues-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2020-molweni-challenge-multiparty-dialogues-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure

# Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure

**Authors:** [Jiaqi Li](https://md.research.elamur.ai/free/en/authors/jiaqi-li/), [Ming Liu](https://md.research.elamur.ai/free/en/authors/ming-liu/), [Min-Yen Kan](https://md.research.elamur.ai/free/en/authors/min-yen-kan/), Zihao Zheng, [Zekun Wang](https://md.research.elamur.ai/free/en/authors/zekun-wang/), [Wenqiang Lei](https://md.research.elamur.ai/free/en/authors/wenqiang-lei/), [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) and [Bing Qin](https://md.research.elamur.ai/free/en/authors/bing-qin/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.238/) · [PDF](https://aclanthology.org/2020.coling-main.238.pdf) · [DOI 10.18653/v1/2020.coling-main.238](https://doi.org/10.18653/v1/2020.coling-main.238) · [OpenAlex](https://openalex.org/W3015871652)

**Citations:** 79 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/li-2020-molweni-challenge-multiparty-dialogues-based/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most machine reading comprehension (MRC) benchmarks use well-edited prose. Multiparty chat is different: consecutive messages often do not connect, and replies can target turns far back in the conversation. Earlier dialog MRC sets were either small (FriendsQA) or mostly two-party (DREAM), and none of them annotated discourse structure.

The authors built Molweni from technical support conversations in the Ubuntu Chat Corpus. They kept dialogs of 8–15 turns with 2–9 participants and no message longer than 20 words. Annotators did two things for each dialog:
- marked discourse dependencies in a modified SDRT scheme, treating each utterance as a node in a graph;
- wrote extractive questions, both answerable and unanswerable, with plausible answers given for the unanswerable ones.

BERT-wwm performs close to human level on SQuAD 2.0 but falls well short of humans on Molweni. Its errors often come from losing track of who said what. The discourse annotations are about seven times larger than STAC, the only earlier corpus of this kind. Molweni is therefore a useful testbed for both dialog MRC and data-driven multiparty discourse parsing.

## Key points
- **Scale:** the source sample is about 10,000 Ubuntu dialogs. Of these, 9,754 were annotated, with 30,066 questions (25,779 answerable, 4,287 unanswerable) and 78,245 discourse relations. Dialogs average 3.51 speakers.
- **Splits:** 8,771 dialogs for train and 883 for dev. The test set is 100 dialogs shared by all ten annotators, with 2,871 questions. Most questions are 5W1H types.
- **MRC results:** BERT-wwm reaches 54.7% EM and 67.7% F1 on Molweni, against 89.1% F1 on SQuAD 2.0. Human F1 on Molweni is 80.2%.
- **Discourse parsing:** the Deep Sequential model (Shi and Huang, 2019) scores 78.1 F1 for link prediction and 54.8 F1 for links plus relations. This is on par with or better than its STAC results.
- **Combined training:** adding Molweni's training data to STAC's lifts link prediction on STAC from 73.2 to 78.0 F1. The dataset is released at github.com/HIT-SCIR/Molweni.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 76 works · 27,918 citations
- [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) — Research institute, Shenzhen, China · 18 works · 2,349 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (3): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 243 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 401 works · [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 394 works

## Cite

Cite as: Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu and Bing Qin. 2020. Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2020molweni,
  title = {{Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure}},
  author = {Jiaqi Li, Ming Liu, Min-Yen Kan, Zihao Zheng, Zekun Wang, Wenqiang Lei, Ting Liu and Bing Qin},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.238},
  url = {https://doi.org/10.18653/v1/2020.coling-main.238}
}
```

- [← previous Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)
- [next → On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

28 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stergos Afantenos, Eric Kow, Nicholas Asher, and Jérémy Perret. 2015. [Discourse parsing for multi-party chat dialogues](https://md.research.elamur.ai/free/en/papers/afantenos-2015-discourse-parsing-multi-party-chat/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 928–937. Association for Computational Linguistics.
2. Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos. 2016. [Discourse structure and dialogue acts in multiparty dialogue: the stac corpus](https://md.research.elamur.ai/free/en/papers/asher-2016-discourse-structure-dialogue-acts-multiparty/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2721–2727.
3. Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pages 2787–2795. [find](https://scholar.google.com/scholar?q=Antoine+Bordes%2C+Nicolas+Usunier%2C+Alberto+Garcia%2DDuran%2C+Jason+Weston%2C+and+Oksana+Yakhnenko%2E+2013%2E+Translating+embeddings+for+modeling+multi%2Drelational+data%2E+In+Advances+in+neural+information+processing)
4. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. [Quac: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
6. Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, and Rui Yan. 2019. Gsn: A graph-structured network for multi-party dialogues. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5010–5016. International Joint Conferences on Ar… [find](https://scholar.google.com/scholar?q=Wenpeng+Hu%2C+Zhangming+Chan%2C+Bing+Liu%2C+Dongyan+Zhao%2C+Jinwen+Ma%2C+and+Rui+Yan%2E+2019%2E+Gsn%3A+A+graph%2Dstructured+network+for+multi%2Dparty+dialogues%2E+In+Proceedings+of+the+Twenty%2DEighth+International+Joint+Con)
7. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
8. Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328.
9. Jonathan K. Kummerfeld, Sai R. Gouravajhala, Joseph J. Peper, Vignesh Athreya, Chulaka Gunasekara, Jatin Ganhotra, Siva Sankalp Patel, Lazaros C Polymenakos, and Walter Lasecki. 2019. A large-scale corpus for conversation disentanglement. In Proceedings of the 57th Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Jonathan+K%2E+Kummerfeld%2C+Sai+R%2E+Gouravajhala%2C+Joseph+J%2E+Peper%2C+Vignesh+Athreya%2C+Chulaka+Gunasekara%2C+Jatin+Ganhotra%2C+Siva+Sankalp+Patel%2C+Lazaros+C+Polymenakos%2C+and+Walter+Lasecki%2E+2019%2E+A+large%2Dscale+co)
10. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794.
11. Wenqiang Lei, Xuancong Wang, Meichun Liu, Ilija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: A simple word interaction model for implicit discourse relation recognition. In IJCAI, pages 4026–4032. [find](https://scholar.google.com/scholar?q=Wenqiang+Lei%2C+Xuancong+Wang%2C+Meichun+Liu%2C+Ilija+Ilievski%2C+Xiangnan+He%2C+and+Min%2DYen+Kan%2E+2017%2E+Swim%3A+A+simple+word+interaction+model+for+implicit+discourse+relation+recognition%2E+In+IJCAI%2C+pages+4026%E2%80%9340)
12. Wenqiang Lei, Yuanxin Xiang, Yuwei Wang, Qian Zhong, Meichun Liu, and Min-Yen Kan. 2018. Linguistic properties matter for implicit discourse relation recognition: Combining semantic interaction, topic continuity and attribution. The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). [find](https://scholar.google.com/scholar?q=Wenqiang+Lei%2C+Yuanxin+Xiang%2C+Yuwei+Wang%2C+Qian+Zhong%2C+Meichun+Liu%2C+and+Min%2DYen+Kan%2E+2018%2E+Linguistic+properties+matter+for+implicit+discourse+relation+recognition%3A+Combining+semantic+interaction%2C+topic)
13. Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. 2019. [Keep meeting summaries on topic: Abstractive multi-modal meeting summarization](https://md.research.elamur.ai/free/en/papers/li-2019-keep-meeting-summaries-topic-abstractive/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2190–2196, Florence, Italy, July. Association for Computati…
14. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. [The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems](https://md.research.elamur.ai/free/en/papers/lowe-2015-ubuntu-dialogue-corpus-large-dataset/). In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285–294.
15. Kaixin Ma, Tomasz Jurczyk, and Jinho D Choi. 2018. Challenging reading comprehension on daily conversation: Passage completion on multiparty dialog. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu… [find](https://scholar.google.com/scholar?q=Kaixin+Ma%2C+Tomasz+Jurczyk%2C+and+Jinho+D+Choi%2E+2018%2E+Challenging+reading+comprehension+on+daily+conversation%3A+Passage+completion+on+multiparty+dialog%2E+In+Proceedings+of+the+2018+Conference+of+the+North+)
16. Jérémy Perret, Stergos Afantenos, Nicholas Asher, and Mathieu Morey. 2016. [Integer linear programming for discourse parsing](https://md.research.elamur.ai/free/en/papers/perret-2016-integer-linear-programming-discourse-parsing/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 99–109. Association…
17. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind K Joshi, and Bonnie L Webber. 2008. [The penn discourse treebank 2.0](https://md.research.elamur.ai/free/en/papers/prasad-2008-penn-discourse-treebank-2-0/). In LREC. Citeseer.
18. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
19. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don¡t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789.
20. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266, March.
21. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. [MCTest: A challenge dataset for the open-domain machine comprehension of text](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Seattle, Washington, USA, October. Association…
22. Zhouxing Shi and Minlie Huang. 2019. A deep sequential model for discourse parsing on multi-party dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7007–7014. [find](https://scholar.google.com/scholar?q=Zhouxing+Shi+and+Minlie+Huang%2E+2019%2E+A+deep+sequential+model+for+discourse+parsing+on+multi%2Dparty+dialogues%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+33%2C+pages+7007%E2%80%93701)
23. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. [Dream: A challenge data set and models for dialogue-based reading comprehension](https://md.research.elamur.ai/free/en/papers/sun-2019-dream-challenge-data-set-models/). Transactions of the Association for Computational Linguistics, 7:217–231.
24. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. [Newsqa: A machine comprehension dataset](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/). In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200.
25. Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. [What is the jeopardy model? a quasi-synchronous grammar for qa](https://md.research.elamur.ai/free/en/papers/wang-2007-what-is-jeopardy-model-quasi/). In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 22–32.
26. Zhengzhe Yang and Jinho D. Choi. 2019. FriendsQA: Open-domain question answering on TV show transcripts. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 188–197, Stockholm, Sweden, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhengzhe+Yang+and+Jinho+D%2E+Choi%2E+2019%2E+FriendsQA%3A+Open%2Ddomain+question+answering+on+TV+show+transcripts%2E+In+Proceedings+of+the+20th+Annual+SIGdial+Meeting+on+Discourse+and+Dialogue%2C+pages+188%E2%80%93197%2C+Sto)
27. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. [Wikiqa: A challenge dataset for open-domain question answering](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018.
28. Zhou Zhao, Haojie Pan, Changjie Fan, Yan Liu, Linlin Li, and Min Yang. 2019. Abstractive meeting summarization via hierarchical adaptive segmental network learning. In The World Wide Web Conference, pages 3455–3461. ACM. [find](https://scholar.google.com/scholar?q=Zhou+Zhao%2C+Haojie+Pan%2C+Changjie+Fan%2C+Yan+Liu%2C+Linlin+Li%2C+and+Min+Yang%2E+2019%2E+Abstractive+meeting+summarization+via+hierarchical+adaptive+segmental+network+learning%2E+In+The+World+Wide+Web+Conference%2C+p)

*Updated 24 October 2026*