--- title: "Improving Crowdsourcing-Based Annotation of — Kishimoto 2018" description: "Discourse parsing needs annotated corpora, but such resources are rare outside English and Chinese, and building them by hand is slow and costly." authors: ["Yudai Kishimoto", "Shinnosuke Sawada", "Yugo Murawaki", "Daisuke Kawahara", "Sadao Kurohashi"] year: 2018 venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)" source: "https://aclanthology.org/L18-1637/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/kishimoto-2018-improving-crowdsourcing-based-annotation-japanese/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kishimoto-2018-improving-crowdsourcing-based-annotation-japanese/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/2/) / Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations # Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations **Authors:** [Yudai Kishimoto](https://md.research.elamur.ai/free/authors/yudai-kishimoto/), Shinnosuke Sawada, [Yugo Murawaki](https://md.research.elamur.ai/free/authors/yugo-murawaki/), [Daisuke Kawahara](https://md.research.elamur.ai/free/authors/daisuke-kawahara/), [Sadao Kurohashi](https://md.research.elamur.ai/free/authors/sadao-kurohashi/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L18-1637/) · [PDF](https://aclanthology.org/L18-1637.pdf) · [OpenAlex](https://openalex.org/W2805315173) **Citations:** 6 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Discourse parsing needs annotated corpora, but such resources are rare outside English and Chinese, and building them by hand is slow and costly. Kawahara et al. (2014) used crowdsourcing to build a Japanese discourse corpus of 10,000 web documents, but they never measured how reliable those labels were. This paper fills that gap. The authors had trained linguists annotate a gold subset and compared the crowd labels against it. The crowd-produced "Old Annotation" reached only 49.5% micro-averaged F-measure. Error analysis found two main problems: - Workers frequently overlooked explicit discourse connectives. - The task instructions did not make the labeling criteria clear. To address this, the authors revised the instructions to include descriptions of connective phrases. These descriptions push workers to apply positive language tests, such as checking whether "if" can be inserted between two clauses. Re-running the crowdsourcing produced a "New Annotation" that improved F-measure by about 3 points, with a larger gain on explicit relations. The work shows that small changes to task design can raise crowd annotation quality. It also identifies implicit relations as the main remaining weakness. ## Key points - **Base resource:** Each document is the first three sentences of a page from the Kyoto University Web Leads Corpus. - Clauses were segmented with the KNP parser. - Labels use a two-level tagset: CONTINGENCY covers Cause/Reason, Purpose, Condition and Ground; COMPARISON covers Contrast and Concession; OTHER is a separate label. - **Original pipeline:** Annotation ran in two stages on Yahoo! Crowdsourcing, with 10 workers per item and answers aggregated using GLAD. - The first stage filtered pairs labeled OTHER; the second stage classified the remaining relations. - The full annotation took under eight hours. - **Gold data:** Three professional annotators labeled 500 documents. Of these, 313 were sampled for evaluation. - **Main error source:** 40.2% (384/955) of explicit clause pairs were wrongly filtered out as OTHER. Roughly 47% of those pairs contained Condition-type connectives. - **Results:** Micro F-measure rose from 0.495 to 0.528, and recall rose by about 7.5 points. On explicit relations, F-measure improved from 0.513 to 0.605. - The authors plan a cloze-style connective-insertion test, modeled on the PDTB procedure, to handle implicit relations. - They also plan to release the re-annotated data publicly. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/) ## Cited by Cited by 11 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 6 not yet. - [QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines](https://md.research.elamur.ai/free/papers/pyatkin-2020-qadiscourse-discourse-relations-as-qa/) - [Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design](https://md.research.elamur.ai/free/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/) - [Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training](https://md.research.elamur.ai/free/papers/scholman-2022-design-choices-crowdsourcing-discourse-relation/) - [DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations](https://md.research.elamur.ai/free/papers/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/) - [Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task](https://md.research.elamur.ai/free/papers/yung-2019-crowdsourcing-discourse-relation-annotations-two/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kishimoto2018improving, title = {{Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations}}, author = {Kishimoto, Yudai and Sawada, Shinnosuke and Murawaki, Yugo and Kawahara, Daisuke and Kurohashi, Sadao}, booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)}, year = {2018}, doi = {10.63317/28vbj6p6xyoq}, url = {https://doi.org/10.63317/28vbj6p6xyoq} } ``` - [← previous Implicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning](https://md.research.elamur.ai/free/papers/guo-2018-implicit-discourse-relation-recognition-neural/) - [next → Improving Event Coreference Resolution by Modeling Correlations between Event Coreference Chains and Document Topic Structures](https://md.research.elamur.ai/free/papers/choubey-2018-improving-event-coreference-resolution-modeling/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/2/) ## References 8 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Asher, N. and Lascarides, A. (2003). Logics of Conversation. [find](https://scholar.google.com/scholar?q=Asher%2C+N%2E+and+Lascarides%2C+A%2E+%282003%29%2E+Logics+of+Conversation%2E) 2. Cambridge University Press. Carlson, L., Marcu, D., and Okurowski, M. E. (2001). Building a discourse-tagged corpus in the framework of (iii) rhetorical structure theory. In Proceedings of the Second [(and) They made delicious wine.] SIGdial Workshop on Discourse and Dialogue - Volume The discourse… [find](https://scholar.google.com/scholar?q=Cambridge+University+Press%2E+Carlson%2C+L%2E%2C+Marcu%2C+D%2E%2C+and+Okurowski%2C+M%2E+E%2E+%282001%29%2E+Building+a+discourse%2Dtagged+corpus+in+the+framework+of+%28iii%29+rhetorical+structure+theory%2E+In+Proceedings+of+the+Second+) 3. [Tuchiya, who is a sales manager of Odoya, requested (a brand new product) for winery in Ya-magata,] Dublin City University and Association for Computational Linguistics. Kurohashi, S. and Nagao, M. (1994). A syntactic analysis method of long Japanese sentences based on the detection of conjunctive… [find](https://scholar.google.com/scholar?q=%5BTuchiya%2C+who+is+a+sales+manager+of+Odoya%2C+requested+%28a+brand+new+product%29+for+winery+in+Ya%2Dmagata%2C%5D+Dublin+City+University+and+Association+for+Computational+Linguistics%2E+Kurohashi%2C+S%2E+and+Nagao%2C+M%2E+%28) 4. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A., and Webber, B. (2008). The Penn discourse treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC2008), pages 2961–2968. [find](https://scholar.google.com/scholar?q=Prasad%2C+R%2E%2C+Dinesh%2C+N%2E%2C+Lee%2C+A%2E%2C+Miltsakaki%2C+E%2E%2C+Robaldo%2C+L%2E%2C+Joshi%2C+A%2E%2C+and+Webber%2C+B%2E+%282008%29%2E+The+Penn+discourse+treebank+2%2E0%2E+In+Proceedings+of+the+6th+International+Conference+on+Language+Resource) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/) 5. Prasad, R., Webber, B., and Joshi, A. (2014). Reflections on the penn discourse treebank, comparable corpora, and complementary annotation. Computational Linguistics, 40(4):921–950. [find](https://scholar.google.com/scholar?q=Prasad%2C+R%2E%2C+Webber%2C+B%2E%2C+and+Joshi%2C+A%2E+%282014%29%2E+Reflections+on+the+penn+discourse+treebank%2C+comparable+corpora%2C+and+complementary+annotation%2E+Computational+Linguistics%2C+40%284%29%3A921%E2%80%93950%2E) [our card](https://md.research.elamur.ai/free/papers/prasad-2014-reflections-penn-discourse-treebank-comparable/) 6. Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. (2008). Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Snow%2C+R%2E%2C+O%E2%80%99Connor%2C+B%2E%2C+Jurafsky%2C+D%2E%2C+and+Ng%2C+A%2E+%282008%29%2E+Cheap+and+fast+%E2%80%93+but+is+it+good%3F+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+2008+Conference+on+Empiric) 7. Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., and Ru-volo, P. L. (2009). Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Y. Bengio, et al., editors, Advances in Neural Information Processing Systems 22, pages 2035–2043. Curran Associates, Inc… [find](https://scholar.google.com/scholar?q=Whitehill%2C+J%2E%2C+Wu%2C+T%2E%2C+Bergsma%2C+J%2E%2C+Movellan%2C+J%2E+R%2E%2C+and+Ru%2Dvolo%2C+P%2E+L%2E+%282009%29%2E+Whose+vote+should+count+more%3A+Optimal+integration+of+labels+from+labelers+of+unknown+expertise%2E+In+Y%2E+Bengio%2C+et+al%2E%2C+ed) 8. Zhou, Y. and Xue, N. (2012). PDTB-style discourse annotation of Chinese text. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 69–77. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhou%2C+Y%2E+and+Xue%2C+N%2E+%282012%29%2E+PDTB%2Dstyle+discourse+annotation+of+Chinese+text%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+)