---
title: "Design Choices for Crowdsourcing Implicit — Pyatkin 2023"
description: "Crowdsourced annotations can be skewed by how the task is phrased, not only by who annotates or by the label scheme."
authors: ["Valentina Pyatkin", "Frances Yung", "Merel C. J. Scholman", "Reut Tsarfaty", "Ido Dagan", "Vera Demberg"]
year: 2023
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.tacl-1.57/"
doi: "10.1162/tacl_a_00586"
arxiv: "2304.00815"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/pyatkin-2023-design-choices-crowdsourcing-implicit-discourse/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/2/) / Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

# Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design

**Authors:** [Valentina Pyatkin](https://md.research.elamur.ai/free/authors/valentina-pyatkin/), [Frances Yung](https://md.research.elamur.ai/free/authors/frances-yung/), [Merel C. J. Scholman](https://md.research.elamur.ai/free/authors/merel-c-j-scholman/), [Reut Tsarfaty](https://md.research.elamur.ai/free/authors/reut-tsarfaty/), [Ido Dagan](https://md.research.elamur.ai/free/authors/ido-dagan/), [Vera Demberg](https://md.research.elamur.ai/free/authors/vera-demberg/) · **Year:** 2023 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.tacl-1.57/) · [PDF](https://aclanthology.org/2023.tacl-1.57.pdf) · [DOI 10.1162/tacl\_a\_00586](https://doi.org/10.1162/tacl_a_00586) · [arXiv 2304.00815](https://arxiv.org/abs/2304.00815) · [OpenAlex](https://openalex.org/W4385859641)

**Citations:** 2 citations (openalex-snapshot · 2026-10-14) · [17](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Crowdsourced annotations can be skewed by how the task is phrased, not only by who annotates or by the label scheme. This study examines that "task design bias" for implicit discourse relations, a notoriously ambiguous annotation problem. The authors compared two crowdsourcing approaches that rely on natural language rather than abstract labels. In the DC method (Yung et al., 2019), workers insert a discourse connective. In the QA method (Pyatkin et al., 2020), workers write question–answer pairs. Both outputs were mapped to PDTB 3.0 senses.

They gathered ten judgments per item for 1,200 inter-sentential implicit relations across four domains and compared the resulting label distributions. Both methods yield broadly similar, valid label sets, but each favors particular senses, and the domain shifts these tendencies. Classification experiments show that crossing annotation methods hurts more than crossing domains. They also show that combining data from both methods helps, and that soft-label training generally beats hard-label training. The work argues that dataset builders and model developers should account for biases introduced by task design.

## Key points
- The data cover 1,200 implicit discourse relations from four domains, each annotated by both the DC and QA methods, with 10 crowd annotations per relation. All labels were mapped onto the PDTB 3.0 sense hierarchy, with belief and speech-act senses merged into their general senses.
- Each method agrees more with the reference label than with the other method. This suggests the two methods are complementary and capture different sense types.
- Classifiers were trained on DiscoGem plus 300 extra DC-annotated PDTB instances. Models trained on DC distributions always scored lower on QA test data than on DC test data, so the change of task mattered more than the change of domain.
- Training with a soft loss on label distributions usually matched or beat hard-loss training, in line with earlier findings reported by Uma et al. (2020) as well as by Peterson et al. (2019). Mixing data from both methods improved results on traditionally annotated data.
- The authors propose several improvements:
  - For QA, use question prefixes that start with a Wh-word.
  - For DC, let workers highlight argument spans and allow minor text edits.
  - Combine the methods, for example by using cheap DC annotation at scale and applying QA to high-entropy items.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 16 not yet.

- [Exploring Soft-Label Training for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/papers/costa-2024-exploring-soft-label-training-implicit/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{pyatkin2023design,
  title = {{Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design}},
  author = {Pyatkin, Valentina and Yung, Frances and Scholman, Merel C. J. and Tsarfaty, Reut and Dagan, Ido and Demberg, Vera},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00586},
  eprint = {2304.00815},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00586}
}
```

- [← previous Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://md.research.elamur.ai/free/papers/wang-2023-describe-explain-plan-select-interactive/)
- [next → Designing the LECOR Learner Corpus for Romanian](https://md.research.elamur.ai/free/papers/barbu-2023-designing-lecor-learner-corpus-romanian/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/2/)

## References

72 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahul Aralikatte, Matthew Lamm, Daniel Hardt, and Anders Søgaard. 2021. Ellipsis resolution as question answering: An evaluation. In 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 810–817, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.eacl-main.68 [↗](https://aclanthology.org/2021.eacl-main.68/) [our card](https://md.research.elamur.ai/free/papers/eacl-2021-aralikatte-ellipsis-resolution-as-question-answering/)
2. Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013 ACM, 2013(2013). [↗](https://doi.org/10.6084/m9.figshare.679997)
3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4): 555–596. https://doi.org/10.1162/coli.07-034-R2 [↗](https://doi.org/10.1162/coli.07-034-r2)
4. N. Asher. 1993. Reference to Abstract Objects in Discourse, volume 50. Kluwer, Norwell, MA, Dordrecht. https://doi.org/10.1007/978-94-011-1715-9 [↗](https://doi.org/10.1007/978-94-011-1715-9)
5. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.bppf-1.3 [↗](https://aclanthology.org/2021.bppf-1.3/) [our card](https://md.research.elamur.ai/free/papers/basile-2021-we-need-consider-disagreement-evaluation/)
6. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10 .18653/v1/D15-1075 [↗](https://aclanthology.org/D15-1075/)
7. Samuel Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843–4855, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.385 [↗](https://aclanthology.org/2021.naacl-main.385/)
8. Sven Buechel and Udo Hahn. 2017a. Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 578–585, Valencia, Spain. Association for Computational Linguistics. https://doi.org/10 .18653/v1/E17-2092 [↗](https://aclanthology.org/E17-2092/)
9. Sven Buechel and Udo Hahn. 2017b. Readers vs. writers vs. texts: Coping with different perspectives of text understanding in emotion annotation. In Proceedings of the 11th Linguistic Annotation Workshop, pages 1–12, Valencia, Spain. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W17-0801 [↗](https://aclanthology.org/W17-0801/)
10. Lynn Carlson and Daniel Marcu. 2001. Discourse tagging reference manual. ISI Technical Report ISI-TR-545, 54:1–56. [find](https://scholar.google.com/scholar?q=Lynn+Carlson+and+Daniel+Marcu%2E+2001%2E+Discourse+tagging+reference+manual%2E+ISI+Technical+Report+ISI%2DTR%2D545%2C+54%3A1%E2%80%9356%2E)
11. Nancy Chang, Russell Lee-Goldman, and Michael Tseng. 2016. Linguistic wisdom from the crowd. In Third AAAI Conference on Human Computation and Crowdsourcing. https://doi.org/10.1609/hcomp.v3i1.13266 [↗](https://doi.org/10.1609/hcomp.v3i1.13266)
12. Tongfei Chen, Zheng Ping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.774 [↗](https://aclanthology.org/2020.acl-main.774/) [our card](https://md.research.elamur.ai/free/papers/chen-2020-uncertain-natural-language-inference/)
13. John Joon Young Chung, Jean Y. Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S. Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–25. https://doi.org/10.1145/3359164 [↗](https://doi.org/10.1145/3359164)
14. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. https://doi.org/10.1177/001316446002000104 [↗](https://doi.org/10.1177/001316446002000104)
15. Alan Cowen, Disa Sauter, Jessica L. Tracy, and Dacher Keltner. 2019. Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression. Psychological Science in the Public Interest, 20(1):69–90. https://doi.org/10 .1177/1529100619850176, PubMed: 31313637 [↗](https://doi.org/10.1177/1529100619850176)
16. Marie-Catherine De Marneffe, Christopher D. Manning, and Christopher Potts. 2012. Did it happen? The pragmatic complexity of veridicality assessment. Computational Linguistics, 38(2):301–333. https://doi.org/10.1162/COLI_a_00097 [↗](https://doi.org/10.1162/coli_a_00097) [our card](https://md.research.elamur.ai/free/papers/marneffe-2012-did-it-happen-pragmatic-complexity/)
17. Vera Demberg, Merel C. J. Scholman, and Fatemeh Torabi Asr. 2019. How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations. Dialogue & Discourse, 10(1):87–135. https://doi.org/10.5087/dad .2019.104 [↗](https://doi.org/10.5087/dad)
18. Mark Dı́az, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–14. https://doi.org/10.1145/3173574.3173986 [↗](https://doi.org/10.1145/3173574.3173986)
19. Anca Dumitrache. 2015. Crowdsourcing disagreement for collecting semantic annotation. In European Semantic Web Conference, pages 701–710. Springer. https://doi.org/10.1007/978-3-319-18818-8 43 [↗](https://doi.org/10.1007/978-3-319-18818-8)
20. Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, and Chris Welty. 2018. CrowdTruth 2.0: Quality metrics for crowdsourcing with disagreement. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing, and Short Paper 1st Workshop on Disentangling the Relation Between Crowdsourcing and Bias Management, SAD+ CrowdBias 2018, pages 11–18. CEUR-WS. [↗](https://doi.org/10.48550/arxiv.1808.06080)
21. Anca Dumitrache, Oana Inel, Benjamin Timmermans, Carlos Ortiz, Robert-Jan Sips, Lora Aroyo, and Chris Welty. 2021. Empirical methodology for crowdsourcing ground truth. Semantic Web, 12(3):403–421. https://doi.org/10.3233/SW -200415 [↗](https://doi.org/10.3233/SW)
22. Yanai Elazar, Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2022. Text-based np enrichment. Transactions of the Association for Com putational Linguistics, 10:764–784. https://doi.org/10.1162/tacl a 00488 [↗](https://doi.org/10.1162/tacl)
23. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Katrin+Erk+and+Diana+McCarthy%2E+2009%2E+Graded+word+sense+assignment%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+440%E2%80%93449%2C+Singapore%2E+Association+for+)
24. Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2021. Did they answer? Subjective acts and intents in conversational discourse. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1626–1644, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.129 [↗](https://aclanthology.org/2021.naacl-main.129/)
25. Nicholas Fitzgerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale qa-srl parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18 -1191 [↗](https://doi.org/10.18653/v1/p18) [our card](https://md.research.elamur.ai/free/papers/fitzgerald-2018-large-scale-qa-srl-parsing/)
26. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2006.03654) [our card](https://md.research.elamur.ai/free/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
27. Yufang Hou. 2020. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.132 [↗](https://aclanthology.org/2020.acl-main.132/) [our card](https://md.research.elamur.ai/free/papers/hou-2020-bridging-anaphora-resolution-as-question/)
28. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Jurgens%2E+2013%2E+Embracing+ambiguity%3A+A+comparison+of+annotation+methodologies+for+crowdsourcing+word+sense+labels%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Assoc)
29. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1120–1130, Atlanta, Georgia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Taylor+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+whom+to+trust+with+MACE%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/papers/hovy-2013-learning-whom-trust-mace/)
30. Daisuke Kawahara, Yuichiro Machida, Tomohide Shibata, Sadao Kurohashi, Hayato Kobayashi, and Manabu Sassano. 2014. Rapid development of a corpus with discourse annotations using two-stage crowdsourcing. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 269–278, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Daisuke+Kawahara%2C+Yuichiro+Machida%2C+Tomohide+Shibata%2C+Sadao+Kurohashi%2C+Hayato+Kobayashi%2C+and+Manabu+Sassano%2E+2014%2E+Rapid+development+of+a+corpus+with+discourse+annotations+using+two%2Dstage+crowdsourcin)
31. Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019. Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–12, New York, NY. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Christoph+Hube%2C+Besnik+Fetahu%2C+and+Ujwal+Gadiraju%2E+2019%2E+Understanding+and+mitigating+worker+biases+in+the+crowdsourced+collection+of+subjective+judgments%2E+In+Proceedings+of+the+2019+CHI+Conference+on)
32. Yudai Kishimoto, Shinnosuke Sawada, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2018. Improving crowdsourcing-based annotation of Japanese discourse relations. In LREC. [find](https://scholar.google.com/scholar?q=Yudai+Kishimoto%2C+Shinnosuke+Sawada%2C+Yugo+Murawaki%2C+Daisuke+Kawahara%2C+and+Sadao+Kurohashi%2E+2018%2E+Improving+crowdsourcing%2Dbased+annotation+of+Japanese+discourse+relations%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/papers/kishimoto-2018-improving-crowdsourcing-based-annotation-japanese/)
33. Terne Sasha Thorn Jakobsen, Maria Barrett, Anders Søgaard, and David Lassen. 2022. The sensitivity of annotator bias to task definitions in argument mining. In Proceedings of the 16th Lingusitic Annotation Workshop (LAW-XVI) within LREC2022, pages 44–61, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Terne+Sasha+Thorn+Jakobsen%2C+Maria+Barrett%2C+Anders+S%C3%B8gaard%2C+and+David+Lassen%2E+2022%2E+The+sensitivity+of+annotator+bias+to+task+definitions+in+argument+mining%2E+In+Proceedings+of+the+16th+Lingusitic+Annot)
34. Nanjiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics, 10:1357–1374. https://doi.org/10 .1162/tacl_a_00523 [↗](https://doi.org/10.1162/tacl_a_00523) [our card](https://md.research.elamur.ai/free/papers/jiang-2022-investigating-reasons-disagreement-natural-language/)
35. Youxuan Jiang, Jonathan K. Kummerfeld, and Walter Lasecki. 2017. Understanding task design trade-offs in crowdsourced paraphrase collection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 103–109, Vancouver, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/P17 -2017 [↗](https://doi.org/10.18653/v1/p17)
36. Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, and Junyi Jessy Li. 2021. Discourse comprehension: A question answering framework to represent sentence connections. arXiv preprint arXiv:2111.00701. [↗](https://arxiv.org/abs/2111.00701)
37. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of MT Summit X, pages 79–86. Phuket, Thailand. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+MT+Summit+X%2C+pages+79%E2%80%9386%2E+Phuket%2C+Thailand%2E)
38. Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296–3315, online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.296 [↗](https://aclanthology.org/2020.findings-emnlp.296/)
39. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. https://doi.org/10.1515/text.1.1988.8.3.243 [↗](https://doi.org/10.1515/text.1.1988.8.3.243) [our card](https://md.research.elamur.ai/free/papers/enud-1988-mann-rhetorical-structure-theory-functional/)
40. Christopher D. Manning. 2006. Local textual inference: It’s hard to circumscribe, but you know it when you see it—and NLP needs it. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning%2E+2006%2E+Local+textual+inference%3A+It%E2%80%99s+hard+to+circumscribe%2C+but+you+know+it+when+you+see+it%E2%80%94and+NLP+needs+it%2E)
41. Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3659–3668, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [↗](https://doi.org/10.22028/d291-38835)
42. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Am-bigqa: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.466 [↗](https://aclanthology.org/2020.emnlp-main.466/)
43. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.734 [↗](https://aclanthology.org/2020.emnlp-main.734/) [our card](https://md.research.elamur.ai/free/papers/nie-2020-what-can-we-learn-collective/)
44. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. https://doi.org/10.1162/tacl_a_00185 [↗](https://doi.org/10.1162/tacl_a_00185)
45. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. https://doi.org/10.1162/tacl_a_00293 [↗](https://doi.org/10.1162/tacl_a_00293) [our card](https://md.research.elamur.ai/free/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/)
46. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626. https://doi.org/10 .1109/ICCV.2019.00971 [↗](https://doi.org/10)
47. Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 507–511, Baltimore, Maryland. Association for Computational Linguistics. https://doi.org/10.3115/v1/P14 -2083 [↗](https://doi.org/10.3115/v1/p14)
48. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83. https://doi.org/10 .3115/1608829.1608840 [↗](https://doi.org/10) [our card](https://md.research.elamur.ai/free/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/)
49. Massimo Poesio, Patrick Sturt, Ron Artstein, and Ruth Filik. 2006. Underspecification and anaphora: Theoretical issues and preliminary evidence. Discourse processes, 42(2):157–175. https://doi.org/10.1207/s15326950dp4202 4 [↗](https://doi.org/10.1207/s15326950dp4202_4)
50. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.law-1.14 [↗](https://aclanthology.org/2021.law-1.14/) [our card](https://md.research.elamur.ai/free/papers/prabhakaran-2021-releasing-annotator-level-labels-information/)
51. Valentina Pyatkin, Ayal Klein, Reut Tsarfaty, and Ido Dagan. 2020. QADiscourse-Discourse Relations as QA Pairs: Representation, crowdsourcing and baselines. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2804–2819, Online. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2020.emnlp-main.224 [↗](https://aclanthology.org/2020.emnlp-main.224/) [our card](https://md.research.elamur.ai/free/papers/pyatkin-2020-qadiscourse-discourse-relations-as-qa/)
52. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2124 [↗](https://aclanthology.org/P18-2124/) [our card](https://md.research.elamur.ai/free/papers/rajpurkar-2018-know-what-you-don-t/)
53. Ines Rehbein, Merel Scholman, and Vera Demberg. 2016. Annotating discourse relations in spoken language: A comparison of the PDTB and CCR frameworks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1039–1046, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Ines+Rehbein%2C+Merel+Scholman%2C+and+Vera+Demberg%2E+2016%2E+Annotating+discourse+relations+in+spoken+language%3A+A+comparison+of+the+PDTB+and+CCR+frameworks%2E+In+Proceedings+of+the+Tenth+International+Conferen) [our card](https://md.research.elamur.ai/free/papers/rehbein-2016-annotating-discourse-relations-spoken-language/)
54. Stefan Riezler. 2014. On the problem of theoretical terms in empirical computational linguistics. Computational Linguistics, 40(1):235–245. https://doi.org/10.1162/COLI_a_00182 [↗](https://doi.org/10.1162/coli_a_00182)
55. Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher Clark, Annie Louis, and Bonnie Webber. 2016. Filling in the blanks in understanding discourse adverbials: Consistency, conflict, and context-dependence in a crowdsourced elicitation task. In Proceedings of the 10th Linguistic Annotation Workshop (LAW X), pages 49–58, Berlin, Germany. https://doi.org/10.18653/v1/W16 -1707 [↗](https://doi.org/10.18653/v1/w16)
56. Ted J. M. Sanders, Wilbert P. M. S. Spooren, and Leo G. M. Noordman. 1992. Toward a taxonomy of coherence relations. Discourse Processes, 15(1):1–35. https://doi.org/10.1080/01638539209544800 [↗](https://doi.org/10.1080/01638539209544800)
57. Merel C. J. Scholman and Vera Demberg. 2017. Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task. In Proceedings of the 11th Linguistic Annotation Workshop (LAW), pages 24–33, Valencia, Spain. Association for Computational Linguistics. https://doi.org/10.18653/v1/W17 -0803 [↗](https://doi.org/10.18653/v1/w17)
58. Merel C. J. Scholman, Tianai Dong, Frances Yung, and Vera Demberg. 2022a. Discogem: A crowdsourced corpus of genre-mixed implicit discourse relations. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC’22), Marseille, France. European Language Resources Association (ELRA). [↗](https://doi.org/10.22028/d291-38852) [our card](https://md.research.elamur.ai/free/papers/scholman-2022-discogem-crowdsourced-corpus-genre-mixed/)
59. Merel C. J. Scholman, Valentina Pyatkin, Frances Yung, Ido Dagan, Reut Tsarfaty, and Vera Demberg. 2022b. Design choices in crowdsourcing discourse relation annotations: The effect of worker selection and training. In Proceedings of the Thirteenth International Conference on Language Resources and Evaluation (LREC’22), Marseille, France. European Language Resources Association (ELRA). [↗](https://doi.org/10.22028/d291-38851) [our card](https://md.research.elamur.ai/free/papers/scholman-2022-design-choices-crowdsourcing-discourse-relation/)
60. Wei Shi and Vera Demberg. 2019. Learning to explicitate connectives with Seq2Seq network for implicit discourse relation classification. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 188–199, Gothenburg, Sweden. Association for Computational Linguistics. https://doi.org/10 .18653/v1/W19-0416 [↗](https://aclanthology.org/W19-0416/) [our card](https://md.research.elamur.ai/free/papers/shi-2019-learning-explicitate-connectives-seq2seq-network/)
61. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 254–263, Honolulu, Hawaii. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rion+Snow%2C+Brendan+O%E2%80%99Connor%2C+Daniel+Jurafsky%2C+and+Andrew+Y%2E+Ng%2E+2008%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F+Evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+Conference+on+E)
62. Wilbert P. M. S. Spooren and Liesbeth Degand. 2010. Coding coherence relations: Reliability and validity. Corpus Linguistics and Linguistic Theory, 6(2):241–266. https://doi.org/10.1515/cllt.2010.009 [↗](https://doi.org/10.1515/cllt.2010.009)
63. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 173–177. https://doi.org/10.1609/hcomp.v8i1.7478 [↗](https://doi.org/10.1609/hcomp.v8i1.7478)
64. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. https://doi.org/10.1613/jair.1.12752 [↗](https://doi.org/10.1613/jair.1.12752)
65. Zeerak Waseem. 2016. Are you a racist or am I seeing things? Annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas. Association for Computational Linguistics. https://doi.org/10.18653/v1/W16 -5618 [↗](https://doi.org/10.18653/v1/w16)
66. Bonnie Webber. 2009. Genre distinctions for discourse in the Penn TreeBank. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 674–682. https://doi.org/10.3115/1690219.1690240 [↗](https://doi.org/10.3115/1690219.1690240)
67. Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The Penn Discourse Treebank 3.0 annotation manual. Philadelphia, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2C+Rashmi+Prasad%2C+Alan+Lee%2C+and+Aravind+Joshi%2E+2019%2E+The+Penn+Discourse+Treebank+3%2E0+annotation+manual%2E+Philadelphia%2C+University+of+Pennsylvania%2E) [our card](https://md.research.elamur.ai/free/papers/miltsakaki-2004-penn-discourse-treebank/)
68. Ogrodniczuk. 2019. Ted multilingual discourse bank (TED-MDB): A parallel corpus annotated in the PDTB style. Language Resources and Evaluation, 1–27. [find](https://scholar.google.com/scholar?q=Ogrodniczuk%2E+2019%2E+Ted+multilingual+discourse+bank+%28TED%2DMDB%29%3A+A+parallel+corpus+annotated+in+the+PDTB+style%2E+Language+Resources+and+Evaluation%2C+1%E2%80%9327%2E)
69. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2020.emnlp-demos.6 [↗](https://aclanthology.org/2020.emnlp-demos.6/) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/)
70. Deniz Zeyrek, Amália Mendes, Yulia Grishina, Murathan Kurfalı, Samuel Gibbon, and Maciej Ogrodniczuk. 2020. Ted multilingual discourse bank (TED-MDB): A parallel corpus annotated in the PDTB style. Language Resources and Evaluation, 54(2):587–613. https://doi.org/10.1007/s10579-019-09445-9 [↗](https://doi.org/10.1007/s10579-019-09445-9)
71. Frances Yung, Vera Demberg, and Merel Scholman. 2019. Crowdsourcing discourse relation annotations by a two-step connective insertion task. In Proceedings of the 13th Linguistic Annotation Workshop, pages 16–25, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-4003 Deniz Zeyrek, Amália Mendes, Yulia Grishina, Murathan Kurfalı, Samuel Gibbon, and Maciej Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7620–7632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.emnlp-main.601 [↗](https://aclanthology.org/W19-4003/) [our card](https://md.research.elamur.ai/free/papers/yung-2019-crowdsourcing-discourse-relation-annotations-two/)
72. Šárka Zikánová, Jiřı́ Mı́rovskỳ, and Pavlı́na Synková. 2019. Explicit and implicit discourse relations in the Prague Discourse Treebank. In Text, Speech, and Dialogue: 22nd International Conference, TSD 2019, Ljubljana, Slovenia, September 11–13, 2019, Proceedings 22, pages 236–248. Springer. https://doi.org/10.1007/978-3-030-27947-9_20 [↗](https://doi.org/10.1007/978-3-030-27947-9_20)