---
title: "Automatic Argument Quality Assessment - New Datasets and"
description: "The paper tackles automatic estimation of how good an argument is, a task often seen as subjective… 26 references, cited by 62 papers here."
authors: ["Assaf Toledo", "Shai Gretz", "Edo Cohen-Karlik", "Roni Friedman", "Elad Venezian", "Dan Lahav", "Michal Jacovi", "Ranit Aharonov", "Noam Slonim"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1564/"
doi: "10.18653/v1/d19-1564"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/toledo-2019-automatic-argument-quality-assessment-new/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/toledo-2019-automatic-argument-quality-assessment-new/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Automatic Argument Quality Assessment - New Datasets and Methods

# Automatic Argument Quality Assessment - New Datasets and Methods

**Authors:** [Assaf Toledo](https://md.research.elamur.ai/free/en/authors/assaf-toledo/), Shai Gretz, Edo Cohen-Karlik, [Roni Friedman](https://md.research.elamur.ai/free/en/authors/roni-friedman/), Elad Venezian, [Dan Lahav](https://md.research.elamur.ai/free/en/authors/dan-lahav/), Michal Jacovi, [Ranit Aharonov](https://md.research.elamur.ai/free/en/authors/ranit-aharonov/) and [Noam Slonim](https://md.research.elamur.ai/free/en/authors/noam-slonim/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1564/) · [PDF](https://aclanthology.org/D19-1564.pdf) · [DOI 10.18653/v1/d19-1564](https://doi.org/10.18653/v1/d19-1564) · [OpenAlex](https://openalex.org/W2970566229)

**Citations:** 60 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/toledo-2019-automatic-argument-quality-assessment-new/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles automatic estimation of how good an argument is, a task often seen as subjective and short on large, clean data. Within IBM's Speech by Crowd effort, the authors collected 6,257 arguments on 22 motions through a purpose-built interface. Contributors ranged from novice and expert debate club members to a general audience. Every argument got an explicit quality score from crowd judgments. Separately, about 14k same-motion pairs were labeled for which member is stronger. The two labeling schemes, absolute and relative, turned out to agree to a surprising degree. The authors also built neural models on the BERT language model (Devlin et al., 2018) for two tasks. For ranking single arguments, the results match the state of the art. For deciding which argument in a pair is better, they clearly beat earlier methods. The cleaned datasets are released publicly. This gives the community a resource more than five times larger than UKPRank. It is also the first dataset that lets researchers directly compare pairwise and individual quality labels.

## Key points
- **Released datasets:**
  - **IBM-ArgQ-6.3kArgs**: all arguments, each with a score in [0, 1].
  - **IBM-ArgQ-14kPairs**: pairs with a preference label.
  - **IBM-ArgQ-5.3kArgs (IBMRank)**: cleansed subset of the scored arguments.
  - **IBM-ArgQ-9.1kPairs (IBMPairs)**: cleansed subset of the pairs.
- **Collection design:**
  - 11 controversial concepts (e.g., Doping in Sports, Flu vaccination, Autonomous Cars).
  - Each concept was framed as two opposing motions.
  - The UI rejected arguments shorter than 8 or longer than 36 words.
- **Annotation setup:**
  - Labeling was done on Figure Eight, with 15–17 annotators per instance.
  - Individual quality was a binary question: would you advise a friend writing a speech to use this argument unchanged?
  - The final score is the share of "yes" answers.
- **Quality control:**
  - Each item began with a stance test question (pro/con) to confirm the annotator had read it.
  - Pairwise Cohen's kappa was computed for annotators sharing at least 50 judgments.
- **Comparisons:** The models were benchmarked against prior work on the UKP datasets (UKPRank, UKPConvArgAll). The GPPL regressor could not be used on IBMRank, because it needs pairwise labels to produce scores.

## Interesting ideas
- **Collect actively instead of scraping.** Gathering arguments through a dedicated interface gives control over content and noise that debate-portal scraping lacks. Built-in guidance, an example argument, and length limits all help. This could improve any crowdsourced text corpus.
- **Ask about a concrete use, not abstract quality.** Annotators judged whether an argument would work unchanged in a persuasive speech, while setting aside their own opinion. Anchoring a subjective judgment in a practical scenario makes it easier to answer. It also captures delivery aspects such as typos and wording efficiency.
- **Use comprehension checks as embedded test questions.** A simple stance question before each judgment forces careful reading. It also produces data for filtering annotators and selecting which items to label next.
- **Combine cheap absolute labels with targeted pairwise labels.** Individual labels scale linearly, while pairs scale quadratically. Pairs with a large gap in individual scores already agree with pairwise judgments, so costly pair labeling can be reserved for close calls. This resembles active learning and applies to any ranking annotation budget.
- **Analyze the most confident mistakes.** Examining the errors the model was most sure about revealed systematic blind spots. These include strong arguments with poor delivery, provocative or ungrounded claims, and arguments annotators preferred regardless of the opponent. This analysis technique transfers to most classification work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 85 works · 66,118 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤔 Philosophy](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/philosophy/) · 211 works

## Cite

Cite as: Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov and Noam Slonim. 2019. Automatic Argument Quality Assessment - New Datasets and Methods. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{toledo2019automatic,
  title = {{Automatic Argument Quality Assessment - New Datasets and Methods}},
  author = {Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov and Noam Slonim},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1564},
  url = {https://doi.org/10.18653/v1/d19-1564}
}
```

- [← previous Better Rewards Yield Better Summaries: Learning to Summarise Without References](https://md.research.elamur.ai/free/en/papers/bohm-2019-better-rewards-yield-better-summaries/)
- [next → EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nora Aranberri, Gorka Labaka, Arantza Dı́az de Ilarraza, and Kepa Sarasola. 2017. Ebaluatoia: crowd evaluation for english–basque machine translation. Language Resources and Evaluation, 51(4):1053– 1084. [find](https://scholar.google.com/scholar?q=Nora+Aranberri%2C+Gorka+Labaka%2C+Arantza+D%C4%B1%CC%81az+de+Ilarraza%2C+and+Kepa+Sarasola%2E+2017%2E+Ebaluatoia%3A+crowd+evaluation+for+english%E2%80%93basque+machine+translation%2E+Language+Resources+and+Evaluation%2C+51%284%29%3A1053%E2%80%93+10)
2. Aristotle, G.A. Kennedy, and G.A. Kennedy. 1991. On Rhetoric: A Theory of Civic Discourse. Oxford University Press. [find](https://scholar.google.com/scholar?q=Aristotle%2C+G%2EA%2E+Kennedy%2C+and+G%2EA%2E+Kennedy%2E+1991%2E+On+Rhetoric%3A+A+Theory+of+Civic+Discourse%2E+Oxford+University+Press%2E)
3. Roy Bar-Haim, Indrajit Bhattacharya, Francesco Din-uzzo, Amrita Saha, and Noam Slonim. 2017. [Stance classification of context-dependent claims](https://md.research.elamur.ai/free/en/papers/haim-2017-stance-classification-context-dependent-claims/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261, Valenc…
4. T Bench-Capon, K Atkinson, and Peter McBurney. 2009. Altruism and agents: an argumentation based approach to designing agent decision mechanisms, pages 1073 – 1080. Unknown Publisher. [find](https://scholar.google.com/scholar?q=T+Bench%2DCapon%2C+K+Atkinson%2C+and+Peter+McBurney%2E+2009%2E+Altruism+and+agents%3A+an+argumentation+based+approach+to+designing+agent+decision+mechanisms%2C+pages+1073+%E2%80%93+1080%2E+Unknown+Publisher%2E)
5. Xi Chen, Paul N. Bennett, Kevyn Collins-Thompson, and Eric Horvitz. 2013. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, WSDM ’13, pages 193–202, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Paul+N%2E+Bennett%2C+Kevyn+Collins%2DThompson%2C+and+Eric+Horvitz%2E+2013%2E+Pairwise+ranking+aggregation+in+a+crowdsourced+setting%2E+In+Proceedings+of+the+Sixth+ACM+International+Conference+on+Web+Search)
6. Cohen. 1960. A coefficient of agreement for nominal scales. Educ Psychol Meas, pages 37–46. [find](https://scholar.google.com/scholar?q=Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educ+Psychol+Meas%2C+pages+37%E2%80%9346%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805.
8. Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, and Noam Slonim. 2019. Are you convinced? choosing the more convincing evidence with a Siamese network. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 967–976,… [find](https://scholar.google.com/scholar?q=Martin+Gleize%2C+Eyal+Shnarch%2C+Leshem+Choshen%2C+Lena+Dankin%2C+Guy+Moshkowich%2C+Ranit+Aharonov%2C+and+Noam+Slonim%2E+2019%2E+Are+you+convinced%3F+choosing+the+more+convincing+evidence+with+a+Siamese+network%2E+In+Pro)
9. Ivan Habernal and Iryna Gurevych. 2016a. [What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation](https://md.research.elamur.ai/free/en/papers/habernal-2016-what-makes-convincing-argument-empirical/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214–1223, Austin, Texas. Association for Co…
10. Ivan Habernal and Iryna Gurevych. 2016b. [Which argument is more convincing? analyzing and predicting convincingness of web arguments using bidirectional lstm](https://md.research.elamur.ai/free/en/papers/habernal-2016-which-argument-is-more-convincing/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599. Associat…
11. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. [Learning whom to trust with MACE](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, Georgia. Associati…
12. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. [Context dependent claim detection](https://md.research.elamur.ai/free/en/papers/levy-2014-context-dependent-claim-detection/). In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1489– 1500, Dublin, Ireland. Dublin City University and Associati…
13. Marco Lippi and Paolo Torroni. 2016. Argumentation mining: State of the art and emerging trends. ACM Trans. Internet Technol., 16(2):10:1–10:25. [find](https://scholar.google.com/scholar?q=Marco+Lippi+and+Paolo+Torroni%2E+2016%2E+Argumentation+mining%3A+State+of+the+art+and+emerging+trends%2E+ACM+Trans%2E+Internet+Technol%2E%2C+16%282%29%3A10%3A1%E2%80%9310%3A25%2E)
14. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In In EMNLP.
15. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for squad](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). CoRR, abs/1806.03822.
16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. Association for Computational Linguistics.
17. Chris Reed. 2016. Proceedings of the third workshop on argument mining (argmining2016). In Proceedings of the Third Workshop on Argument Mining (ArgMining2016). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Reed%2E+2016%2E+Proceedings+of+the+third+workshop+on+argument+mining+%28argmining2016%29%2E+In+Proceedings+of+the+Third+Workshop+on+Argument+Mining+%28ArgMining2016%29%2E+Association+for+Computational+Linguisti)
18. Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. [Classification and clustering of arguments with contextualized word embeddings](https://md.research.elamur.ai/free/en/papers/reimers-2019-classification-clustering-arguments-contextualized-word/). CoRR, abs/1906.09821.
19. Ruty Rinott, Lena Dankin, Carlos Alzate Perez, Mitesh M. Khapra, Ehud Aharoni, and Noam Slonim. 2015. [Show me your evidence - an automatic method for context dependent evidence detection](https://md.research.elamur.ai/free/en/papers/rinott-2015-show-me-your-evidence-automatic/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 440–450, Lisbon…
20. Burr Settles. 2009. [Active learning literature survey](https://md.research.elamur.ai/free/en/papers/settles-2009-active-learning-literature-survey/). Computer Sciences Technical Report 1648, University of Wisconsin–Madison.
21. Edwin D Simpson and Iryna Gurevych. 2018. Finding convincing arguments using scalable bayesian preference learning. Transactions of the Association for Computational Linguistics, 6:357–371. [find](https://scholar.google.com/scholar?q=Edwin+D+Simpson+and+Iryna+Gurevych%2E+2018%2E+Finding+convincing+arguments+using+scalable+bayesian+preference+learning%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+6%3A357%E2%80%93371%2E)
22. Christian Stab and Iryna Gurevych. 2014. [Annotating argument components and relations in persuasive essays](https://md.research.elamur.ai/free/en/papers/stab-2014-annotating-argument-components-relations-persuasive/). In COLING, pages 1501–1510. ACL.
23. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017a. [Computational argumentation quality assessment in natural language](https://md.research.elamur.ai/free/en/papers/wachsmuth-2017-computational-argumentation-quality-assessment-natural/). In Proceedings of the 15th Conference of the European Chapter of the Association for Compu…
24. Henning Wachsmuth, Martin Potthast, Khalid Al Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. 2017b. Building an argument search engine for the web. In ArgMining@EMNLP, pages 49–59. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Henning+Wachsmuth%2C+Martin+Potthast%2C+Khalid+Al+Khatib%2C+Yamen+Ajjour%2C+Jana+Puschmann%2C+Jiani+Qu%2C+Jonas+Dorsch%2C+Viorel+Morari%2C+Janek+Bevendorff%2C+and+Benno+Stein%2E+2017b%2E+Building+an+argument+search+engine+)
25. Douglas Walton, Chris Reed, and Fabrizio Macagno. 2008. Argumentation Schemes. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Douglas+Walton%2C+Chris+Reed%2C+and+Fabrizio+Macagno%2E+2008%2E+Argumentation+Schemes%2E+Cambridge+University+Press%2E)
26. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). CoRR, abs/1804.07461.

*Updated 24 October 2026*