---
title: "A Benchmark Dataset for Automatic Detection of Claims and"
description: "This paper introduces a benchmark corpus of claims and evidence tied to specific controversial topics… 8 references, cited by 70 papers here."
authors: ["Ehud Aharoni", "Anatoly Polnarov", "Tamar Lavee", "Daniel Hershcovich", "Ran Levy", "Ruty Rinott", "Dan Gutfreund", "Noam Slonim"]
year: 2014
venue: "Proceedings of the First Workshop on Argumentation Mining"
source: "https://aclanthology.org/W14-2109/"
doi: "10.3115/v1/w14-2109"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aharoni-2014-benchmark-dataset-automatic-detection-claims/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aharoni-2014-benchmark-dataset-automatic-detection-claims/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics

# A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics

**Authors:** [Ehud Aharoni](https://md.research.elamur.ai/free/en/authors/ehud-aharoni/), Anatoly Polnarov, Tamar Lavee, [Daniel Hershcovich](https://md.research.elamur.ai/free/en/authors/daniel-hershcovich/), [Ran Levy](https://md.research.elamur.ai/free/en/authors/ran-levy/), [Ruty Rinott](https://md.research.elamur.ai/free/en/authors/ruty-rinott/), Dan Gutfreund and [Noam Slonim](https://md.research.elamur.ai/free/en/authors/noam-slonim/) · **Year:** 2014 · **Venue:** [Proceedings of the First Workshop on Argumentation Mining](https://md.research.elamur.ai/free/en/venues/workshop-on-argumentation-mining/) ([Workshop on Argumentation Mining 2014](https://md.research.elamur.ai/free/en/venues/workshop-on-argumentation-mining/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W14-2109/) · [PDF](https://aclanthology.org/W14-2109.pdf) · [DOI 10.3115/v1/w14-2109](https://doi.org/10.3115/v1/w14-2109) · [OpenAlex](https://openalex.org/W2251448848)

**Citations:** 125 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/aharoni-2014-benchmark-dataset-automatic-detection-claims/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Argumentation mining has lacked high-quality annotated data. This paper introduces a benchmark corpus of claims and evidence tied to specific controversial topics, which earlier corpora such as AraucariaDB and the V/IP Corpus did not do. Rather than the full Toulmin model, the authors use a deliberately minimal scheme. A Context Dependent Claim (CDC) is a concise, general statement that backs or opposes a debate motion. A Context Dependent Evidence (CDE) is a passage that supports a given CDC with respect to that motion. Each CDE is tagged as Study, Expert, or Anecdotal. About 20 trained in-house annotators worked in the GATE environment on English Wikipedia articles. The pipeline had separate stages for search, detection, and confirmation, each with five people working independently, and an item was kept only if at least three confirmed it. The final dataset contains 2,683 argument elements across 33 motions. Pairwise kappa was moderate, and estimated recall was about 90%. Because the resource covers many topics, it can support topic-independent tools for finding arguments.

## Key points
- The 33 motions were sampled at random from the idebate.org database. Annotators labeled 586 Wikipedia articles from the April 3, 2012 dump and found 1,392 CDCs in 321 of them.
- Evidence annotation was completed for only 12 motions, covering 350 CDCs in 104 articles. This produced 1,291 CDEs: 431 Study, 516 Expert, and 529 Anecdotal. A CDE can carry several types; for example, 118 are both Study and Expert.
- A candidate counts as a CDC only if it meets all five criteria:
  - Strength
  - Generality
  - Phrasing
  - Keeping text spirit
  - Topic unity
- CDEs use the same criteria minus generality. Annotators could make one minor permitted fix, such as resolving an anaphor, and the corrected version is included in the data.
- Average pairwise Cohen's kappa was 0.39 for claim confirmation and 0.4 for evidence confirmation. Estimated average recall was 90.6% for CDCs and 90.0% for CDEs.

## Interesting ideas
- **Detect, then confirm.** One group of annotators casts a wide net for candidates, and a separate group votes on the pooled list. This trades some noise at the start for a majority filter at the end, which suits fuzzy annotation targets like stance or argumentativeness.
- **Rejections must name a criterion.** An annotator who rejects a candidate has to cite which rule it breaks. This makes disagreements easier to diagnose and gives concrete material for refining the guidelines.
- **Recall without a gold standard.** The authors assume each annotator independently finds some fixed fraction of the true items in an article. From the overlap between annotators' detections, they infer how many items exist in total and therefore how many were missed. The same approach could estimate coverage in any crowd- or team-based extraction task.
- **Agreement when annotators don't overlap fully.** Since no two annotators had identical assignments, kappa was averaged over annotator pairs, and pairs with fewer than 100 shared items were dropped. This is a practical fallback when standard multi-rater measures cannot be applied.
- **Consensus answer keys for ongoing feedback.** Experienced annotators first solved selected tasks independently, then agreed on a reference answer. Everyone else was compared against it and received a feedback report roughly every week. This keeps quality monitoring feasible even when full review of every task is too expensive.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hebrew University of Jerusalem](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/) — University, Jerusalem, Israel · 50 works · 9,147 citations
- [IBM Research - Haifa](https://md.research.elamur.ai/free/en/labs/ibm-research-haifa/) — Research institute, Haifa, Israel · 5 works · 1,112 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund and Noam Slonim. 2014. A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics. Proceedings of the First Workshop on Argumentation Mining.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aharoni2014benchmark,
  title = {{A Benchmark Dataset for Automatic Detection of Claims and Evidence in the Context of Controversial Topics}},
  author = {Ehud Aharoni, Anatoly Polnarov, Tamar Lavee, Daniel Hershcovich, Ran Levy, Ruty Rinott, Dan Gutfreund and Noam Slonim},
  booktitle = {Proceedings of the First Workshop on Argumentation Mining},
  year = {2014},
  doi = {10.3115/v1/w14-2109},
  url = {https://doi.org/10.3115/v1/w14-2109}
}
```

- [← previous Humans Require Context to Infer Ironic Intent (so Computers Probably do, too)](https://md.research.elamur.ai/free/en/papers/wallace-2014-humans-require-context-infer-ironic/)
- [next → Report on the 11th IWSLT evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2014-report-11th-iwslt-evaluation-campaign/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Austin J. Freeley and David L. Steinberg. 2008. Argumentation and Debate. Wadsworth, Belmont, California. [find](https://scholar.google.com/scholar?q=Austin+J%2E+Freeley+and+David+L%2E+Steinberg%2E+2008%2E+Argumentation+and+Debate%2E+Wadsworth%2C+Belmont%2C+California%2E)
2. Chris Reed. 2005. ―Preliminary Results from an Argument Corpus‖ in Proceedings of the IX Symposium on Social Communication, Santiago de Cuba, pp. 576-580. [find](https://scholar.google.com/scholar?q=Chris+Reed%2E+2005%2E+%E2%80%95Preliminary+Results+from+an+Argument+Corpus%E2%80%96+in+Proceedings+of+the+IX+Symposium+on+Social+Communication%2C+Santiago+de+Cuba%2C+pp%2E+576%2D580%2E)
3. J. Richard Landis and Gary G. Koch. 1977. ―The measurement of observer agreement for categorical data.‖ Biometrics 33:159-174. [find](https://scholar.google.com/scholar?q=J%2E+Richard+Landis+and+Gary+G%2E+Koch%2E+1977%2E+%E2%80%95The+measurement+of+observer+agreement+for+categorical+data%2E%E2%80%96+Biometrics+33%3A159%2D174%2E)
4. Kevid D. Ashley and Vern R. Walker. 2013. ―Toward Constructing Evidence-Based Legal Arguments Using Legal Decision Documents and Machine Learning‖ in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Law (ICAIL ’13), Rome, Italy, pp. 176-180. [find](https://scholar.google.com/scholar?q=Kevid+D%2E+Ashley+and+Vern+R%2E+Walker%2E+2013%2E+%E2%80%95Toward+Constructing+Evidence%2DBased+Legal+Arguments+Using+Legal+Decision+Documents+and+Machine+Learning%E2%80%96+in+Proceedings+of+the+Fourteenth+International+Confer)
5. Marie-Francine Moens, Erik Boiy, Raquel Mochales Palau, and Chris Reed. 2007. ―Automatic Detection of Arguments in Legal Texts‖ in Proceedings of the International Conference on AI & Law (ICAIL-2007), Stanford, CA, pp. 225-230. [find](https://scholar.google.com/scholar?q=Marie%2DFrancine+Moens%2C+Erik+Boiy%2C+Raquel+Mochales+Palau%2C+and+Chris+Reed%2E+2007%2E+%E2%80%95Automatic+Detection+of+Arguments+in+Legal+Texts%E2%80%96+in+Proceedings+of+the+International+Conference+on+AI+%26+Law+%28ICAIL%2D2007%29%2C)
6. Richard D. Rieke, Malcolm O. Sillars and Tarla Rai Peterson. 2012. Argumentation and Critical Decision Making (8e). Prentice Hall, USA. [find](https://scholar.google.com/scholar?q=Richard+D%2E+Rieke%2C+Malcolm+O%2E+Sillars+and+Tarla+Rai+Peterson%2E+2012%2E+Argumentation+and+Critical+Decision+Making+%288e%29%2E+Prentice+Hall%2C+USA%2E)
7. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni and Noam Slonim. ―[Context Dependent Claim Detection](https://md.research.elamur.ai/free/en/papers/levy-2014-context-dependent-claim-detection/).‖ Submitted Ruty Rinott, Lena Dankin, Carlos Alzate, Ehud Aharoni and Noam Slonim. "Show Me Your Evidence – an Automatic Method for Context Dependent Evidence Detection.‖ Submitted.
8. Zachary Seech. 2008. Writing Philosophy Papers (5 th edition). Wadsworth, Cengage Learning, Belmont, California. [find](https://scholar.google.com/scholar?q=Zachary+Seech%2E+2008%2E+Writing+Philosophy+Papers+%285+th+edition%29%2E+Wadsworth%2C+Cengage+Learning%2C+Belmont%2C+California%2E)

*Updated 24 October 2026*