---
title: "A Corpus for Research on Deliberation and — Walker 2012"
description: "Empirical research on argumentation has been limited by a shortage of freely available, unscripted argumentative dialogue."
authors: ["Marilyn Walker", "Jean Fox Tree", "Pranav Anand", "Rob Abbott", "Joseph King"]
year: 2012
venue: "Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)"
source: "https://aclanthology.org/L12-1643/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/walker-2012-corpus-research-deliberation-debate/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/walker-2012-corpus-research-deliberation-debate/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2012](https://md.research.elamur.ai/free/papers/2012/) / A Corpus for Research on Deliberation and Debate

# A Corpus for Research on Deliberation and Debate

**Authors:** [Marilyn Walker](https://md.research.elamur.ai/free/authors/marilyn-walker/), [Jean Fox Tree](https://md.research.elamur.ai/free/authors/jean-fox-tree/), [Pranav Anand](https://md.research.elamur.ai/free/authors/pranav-anand/), Rob Abbott, Joseph King · **Year:** 2012 · **Venue:** Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L12-1643/) · [PDF](https://aclanthology.org/L12-1643.pdf) · [OpenAlex](https://openalex.org/W1733704070)

**Citations:** 223 citations (openalex-snapshot · 2026-10-14) · [128](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Empirical research on argumentation has been limited by a shortage of freely available, unscripted argumentative dialogue. The authors address this gap with the Internet Argument Corpus (IAC). It was scraped from the political debate forum 4forums.com and contains 390,704 posts in 11,800 discussions by 3,317 authors. Beyond the raw text, the corpus keeps the forum structure: reply trees, quotes, formatting, timestamps, polls and emoticons. The team also recovered the source posts of quotes that lacked links.

A topic-labeled extract of 2,866 threads (130,206 posts) covers contentious issues such as abortion, evolution and gun control. Subsets of this extract were annotated through Amazon Mechanical Turk for several dialogic dimensions:
- agreement
- cordiality versus nastiness
- audience direction
- attack
- assertiveness versus questioning
- emotional versus factual argument
- sarcasm

As a demonstration, the authors analyse how these dimensions relate to each other. Sarcasm turns out to be emotional and personal, but it is not necessarily a sign of disagreement. The resource matters because it supplies large-scale, naturally occurring debate data for work on stance, agreement and sarcasm.

## Key points
- The IAC holds 390,704 posts in 11,800 threads by 3,317 authors from 4forums.com, and 72.3% of posts contain at least one quote.
- Two annotation datasets were built:
  - 10,003 quote-response (Q-R) pairs
  - 6,797 three-post chains (P123), each with three (context, response) pairs: P1–P2, P2–P3 and the non-adjacent P1–P3
- Each Mechanical Turk HIT contained seven context-response pairs. Pairs from the same discussion were not shown together within a HIT.
- The sampling deliberately favoured discourse-marker keywords:
  - 5,000 Q-R pairs open with a marker term
  - 2,003 pairs have a term within the first 10 tokens
  - 3,000 pairs have no such term
  - As a result, only about 3,000 Q-R examples can reproduce a natural distribution.
- Across topic-level aggregates, sarcasm correlates strongly with:
  - nastiness (r(16) = .87)
  - questioning rather than asserting (.76)
  - addressing a single poster (.74)
  - emotionality (.62)

## Interesting ideas
- **Rebuilding missing reply links.** Tracing quotes back to their source posts produced a more accurate reply structure than the original website shows. Any scraped forum corpus could gain precise conversational context this way.
- **Quote-response pairs as the unit of annotation.** Splitting posts into the quoted text and the reply that follows yields tightly scoped context. This makes dialogic judgements, such as agreement, easier for crowd workers.
- **A non-adjacent control pair.** Annotators also judged P1 against P3, which is not a direct reply. This tests whether stance relations are transitive and whether surface wording or actual content drives judgements, a check that could be reused in other dialogue annotation projects.
- **Biased sampling with a recoverable natural subset.** Oversampling keyword-initial items supports studies of specific phenomena, while a reserved keyword-free portion still allows a natural-distribution sample. The authors explicitly warn that users must resample, which is good documentation practice.
- **Scalar rather than binary labels.** Dimensions such as nice–nasty and emotional–factual were rated on scales and thresholded only later. This keeps graded information that supports both correlation analyses and classification.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

128 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 110 not yet.

- [DAICT: A Dialectal Arabic Irony Corpus Extracted from Twitter](https://md.research.elamur.ai/free/papers/abbes-2020-daict-dialectal-arabic-irony-corpus/)
- [Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations](https://md.research.elamur.ai/free/papers/allaway-2020-zero-shot-stance-detection-dataset/)
- [Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection](https://md.research.elamur.ai/free/papers/arakelyan-2023-topic-guided-sampling-data-efficient/)
- [Acquired TASTE: Multimodal Stance Detection with Textual and Structural Embeddings](https://md.research.elamur.ai/free/papers/barel-2025-acquired-taste-multimodal-stance-detection/)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://md.research.elamur.ai/free/papers/felbo-2017-millions-emoji-occurrences-learn-any/)
- [Sarcasm Analysis Using Conversation Context](https://md.research.elamur.ai/free/papers/ghosh-2018-sarcasm-analysis-conversation-context/)
- [Cross-Domain Label-Adaptive Stance Detection](https://md.research.elamur.ai/free/papers/hardalov-2021-cross-domain-label-adaptive-stance/)
- [Generalizable Sarcasm Detection is Just Around the Corner, of Course!](https://md.research.elamur.ai/free/papers/jang-2024-generalizable-sarcasm-detection-is-just/)
- [Human Rationales as Attribution Priors for Explainable Stance Detection](https://md.research.elamur.ai/free/papers/jayaram-2021-human-rationales-as-attribution-priors/)
- [Harnessing Context Incongruity for Sarcasm Detection](https://md.research.elamur.ai/free/papers/joshi-2015-harnessing-context-incongruity-sarcasm-detection/)
- [How Do Cultural Differences Impact the Quality of Sarcasm Annotation?: A Case Study of Indian Annotators and American Text](https://md.research.elamur.ai/free/papers/joshi-2016-how-do-cultural-differences-impact/)
- [A Large Self-Annotated Corpus for Sarcasm](https://md.research.elamur.ai/free/papers/khodak-2018-large-self-annotated-corpus-sarcasm/)
- [Argument Mining: A Survey](https://md.research.elamur.ai/free/papers/lawrence-2019-argument-mining-survey/)
- [Really? Well. Apparently Bootstrapping Improves the Performance of Sarcasm and Nastiness Classifiers for Online Dialogue](https://md.research.elamur.ai/free/papers/lukin-2013-really-well-apparently-bootstrapping-improves/)
- [Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue](https://md.research.elamur.ai/free/papers/oraby-2016-creating-characterizing-diverse-corpus-sarcasm/)
- [Parsing Argumentation Structures in Persuasive Essays](https://md.research.elamur.ai/free/papers/stab-2017-parsing-argumentation-structures-persuasive-essays/)
- [Getting Reliable Annotations for Sarcasm in Online Dialogues](https://md.research.elamur.ai/free/papers/swanson-2014-getting-reliable-annotations-sarcasm-online/)
- [OpenStance: Real-world Zero-shot Stance Detection](https://md.research.elamur.ai/free/papers/xu-2022-openstance-real-world-zero-shot/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{walker2012corpus,
  title = {{A Corpus for Research on Deliberation and Debate}},
  author = {Walker, Marilyn and Tree, Jean Fox and Anand, Pranav and Abbott, Rob and King, Joseph},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/2ommjodrhqvf},
  url = {https://doi.org/10.63317/2ommjodrhqvf}
}
```

- [← previous A Comparative Study of Root-Based and Stem-Based Approaches for Measuring the Similarity between Arabic Words](https://md.research.elamur.ai/free/papers/langstream-2012-froud-comparative-study-root-based-stem/)
- [next → A Discriminative Hierarchical Model for Fast Coreference at Large Scale](https://md.research.elamur.ai/free/papers/wick-2012-discriminative-hierarchical-model-fast-coreference/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/papers/2012/)

## References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Agrawal, S. Rajagopalan, R. Srikant, and Y. Xu. 2003. Mining newsgroups using networks arising from social behavior. In Proceedings of the 12th international conference on World Wide Web, pages 529–535. ACM. [find](https://scholar.google.com/scholar?q=R%2E+Agrawal%2C+S%2E+Rajagopalan%2C+R%2E+Srikant%2C+and+Y%2E+Xu%2E+2003%2E+Mining+newsgroups+using+networks+arising+from+social+behavior%2E+In+Proceedings+of+the+12th+international+conference+on+World+Wide+Web%2C+pages+529)
2. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats Rule and Dogs Drool: Classifying Stance in Online Debate. In Proceedings of the ACL Workshop on Sentiment and Subjectivity. [find](https://scholar.google.com/scholar?q=Pranav+Anand%2C+Marilyn+Walker%2C+Rob+Abbott%2C+Jean+E%2E+Fox+Tree%2C+Robeson+Bowmani%2C+and+Michael+Minor%2E+2011%2E+Cats+Rule+and+Dogs+Drool%3A+Classifying+Stance+in+Online+Debate%2E+In+Proceedings+of+the+ACL+Workshop+)
3. R. Awadallah, M. Ramanath, and G. Weikum. 2010. Language-model-based pro/con classification of political text. In Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 747–748. ACM. [find](https://scholar.google.com/scholar?q=R%2E+Awadallah%2C+M%2E+Ramanath%2C+and+G%2E+Weikum%2E+2010%2E+Language%2Dmodel%2Dbased+pro%2Fcon+classification+of+political+text%2E+In+Proceeding+of+the+33rd+international+ACM+SIGIR+conference+on+Research+and+development+)
4. M. Bansal, C. Cardie, and L. Lee. 2008. The power of negative thinking: Exploiting label disagreement in the min-cut classification framework. Proceedings of COLING: Companion volume: Posters, pages 13–16. [find](https://scholar.google.com/scholar?q=M%2E+Bansal%2C+C%2E+Cardie%2C+and+L%2E+Lee%2E+2008%2E+The+power+of+negative+thinking%3A+Exploiting+label+disagreement+in+the+min%2Dcut+classification+framework%2E+Proceedings+of+COLING%3A+Companion+volume%3A+Posters%2C+pages+1)
5. G.A. Bryant and J.E. Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. [find](https://scholar.google.com/scholar?q=G%2EA%2E+Bryant+and+J%2EE%2E+Fox+Tree%2E+2002%2E+Recognizing+verbal+irony+in+spontaneous+speech%2E+Metaphor+and+symbol%2C+17%282%29%3A99%E2%80%93119%2E)
6. D. Davidov, O. Tsur, and A. Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=D%2E+Davidov%2C+O%2E+Tsur%2C+and+A%2E+Rappoport%2E+2010%2E+Semi%2Dsupervised+recognition+of+sarcastic+sentences+in+twitter+and+amazon%2E+In+Proceedings+of+the+Fourteenth+Conference+on+Computational+Natural+Language+Lea) [our card](https://md.research.elamur.ai/free/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/)
7. E.N. Forsyth and C.H. Martell. 2007. Lexical and discourse analysis of online chat dialog. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=E%2EN%2E+Forsyth+and+C%2EH%2E+Martell%2E+2007%2E+Lexical+and+discourse+analysis+of+online+chat+dialog%2E+IEEE+Computer+Society%2E)
8. J.E. Fox Tree and J.C. Schrock. 1999. Discourse Markers in Spontaneous Speech: Oh What a Difference an Oh Makes. Journal of Memory and Language, 40(2):280– 295. [find](https://scholar.google.com/scholar?q=J%2EE%2E+Fox+Tree+and+J%2EC%2E+Schrock%2E+1999%2E+Discourse+Markers+in+Spontaneous+Speech%3A+Oh+What+a+Difference+an+Oh+Makes%2E+Journal+of+Memory+and+Language%2C+40%282%29%3A280%E2%80%93+295%2E)
9. J.E. Fox Tree and J.C. Schrock. 2002. Basic meanings of you know and I mean. Journal of Pragmatics, 34(6):727–747. [find](https://scholar.google.com/scholar?q=J%2EE%2E+Fox+Tree+and+J%2EC%2E+Schrock%2E+2002%2E+Basic+meanings+of+you+know+and+I+mean%2E+Journal+of+Pragmatics%2C+34%286%29%3A727%E2%80%93747%2E)
10. M. Galley, K. McKeown, J. Hirschberg, and E. Shriberg. 2004. Identifying agreement and disagreement in conversational speech: Use of bayesian networks to model pragmatic dependencies. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 669–es. Association fo… [find](https://scholar.google.com/scholar?q=M%2E+Galley%2C+K%2E+McKeown%2C+J%2E+Hirschberg%2C+and+E%2E+Shriberg%2E+2004%2E+Identifying+agreement+and+disagreement+in+conversational+speech%3A+Use+of+bayesian+networks+to+model+pragmatic+dependencies%2E+In+Proceedings+o) [our card](https://md.research.elamur.ai/free/papers/galley-2004-identifying-agreement-disagreement-conversational-speech/)
11. R.W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1):5–27. [find](https://scholar.google.com/scholar?q=R%2EW%2E+Gibbs%2E+2000%2E+Irony+in+talk+among+friends%2E+Metaphor+and+Symbol%2C+15%281%29%3A5%E2%80%9327%2E)
12. S. Greene and P. Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 503–511. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=S%2E+Greene+and+P%2E+Resnik%2E+2009%2E+More+than+words%3A+Syntactic+packaging+and+implicit+sentiment%2E+In+Proceedings+of+Human+Language+Technologies%3A+The+2009+Annual+Conference+of+the+North+American+Chapter+of+t)
13. M. Groen, J. Noyes, and F. Verstraten. 2010. The Effect of Substituting Discourse Markers on Their Role in Dialogue. Discourse Processes: A Multidisciplinary Journal, 47(5):33. [find](https://scholar.google.com/scholar?q=M%2E+Groen%2C+J%2E+Noyes%2C+and+F%2E+Verstraten%2E+2010%2E+The+Effect+of+Substituting+Discourse+Markers+on+Their+Role+in+Dialogue%2E+Discourse+Processes%3A+A+Multidisciplinary+Journal%2C+47%285%29%3A33%2E)
14. W.H. Lin, T. Wilson, J. Wiebe, and A. Hauptmann. 2006. Which side are you on?: identifying perspectives at the document and sentence levels. In Proceedings of the Tenth Conference on Computational Natural Language Learning, pages 109–116. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=W%2EH%2E+Lin%2C+T%2E+Wilson%2C+J%2E+Wiebe%2C+and+A%2E+Hauptmann%2E+2006%2E+Which+side+are+you+on%3F%3A+identifying+perspectives+at+the+document+and+sentence+levels%2E+In+Proceedings+of+the+Tenth+Conference+on+Computational+Nat)
15. A. Murakami and R. Raymond. 2010. Support or Oppose? [find](https://scholar.google.com/scholar?q=A%2E+Murakami+and+R%2E+Raymond%2E+2010%2E+Support+or+Oppose%3F)
16. Classifying Positions in Online Debates from Reply Activities and Opinion Expressions. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 869–875. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Classifying+Positions+in+Online+Debates+from+Reply+Activities+and+Opinion+Expressions%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Linguistics%3A+Posters%2C+pages+869%E2%80%93875%2E+Associa)
17. R. Snow, B. O’Connor, D. Jurafsky, and A.Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=R%2E+Snow%2C+B%2E+O%E2%80%99Connor%2C+D%2E+Jurafsky%2C+and+A%2EY%2E+Ng%2E+2008%2E+Cheap+and+fast%E2%80%94but+is+it+good%3F%3A+evaluating+non%2Dexpert+annotations+for+natural+language+tasks%2E+In+Proceedings+of+the+Conference+on+Empirical+Method)
18. S. Somasundaran and J. Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 226–234. Association for Computatio… [find](https://scholar.google.com/scholar?q=S%2E+Somasundaran+and+J%2E+Wiebe%2E+2009%2E+Recognizing+stances+in+online+debates%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+and+the+4th+International+Joint+Conference+on+Na)
19. S. Somasundaran and J. Wiebe. 2010. Recognizing stances in ideological on-line debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 116–124. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=S%2E+Somasundaran+and+J%2E+Wiebe%2E+2010%2E+Recognizing+stances+in+ideological+on%2Dline+debates%2E+In+Proceedings+of+the+NAACL+HLT+2010+Workshop+on+Computational+Approaches+to+Analysis+and+Generation+of+Emotion+)
20. M. Thomas, B. Pang, and L. Lee. 2006. Get out the vote: Determining support or opposition from Congressional floor-debate transcripts. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 327–335. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=M%2E+Thomas%2C+B%2E+Pang%2C+and+L%2E+Lee%2E+2006%2E+Get+out+the+vote%3A+Determining+support+or+opposition+from+Congressional+floor%2Ddebate+transcripts%2E+In+Proceedings+of+the+2006+conference+on+empirical+methods+in+nat)
21. Marilyn Walker, Pranav Anand, Rob Abbott, Jean E. Fox Tree, Craig Martell, and Joseph King. That’s your evidence?: Classifying stance in online political debate. [find](https://scholar.google.com/scholar?q=Marilyn+Walker%2C+Pranav+Anand%2C+Rob+Abbott%2C+Jean+E%2E+Fox+Tree%2C+Craig+Martell%2C+and+Joseph+King%2E+That%E2%80%99s+your+evidence%3F%3A+Classifying+stance+in+online+political+debate%2E)