---
title: "Comparative Evaluation of Argument — Stepanov 2013"
description: "The paper addresses argument span extraction for explicit discourse relations in the Penn Discourse Treebank (PDTB)."
authors: ["Evgeny Stepanov", "Giuseppe Riccardi"]
year: 2013
venue: "Proceedings of the 13th International Conference on Parsing Technologies (IWPT 2013)"
source: "https://aclanthology.org/W13-5704/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/stepanov-2013-comparative-evaluation-argument-extraction-algorithms/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/stepanov-2013-comparative-evaluation-argument-extraction-algorithms/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing

# Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing

**Authors:** [Evgeny Stepanov](https://md.research.elamur.ai/free/authors/evgeny-stepanov/), [Giuseppe Riccardi](https://md.research.elamur.ai/free/authors/giuseppe-riccardi/) · **Year:** 2013 · **Venue:** Proceedings of the 13th International Conference on Parsing Technologies (IWPT 2013) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-5704/) · [PDF](https://aclanthology.org/W13-5704.pdf) · [OpenAlex](https://openalex.org/W2955477482)

**Citations:** 11 citations (openalex-snapshot · 2026-10-14) · [6](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses argument span extraction for explicit discourse relations in the Penn Discourse Treebank (PDTB). In this subtask, a parser must mark which tokens belong to Arg1 and Arg2 of a connective. Arg1 may sit in the connective's sentence (intra-sentential, SS) or in an earlier one (inter-sentential, PS). This raises a design question: should one model handle both cases, or should each case get its own model?

The authors start from the method of Ghosh et al. (2011), which labels each token with a CRF and uses a single model over a ±2 sentence window. They compare it with a pipeline that first predicts argument position (SS vs. PS) and then applies dedicated CRF models. For PS relations, this pipeline adds an immediately-previous-sentence heuristic to select the Arg1 candidate.

When position-classifier errors are counted against the separate-model parser, it still significantly beats the single-model parser for both arguments. The gain is largest for inter-sentential Arg1. The results show that splitting the task reduces its complexity. Token-level labeling also performs roughly on par with tree-node and constituent-level parsers.

## Key points
- PDTB statistics: of 18,459 explicit relations, 60.9% are SS, 39.1% are PS, and only 8 have Arg1 in a following sentence. In 44.2% of inter-sentential cases, Arg1 is exactly the whole immediately previous sentence.
- Predicting SS vs. PS from syntactic category and connective position alone gives 95.0% majority accuracy. The trained argument position classifier reaches 98.12%.
- On all relations, F1 scores with error propagation are:
  - single ±2 window parser (W5P): Arg2 77.79, Arg1 53.40
  - separate CRF models (SMP): Arg2 84.67, Arg1 58.25
  - separate models with heuristics (hSMP): Arg2 83.06, Arg1 56.71
- On PS relations, Arg1 F1 improves from 29.01 (W5P) to 40.86 (SMP). Compared with the heuristics, the CRFs help significantly for Arg2 but not significantly for Arg1.
- With training on sections 02-21 and testing on section 23, SMP scores Arg2/Arg1 F1 of 82.35/57.26. Lin et al. (2012) report 82.23/59.15 and Xu et al. (2012) report 81.00/60.69, though the authors consider this comparison not definitive.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 2 not yet.

- [Towards Cross-Domain PDTB-Style Discourse Parsing](https://md.research.elamur.ai/free/papers/stepanov-2014-cross-domain-pdtb-style-discourse/)
- [The UniTN Discourse Parser in CoNLL 2015 Shared Task: Token-level Sequence Labeling with Argument-specific Models](https://md.research.elamur.ai/free/papers/stepanov-2015-unitn-discourse-parser-conll-2015/)
- [UniTN End-to-End Discourse Parser for CoNLL 2016 Shared Task](https://md.research.elamur.ai/free/papers/stepanov-2016-unitn-end-end-discourse-parser/)
- [The DCU Discourse Parser for Connective, Argument Identification and Explicit Sense Classification](https://md.research.elamur.ai/free/papers/wang-2015-dcu-discourse-parser-connective-argument/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{stepanov2013comparative,
  title = {{Comparative Evaluation of Argument Extraction Algorithms in Discourse Relation Parsing}},
  author = {Stepanov, Evgeny and Riccardi, Giuseppe},
  booktitle = {Proceedings of the 13th International Conference on Parsing Technologies (IWPT 2013)},
  year = {2013},
  url = {https://aclanthology.org/W13-5704/}
}
```

- [← previous Chinese Zero Pronoun Resolution: Some Recent Advances](https://md.research.elamur.ai/free/papers/chen-2013-chinese-zero-pronoun-resolution-some/)
- [next → Construction of English MWE Dictionary and its Application to POS Tagging](https://md.research.elamur.ai/free/papers/shigeto-2013-construction-english-mwe-dictionary-its/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/)

## References

20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sabine Buchholz. 2000. Readme for perl script chunklink.pl. [find](https://scholar.google.com/scholar?q=Sabine+Buchholz%2E+2000%2E+Readme+for+perl+script+chunklink%2Epl%2E)
2. Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2005. Attribution and the (non)-alignment of syntactic and discourse arguments of connectives. In Proceedings of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky. [find](https://scholar.google.com/scholar?q=Nikhil+Dinesh%2C+Alan+Lee%2C+Eleni+Miltsakaki%2C+Rashmi+Prasad%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2005%2E+Attribution+and+the+%28non%29%2Dalignment+of+syntactic+and+discourse+arguments+of+connectives%2E+In+Proceeding)
3. Robert Elwell and Jason Baldridge. 2008. Discourse connective argument identification with connective specific rankers. In Proceedings of the IEEE International Conference on Semantic Computing (ICSC 2008). [find](https://scholar.google.com/scholar?q=Robert+Elwell+and+Jason+Baldridge%2E+2008%2E+Discourse+connective+argument+identification+with+connective+specific+rankers%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Semantic+Computing+%28ICSC+)
4. Sucheta Ghosh, Richard Johansson, Giuseppe Riccardi, and Sara Tonelli. 2011. Shallow discourse parsing with conditional random fields. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP 2011). [find](https://scholar.google.com/scholar?q=Sucheta+Ghosh%2C+Richard+Johansson%2C+Giuseppe+Riccardi%2C+and+Sara+Tonelli%2E+2011%2E+Shallow+discourse+parsing+with+conditional+random+fields%2E+In+Proceedings+of+the+5th+International+Joint+Conference+on+Natur) [our card](https://md.research.elamur.ai/free/papers/ghosh-2011-shallow-discourse-parsing-conditional-random/)
5. Sucheta Ghosh, Richard Johansson, Giuseppe Riccardi, and Sara Tonelli. 2012a. Improving the recall of a discourse parser by constraint-based postprocessing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012). [find](https://scholar.google.com/scholar?q=Sucheta+Ghosh%2C+Richard+Johansson%2C+Giuseppe+Riccardi%2C+and+Sara+Tonelli%2E+2012a%2E+Improving+the+recall+of+a+discourse+parser+by+constraint%2Dbased+postprocessing%2E+In+Proceedings+of+the+Eighth+International+)
6. Sucheta Ghosh, Giuseppe Riccardi, and Richard Johansson. 2012b. Global features for shallow discourse parsing. In Proceedings of the SIGDIAL 2012 Conference, The 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue. [find](https://scholar.google.com/scholar?q=Sucheta+Ghosh%2C+Giuseppe+Riccardi%2C+and+Richard+Johansson%2E+2012b%2E+Global+features+for+shallow+discourse+parsing%2E+In+Proceedings+of+the+SIGDIAL+2012+Conference%2C+The+13th+Annual+Meeting+of+the+Special+Int) [our card](https://md.research.elamur.ai/free/papers/ghosh-2012-global-features-shallow-discourse-parsing/)
7. Dan Klein and Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15 (NIPS 2002). [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Christopher+D%2E+Manning%2E+2003%2E+Fast+exact+inference+with+a+factored+model+for+natural+language+parsing%2E+In+Advances+in+Neural+Information+Processing+Systems+15+%28NIPS+2002%29%2E)
8. Alistair Knott. 1996. A Data-Driven Methodology for Motivating a Set of Coherence Relations. Ph.D. thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Alistair+Knott%2E+1996%2E+A+Data%2DDriven+Methodology+for+Motivating+a+Set+of+Coherence+Relations%2E+Ph%2ED%2E+thesis%2C+University+of+Edinburgh%2E)
9. John Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of 18th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+C%2E+N%2E+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proceedings+of+18th+International+Confer)
10. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2012. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 1:1 – 35. [find](https://scholar.google.com/scholar?q=Ziheng+Lin%2C+Hwee+Tou+Ng%2C+and+Min%2DYen+Kan%2E+2012%2E+A+pdtb%2Dstyled+end%2Dto%2Dend+discourse+parser%2E+Natural+Language+Engineering%2C+1%3A1+%E2%80%93+35%2E)
11. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of english. Natural Language Engineering. [find](https://scholar.google.com/scholar?q=Guido+Minnen%2C+John+Carroll%2C+and+Darren+Pearce%2E+2001%2E+Applied+morphological+processing+of+english%2E+Natural+Language+Engineering%2E)
12. Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP Conference. [find](https://scholar.google.com/scholar?q=Emily+Pitler+and+Ani+Nenkova%2E+2009%2E+Using+syntax+to+disambiguate+explicit+discourse+connectives+in+text%2E+In+Proceedings+of+the+ACL%2DIJCNLP+Conference%2E) [our card](https://md.research.elamur.ai/free/papers/pitler-2009-syntax-disambiguate-explicit-discourse-connectives/)
13. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The penn discourse treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). [find](https://scholar.google.com/scholar?q=Rashmi+Prasad%2C+Nikhil+Dinesh%2C+Alan+Lee%2C+Eleni+Miltsakaki%2C+Livio+Robaldo%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2008%2E+The+penn+discourse+treebank+2%2E0%2E+In+Proceedings+of+the+6th+International+Conference+on+) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
14. Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2010. Exploiting scope for shallow discourse parsing. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC-2010). [find](https://scholar.google.com/scholar?q=Rashmi+Prasad%2C+Aravind+Joshi%2C+and+Bonnie+Webber%2E+2010%2E+Exploiting+scope+for+shallow+discourse+parsing%2E+In+Proceedings+of+the+Seventh+International+Conference+on+Language+Resources+and+Evaluation+%28LREC) [our card](https://md.research.elamur.ai/free/papers/prasad-2010-exploiting-scope-shallow-discourse-parsing/)
15. Robert E. Schapire and Yoram Singer. 2000. Boostexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135–168. [find](https://scholar.google.com/scholar?q=Robert+E%2E+Schapire+and+Yoram+Singer%2E+2000%2E+Boostexter%3A+A+boosting%2Dbased+system+for+text+categorization%2E+Machine+Learning%2C+39%282%2F3%29%3A135%E2%80%93168%2E)
16. Maite Taboada and William C. Mann. 2006. Applications of rhetorical structure theory. Discourse Studies, (8):567–88. [find](https://scholar.google.com/scholar?q=Maite+Taboada+and+William+C%2E+Mann%2E+2006%2E+Applications+of+rhetorical+structure+theory%2E+Discourse+Studies%2C+%288%29%3A567%E2%80%9388%2E)
17. Bonnie L. Webber, Markus Egg, and Valia Kordoni. 2011. Discourse structure and language technology. Natural Language Engineering, pages 1 – 54. [find](https://scholar.google.com/scholar?q=Bonnie+L%2E+Webber%2C+Markus+Egg%2C+and+Valia+Kordoni%2E+2011%2E+Discourse+structure+and+language+technology%2E+Natural+Language+Engineering%2C+pages+1+%E2%80%93+54%2E)
18. Ben Wellner and James Pustejovsky. 2007. Automatically identifying the arguments of discourse connectives. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007). [find](https://scholar.google.com/scholar?q=Ben+Wellner+and+James+Pustejovsky%2E+2007%2E+Automatically+identifying+the+arguments+of+discourse+connectives%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
19. Fan Xu, Qiao Ming Zhu, and Guo Dong Zhou. 2012. A unified framework for discourse argument identification via shallow semantic parsing. In Proceedings of 24th International Conference on Computational Linguistics (COLING 2012): Posters. [find](https://scholar.google.com/scholar?q=Fan+Xu%2C+Qiao+Ming+Zhu%2C+and+Guo+Dong+Zhou%2E+2012%2E+A+unified+framework+for+discourse+argument+identification+via+shallow+semantic+parsing%2E+In+Proceedings+of+24th+International+Conference+on+Computational)
20. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of 8th International Workshop on Parsing Technologies. [find](https://scholar.google.com/scholar?q=Hiroyasu+Yamada+and+Yuji+Matsumoto%2E+2003%2E+Statistical+dependency+analysis+with+support+vector+machines%2E+In+Proceedings+of+8th+International+Workshop+on+Parsing+Technologies%2E) [our card](https://md.research.elamur.ai/free/papers/parsers-2003-yamada-statistical-dependency-analysis-support-vector/)