---
title: "Evaluation of Discourse Relation Annotation — Kolachina 2012"
description: "The paper tests whether changes made to the Penn Discourse Treebank (PDTB) scheme while adapting it to Hindi actually improve annotation quality."
authors: ["Sudheer Kolachina", "Rashmi Prasad", "Dipti Misra Sharma", "Aravind Joshi"]
year: 2012
venue: "Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)"
source: "https://aclanthology.org/L12-1533/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/kolachina-2012-evaluation-discourse-relation-annotation-hindi/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kolachina-2012-evaluation-discourse-relation-annotation-hindi/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2012](https://md.research.elamur.ai/free/papers/2012/) / Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank

# Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank

**Authors:** [Sudheer Kolachina](https://md.research.elamur.ai/free/authors/sudheer-kolachina/), [Rashmi Prasad](https://md.research.elamur.ai/free/authors/rashmi-prasad/), [Dipti Misra Sharma](https://md.research.elamur.ai/free/authors/dipti-misra-sharma/), [Aravind Joshi](https://md.research.elamur.ai/free/authors/aravind-joshi/) · **Year:** 2012 · **Venue:** Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L12-1533/) · [PDF](https://aclanthology.org/L12-1533.pdf) · [OpenAlex](https://openalex.org/W2250211448)

**Citations:** 15 citations (openalex-snapshot · 2026-10-14) · [8](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper tests whether changes made to the Penn Discourse Treebank (PDTB) scheme while adapting it to Hindi actually improve annotation quality. These changes came from earlier work on the Hindi Discourse Relation Bank (HDRB) and were meant to make sense classification more uniform and conceptually cleaner. One example is naming arguments by their semantic role rather than by linear order. The authors applied the HDRB guidelines to a subset of a 400K-word Hindi news corpus and ran an inter-annotator agreement study. It covered explicit connectives, implicit connectives, AltLex and EntRel. Annotators worked over continuous raw text with the PDTB annotation tool.

Agreement on relation senses was high, which supports the hierarchical sense scheme. Agreement was weak in other areas:
- annotators disagreed on which tokens count as connectives at all;
- argument spans matched poorly;
- annotators seemed to prefer linear-order argument labels over semantic ones.

The authors conclude that some modifications made the task harder. They propose splitting the workflow into inter- and intra-sentential passes, returning to syntactic argument naming, and allowing implicit relations between non-adjacent sentences. The study is a practical lesson in annotation task design for discourse resources.

## Key points
- **Corpus:** HDRB is built on about 400K words of multi-domain news from the Hindi daily AMAR UJALA. The same corpus also carries dependency, predicate-argument and phrase-structure layers.
- **What was annotated:** The scheme marks explicit connectives, implicit connectives, AltLex and EntRel/NoRel, with Arg1/Arg2 spans following the minimality principle. Unlike PDTB, it also annotates implicit relations across paragraph boundaries.
- **Connective counts and spans:** Annotators produced different numbers of connective tokens, and the line between connectives and AltLex was especially hard to draw. Argument-span agreement was low for both explicit and implicit relations.
- **Senses:** Sense agreement was high for both explicit and implicit connectives. However, annotators were allowed to back off to coarser type-level labels.
- **Proposed revisions:** a two-step workflow (inter-sentential relations first, then intra-sentential), a return to PDTB-style linear-order argument naming, and annotation of implicit connectives between non-adjacent sentences.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

- [Reflections on the Penn Discourse TreeBank, Comparable Corpora, and Complementary Annotation](https://md.research.elamur.ai/free/papers/prasad-2014-reflections-penn-discourse-treebank-comparable/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kolachina2012evaluation,
  title = {{Evaluation of Discourse Relation Annotation in the Hindi Discourse Relation Bank}},
  author = {Kolachina, Sudheer and Prasad, Rashmi and Sharma, Dipti Misra and Joshi, Aravind},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/4vqsnirkgvgm},
  url = {https://doi.org/10.63317/4vqsnirkgvgm}
}
```

- [← previous Easy-first Coreference Resolution](https://md.research.elamur.ai/free/papers/stoyanov-2012-easy-first-coreference-resolution/)
- [next → Event Nominals: Annotation Guidelines and a Manually Annotated Corpus in French](https://md.research.elamur.ai/free/papers/arnulphy-2012-event-nominals-annotation-guidelines-manually/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/papers/2012/)

## References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Al-Saif and K. Markert. 2010. The Leeds Arabic Discourse Treebank: Annotating Discourse Connectives for Arabic. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC-2010), Valletta, Malta. [find](https://scholar.google.com/scholar?q=A%2E+Al%2DSaif+and+K%2E+Markert%2E+2010%2E+The+Leeds+Arabic+Discourse+Treebank%3A+Annotating+Discourse+Connectives+for+Arabic%2E+In+Proceedings+of+the+7th+International+Conference+on+Language+Resources+and+Evaluati) [our card](https://md.research.elamur.ai/free/papers/saif-2010-leeds-arabic-discourse-treebank-annotating/)
2. R. Begum, S. Husain, A. Dhwaj, D.M. Sharma, L. Bai, and R. Sangal. 2008. Dependency Annotation Scheme for Indian Languages. Proceedings of IJCNLP-2008. [find](https://scholar.google.com/scholar?q=R%2E+Begum%2C+S%2E+Husain%2C+A%2E+Dhwaj%2C+D%2EM%2E+Sharma%2C+L%2E+Bai%2C+and+R%2E+Sangal%2E+2008%2E+Dependency+Annotation+Scheme+for+Indian+Languages%2E+Proceedings+of+IJCNLP%2D2008%2E)
3. R. Bhatt, B. Narasimhan, M. Palmer, O. Rambow, D.M. Sharma, and F. Xia. 2009. A Multi-Representational and Multi-Layered Treebank for Hindi/Urdu. In Proceedings of the Third Linguistic Annotation Workshop, pages 186–189. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=R%2E+Bhatt%2C+B%2E+Narasimhan%2C+M%2E+Palmer%2C+O%2E+Rambow%2C+D%2EM%2E+Sharma%2C+and+F%2E+Xia%2E+2009%2E+A+Multi%2DRepresentational+and+Multi%2DLayered+Treebank+for+Hindi%2FUrdu%2E+In+Proceedings+of+the+Third+Linguistic+Annotation+Work)
4. E. Miltsakaki, R. Prasad, A. Joshi, and B. Webber. 2004. Annotating Discourse Connectives And Their Arguments. In Proceedings of the HLT/NAACL Workshop on Frontiers in Corpus Annotation, pages 9–16. [find](https://scholar.google.com/scholar?q=E%2E+Miltsakaki%2C+R%2E+Prasad%2C+A%2E+Joshi%2C+and+B%2E+Webber%2E+2004%2E+Annotating+Discourse+Connectives+And+Their+Arguments%2E+In+Proceedings+of+the+HLT%2FNAACL+Workshop+on+Frontiers+in+Corpus+Annotation%2C+pages+9%E2%80%9316%2E) [our card](https://md.research.elamur.ai/free/papers/miltsakaki-2004-annotating-discourse-connectives-their-arguments/)
5. L. Mladová, Š. Zikánová, and E. Hajičová. 2008. From Sentence to Discourse: Building an Annotation Scheme for Discourse Based on Prague Dependency Treebank. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008). [find](https://scholar.google.com/scholar?q=L%2E+Mladova%CC%81%2C+S%CC%8C%2E+Zika%CC%81nova%CC%81%2C+and+E%2E+Hajic%CC%8Cova%CC%81%2E+2008%2E+From+Sentence+to+Discourse%3A+Building+an+Annotation+Scheme+for+Discourse+Based+on+Prague+Dependency+Treebank%2E+In+Proceedings+of+the+6th+Internation)
6. U. Oza, R. Prasad, S. Kolachina, S. Meena, D.M. Sharma, and A. Joshi. 2009a. Experiments with Annotating Discourse Relations in the Hindi Discourse Relation Bank. In Proceedings of the 7th International Conference on Natural Language Processing (ICON-2009), Hyderabad, India. [find](https://scholar.google.com/scholar?q=U%2E+Oza%2C+R%2E+Prasad%2C+S%2E+Kolachina%2C+S%2E+Meena%2C+D%2EM%2E+Sharma%2C+and+A%2E+Joshi%2E+2009a%2E+Experiments+with+Annotating+Discourse+Relations+in+the+Hindi+Discourse+Relation+Bank%2E+In+Proceedings+of+the+7th+Internation) [our card](https://md.research.elamur.ai/free/papers/oza-2009-hindi-discourse-relation-bank/)
7. U. Oza, R. Prasad, S. Kolachina, D.M. Sharma, and A. Joshi. 2009b. The Hindi Discourse Relation Bank. In Proceedings of the Third Linguistic Annotation Workshop, pages 158–161. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=U%2E+Oza%2C+R%2E+Prasad%2C+S%2E+Kolachina%2C+D%2EM%2E+Sharma%2C+and+A%2E+Joshi%2E+2009b%2E+The+Hindi+Discourse+Relation+Bank%2E+In+Proceedings+of+the+Third+Linguistic+Annotation+Workshop%2C+pages+158%E2%80%93161%2E+Association+for+Computa) [our card](https://md.research.elamur.ai/free/papers/oza-2009-hindi-discourse-relation-bank/)
8. M. Palmer, R. Bhatt, B. Narasimhan, O. Rambow, D.M. Sharma, and F. Xia. 2009. Hindi Syntax: Annotating Dependency, Lexical Predicate-Argument Structure, and Phrase Structure. In The 7th International Conference on Natural Language Processing, pages 14–17. [find](https://scholar.google.com/scholar?q=M%2E+Palmer%2C+R%2E+Bhatt%2C+B%2E+Narasimhan%2C+O%2E+Rambow%2C+D%2EM%2E+Sharma%2C+and+F%2E+Xia%2E+2009%2E+Hindi+Syntax%3A+Annotating+Dependency%2C+Lexical+Predicate%2DArgument+Structure%2C+and+Phrase+Structure%2E+In+The+7th+International+)
9. R. Prasad, N. Dinesh, A. Lee, A. Joshi, and B. Webber. 2007. Attribution and its annotation in the Penn Discourse TreeBank. Traitement Automatique des Langues, Special Issue on Computational Approaches to Document and Discourse, 47(2). [find](https://scholar.google.com/scholar?q=R%2E+Prasad%2C+N%2E+Dinesh%2C+A%2E+Lee%2C+A%2E+Joshi%2C+and+B%2E+Webber%2E+2007%2E+Attribution+and+its+annotation+in+the+Penn+Discourse+TreeBank%2E+Traitement+Automatique+des+Langues%2C+Special+Issue+on+Computational+Approache)
10. R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), pages 2961– 2968. Citeseer. [find](https://scholar.google.com/scholar?q=R%2E+Prasad%2C+N%2E+Dinesh%2C+A%2E+Lee%2C+E%2E+Miltsakaki%2C+L%2E+Robaldo%2C+A%2E+Joshi%2C+and+B%2E+Webber%2E+2008%2E+The+Penn+Discourse+Treebank+2%2E0%2E+In+Proceedings+of+the+6th+International+Conference+on+Language+Resources+and+Ev) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
11. R. T. Rachakonda and D. M. Sharma. 2011. Creating an Annotated Tamil Corpus as a Discourse Resource. In Proceedings of the 5th Linguistic Annotation Workshop, pages 119–123, Portland, OR. [find](https://scholar.google.com/scholar?q=R%2E+T%2E+Rachakonda+and+D%2E+M%2E+Sharma%2E+2011%2E+Creating+an+Annotated+Tamil+Corpus+as+a+Discourse+Resource%2E+In+Proceedings+of+the+5th+Linguistic+Annotation+Workshop%2C+pages+119%E2%80%93123%2C+Portland%2C+OR%2E)
12. S. Tonelli, G. Riccardi, R. Prasad, and A. Joshi. 2010. Annotation of Discourse Relations for Conversational Spoken Dialogs. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta. [find](https://scholar.google.com/scholar?q=S%2E+Tonelli%2C+G%2E+Riccardi%2C+R%2E+Prasad%2C+and+A%2E+Joshi%2E+2010%2E+Annotation+of+Discourse+Relations+for+Conversational+Spoken+Dialogs%2E+In+Proceedings+of+the+Seventh+International+Conference+on+Language+Resource) [our card](https://md.research.elamur.ai/free/papers/tonelli-2010-annotation-discourse-relations-conversational-spoken/)
13. F. Xia, O. Rambow, R. Bhatt, M. Palmer, and D.M. Sharma. 2008. Towards a Multi-Representational Treebank. Proceedings of the 7th International Workshop on Treebanks and Linguistic Theories (TLT-7), pages 207–238. [find](https://scholar.google.com/scholar?q=F%2E+Xia%2C+O%2E+Rambow%2C+R%2E+Bhatt%2C+M%2E+Palmer%2C+and+D%2EM%2E+Sharma%2E+2008%2E+Towards+a+Multi%2DRepresentational+Treebank%2E+Proceedings+of+the+7th+International+Workshop+on+Treebanks+and+Linguistic+Theories+%28TLT%2D7%29%2C+pa)
14. D. Zeyrek and B. Webber. 2008. A Discourse Resource for Turkish: Annotating Discourse Connectives in the METU Corpus. Proceedings of IJCNLP-2008. Hyderabad, India. [find](https://scholar.google.com/scholar?q=D%2E+Zeyrek+and+B%2E+Webber%2E+2008%2E+A+Discourse+Resource+for+Turkish%3A+Annotating+Discourse+Connectives+in+the+METU+Corpus%2E+Proceedings+of+IJCNLP%2D2008%2E+Hyderabad%2C+India%2E) [our card](https://md.research.elamur.ai/free/papers/zeyrek-2008-discourse-resource-turkish-annotating-discourse/)
15. Y. Zhou and N. Xue. 2012. PDTB-style Discourse Annotation of Chinese Text. In Proc. 50 th Annual Meeting of the ACL, Jeju Island, Korea. [find](https://scholar.google.com/scholar?q=Y%2E+Zhou+and+N%2E+Xue%2E+2012%2E+PDTB%2Dstyle+Discourse+Annotation+of+Chinese+Text%2E+In+Proc%2E+50+th+Annual+Meeting+of+the+ACL%2C+Jeju+Island%2C+Korea%2E)