---
title: "Annotation of Information Structure — Ritz 2008"
description: "This paper tests how reliably human annotators can apply the LISA guidelines (Linguistic Information Structure Annotation Guidelines) developed within…"
authors: ["Julia Ritz", "Stefanie Dipper", "Michael Götze"]
year: 2008
venue: "Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)"
source: "https://aclanthology.org/L08-1354/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/ritz-2008-annotation-information-structure-evaluation-across/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/ritz-2008-annotation-information-structure-evaluation-across/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2008](https://md.research.elamur.ai/free/papers/2008/) / Annotation of Information Structure: an Evaluation across different Types of Texts

# Annotation of Information Structure: an Evaluation across different Types of Texts

**Authors:** Julia Ritz, [Stefanie Dipper](https://md.research.elamur.ai/free/authors/stefanie-dipper/), Michael Götze · **Year:** 2008 · **Venue:** Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L08-1354/) · [PDF](https://aclanthology.org/L08-1354.pdf) · [OpenAlex](https://openalex.org/W1571123481)

**Citations:** 69 citations (openalex-snapshot · 2026-10-14) · [17](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper tests how reliably human annotators can apply the LISA guidelines (Linguistic Information Structure Annotation Guidelines) developed within the Potsdam SFB 632 project. The scheme is meant to work across languages and theories. It labels three independent dimensions: information status (given, accessible, new), topic (aboutness vs. frame-setting) and focus (new-information vs. contrastive). It offers core and finer-grained tagsets and uses decision trees and functional tests.

To check reliability, two trained undergraduate annotators labelled three kinds of German material in EXMARaLDA:
- question/answer pairs and map task dialogues elicited with QUIS,
- opinion pieces published in newspapers, drawn from the Potsdam Commentary Corpus (PCC).

Agreement was measured with Cohen's κ, computed over NPs and over tokens. Information status proved the most dependable layer. Topic and focus agreement was much lower and depended strongly on text type: the controlled question/answer data scored best and the commentaries worst.

The authors conclude that information-status annotation is feasible. Topic and focus need more training, text-type-specific instructions and sharper rules for focus boundaries. These findings matter for building IS corpora used in discourse analysis and NLP tasks such as anaphora resolution.

## Key points
- The evaluation corpus totals 1,940 tokens: 42 QuAn pairs (573 tokens), 2 map task dialogues (478 tokens) and 5 PCC commentaries (889 tokens).
- Information status reached κ of about .60–.80 with the core tagset. The extended tagset was slightly but consistently lower (NP-based κ .73 / .61 / .55).
- Topic agreement was high only for question/answer data (NP-based κ ≈ .91, partial evaluation). It fell to roughly .44–.51 for dialogues and commentaries.
- Focus was the weakest layer, dropping to κ = .19 (tokens) on PCC. The main difficulty was deciding where a focus span begins.
- Compared with related work (Nissim et al. 2004, κ = .845; Hempelmann et al. 2005; Paggio 2006 on DanPASS), the authors attribute the gap partly to the short three-day training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/topics/part-of-speech-tagging/)

## Cited by

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 15 not yet.

- [QUD-Based Annotation of Discourse Structure and Information Structure: Tool and Evaluation](https://md.research.elamur.ai/free/papers/kuthy-2018-qud-based-annotation-discourse-structure/)
- [Collective Classification for Fine-grained Information Status](https://md.research.elamur.ai/free/papers/markert-2012-collective-classification-fine-grained-information/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ritz2008annotation,
  title = {{Annotation of Information Structure: an Evaluation across different Types of Texts}},
  author = {Ritz, Julia and Dipper, Stefanie and Götze, Michael},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/3ni4c3iexg5j},
  url = {https://doi.org/10.63317/3ni4c3iexg5j}
}
```

- [← previous Anaphoric Annotation in the ARRAU Corpus](https://md.research.elamur.ai/free/papers/poesio-2008-anaphoric-annotation-arrau-corpus/)
- [next → Arabic Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking](https://md.research.elamur.ai/free/papers/aclbulk-2008-roth-arabic-morphological-tagging-diacritization-lemmatization/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/papers/2008/)

## References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anne H. Anderson, Miles Bader, Ellen G. Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Carrod, Stephen Isard, Jacqlene Kowtko, Jan McAllister, Jim Miller, Catherine Sotillo, Henry Thompson, and Regina Weinert. 1991. The HCRC Map Task Corpus. Language and Speech, 34:351–366. [find](https://scholar.google.com/scholar?q=Anne+H%2E+Anderson%2C+Miles+Bader%2C+Ellen+G%2E+Bard%2C+Elizabeth+Boyle%2C+Gwyneth+Doherty%2C+Simon+Carrod%2C+Stephen+Isard%2C+Jacqlene+Kowtko%2C+Jan+McAllister%2C+Jim+Miller%2C+Catherine+Sotillo%2C+Henry+Thompson%2C+and+Regina+)
2. Ron Artstein and Massimo Poesio. 2005. Kappa  = Alpha (or Beta). CSM 437, Department of Computer Science, University of Essex, UK. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2005%2E+Kappa+%EF%9C%B3+%3D+Alpha+%28or+Beta%29%2E+CSM+437%2C+Department+of+Computer+Science%2C+University+of+Essex%2C+UK%2E)
3. Jean Carletta. 1996. Assessing Agreement on Classification Tasks: The Kappa Statistic. Computational Linguistics, 22(2):249–254. [find](https://scholar.google.com/scholar?q=Jean+Carletta%2E+1996%2E+Assessing+Agreement+on+Classification+Tasks%3A+The+Kappa+Statistic%2E+Computational+Linguistics%2C+22%282%29%3A249%E2%80%93254%2E) [our card](https://md.research.elamur.ai/free/papers/carletta-1996-assessing-agreement-classification-tasks-kappa/)
4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%3A37%E2%80%9346%2E)
5. Stefanie Dipper, Michael Götze, and Stavros Skopeteas, editors. 2007. Information Structure in Cross-Linguistic Corpora: Annotation Guidelines for Phonology, Morphology, Syntax, Semantics, and Information Structure, volume 7 of Interdisciplinary Studies on Information Structure (ISIS). Universität… [find](https://scholar.google.com/scholar?q=Stefanie+Dipper%2C+Michael+Go%CC%88tze%2C+and+Stavros+Skopeteas%2C+editors%2E+2007%2E+Information+Structure+in+Cross%2DLinguistic+Corpora%3A+Annotation+Guidelines+for+Phonology%2C+Morphology%2C+Syntax%2C+Semantics%2C+and+Inform)
6. Jeanette K. Gundel, Nancy Hedberg, and Ron Zacharski. 1993. Cognitive status and the form of referring expressions in discourse. Language, 69(2):279–307. [find](https://scholar.google.com/scholar?q=Jeanette+K%2E+Gundel%2C+Nancy+Hedberg%2C+and+Ron+Zacharski%2E+1993%2E+Cognitive+status+and+the+form+of+referring+expressions+in+discourse%2E+Language%2C+69%282%29%3A279%E2%80%93307%2E)
7. Eva Hajic̆ová, Jarmila Panevová, Petr Sgall, Alena Böhmová, Markéta Ceplová, and Veronika R̆eznı́c̆ková. 2000. A Manual for Tectogrammatical Tagging of the Prague Dependency Treebank. TR 2000-09, ÚFAL/CKL, Prague, Czech Republic. [find](https://scholar.google.com/scholar?q=Eva+Hajic%CC%86ova%CC%81%2C+Jarmila+Panevova%CC%81%2C+Petr+Sgall%2C+Alena+Bo%CC%88hmova%CC%81%2C+Marke%CC%81ta+Ceplova%CC%81%2C+and+Veronika+R%CC%86ezn%C4%B1%CC%81c%CC%86kova%CC%81%2E+2000%2E+A+Manual+for+Tectogrammatical+Tagging+of+the+Prague+Dependency+Treebank%2E+TR+2000%2D0)
8. Nancy Hedberg, Jeanette K. Gundel, and Ron Zacharski. 2007. Directly and Indirectly Anaphoric Demonstrative and Personal Pronouns in Newspaper Articles. In Proceedings of DAARC 2007 (the Sixth Discourse Anaphora and Anaphora Resolution Colloquium), pages 31–36, Lagos, Portugal. [find](https://scholar.google.com/scholar?q=Nancy+Hedberg%2C+Jeanette+K%2E+Gundel%2C+and+Ron+Zacharski%2E+2007%2E+Directly+and+Indirectly+Anaphoric+Demonstrative+and+Personal+Pronouns+in+Newspaper+Articles%2E+In+Proceedings+of+DAARC+2007+%28the+Sixth+Discour)
9. Christian F. Hempelmann, David Dufty, Philip M. McCarthy, Arthur C. Graesser, Zhiqiang Cai, and Danielle S. McNamara. 2005. Using LSA to automatically identify givenness and newness of noun-phrases in written discourse. In B. Bara, editor, Proceedings of the 27th Annual Meetings of the Cognitive Sci… [find](https://scholar.google.com/scholar?q=Christian+F%2E+Hempelmann%2C+David+Dufty%2C+Philip+M%2E+McCarthy%2C+Arthur+C%2E+Graesser%2C+Zhiqiang+Cai%2C+and+Danielle+S%2E+McNamara%2E+2005%2E+Using+LSA+to+automatically+identify+givenness+and+newness+of+noun%2Dphrases+in)
10. Nobo Komagata, 2001. An Evaluation Method for Identifying Information Structure. Department of Computer and Information Science, University of Pennsylvania. Manuscript. [find](https://scholar.google.com/scholar?q=Nobo+Komagata%2C+2001%2E+An+Evaluation+Method+for+Identifying+Information+Structure%2E+Department+of+Computer+and+Information+Science%2C+University+of+Pennsylvania%2E+Manuscript%2E)
11. Costanza Navarretta. 2004. An algorithm for resolving individual and abstract anaphora in danish texts and dialogues. In Proceedings of the Workshop on Reference Resolution and its Applications: ACL 2004, pages 95– 102, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Costanza+Navarretta%2E+2004%2E+An+algorithm+for+resolving+individual+and+abstract+anaphora+in+danish+texts+and+dialogues%2E+In+Proceedings+of+the+Workshop+on+Reference+Resolution+and+its+Applications%3A+ACL+2) [our card](https://md.research.elamur.ai/free/papers/navarretta-2004-algorithm-resolving-individual-abstract-anaphora/)
12. Ani Nenkova, Advaith Siddharthan, and Kathleen McKeown. 2005. Automatically learning cognitive status for multi-document summarization of newswire. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 241–248, Vancouver, Bri… [find](https://scholar.google.com/scholar?q=Ani+Nenkova%2C+Advaith+Siddharthan%2C+and+Kathleen+McKeown%2E+2005%2E+Automatically+learning+cognitive+status+for+multi%2Ddocument+summarization+of+newswire%2E+In+Proceedings+of+Human+Language+Technology+Conferen)
13. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An Annotation Scheme for Information Status in Dialogue. In Proceedings of the 4th Language Resources and Evaluation Conference (LREC 2004), pages 1023–1026, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Malvina+Nissim%2C+Shipra+Dingare%2C+Jean+Carletta%2C+and+Mark+Steedman%2E+2004%2E+An+Annotation+Scheme+for+Information+Status+in+Dialogue%2E+In+Proceedings+of+the+4th+Language+Resources+and+Evaluation+Conference+)
14. Patrizia Paggio. 2006. Annotating Information Structure in a Corpus of Spoken Danish. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), pages 1606–1609, Genova, Italy. [find](https://scholar.google.com/scholar?q=Patrizia+Paggio%2E+2006%2E+Annotating+Information+Structure+in+a+Corpus+of+Spoken+Danish%2E+In+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2006%29%2C+pages+1606%E2%80%9316) [our card](https://md.research.elamur.ai/free/papers/paggio-2006-annotating-information-structure-corpus-spoken/)
15. Massimo Poesio and Renata Vieira. 1998. A Corpus-based Investigation of Definite Description Use. Computational Linguistics, 24(2):183–216. [find](https://scholar.google.com/scholar?q=Massimo+Poesio+and+Renata+Vieira%2E+1998%2E+A+Corpus%2Dbased+Investigation+of+Definite+Description+Use%2E+Computational+Linguistics%2C+24%282%29%3A183%E2%80%93216%2E)
16. Livia Polanyi, Martin van den Berg, and David Ahn. 2003. Discourse Structure and Sentential Information Structure. An Initial Proposal. Journal of Logic, Language and Information, 12(3):337–350. [find](https://scholar.google.com/scholar?q=Livia+Polanyi%2C+Martin+van+den+Berg%2C+and+David+Ahn%2E+2003%2E+Discourse+Structure+and+Sentential+Information+Structure%2E+An+Initial+Proposal%2E+Journal+of+Logic%2C+Language+and+Information%2C+12%283%29%3A337%E2%80%93350%2E)
17. Ellen F. Prince. 1981. Toward a Taxonomy of Given-New Information. Radical Pragmatics, pages 223–55. [find](https://scholar.google.com/scholar?q=Ellen+F%2E+Prince%2E+1981%2E+Toward+a+Taxonomy+of+Given%2DNew+Information%2E+Radical+Pragmatics%2C+pages+223%E2%80%9355%2E)
18. David Reitter and Manfred Stede. 2003. Step by step: underspecified markup in incremental rhetorical analysis. In Proc. of the 4th International Workshop on Linguistically Interpreted Corpora (LINC-03), pages 77–84, Budapest, Hungary. [find](https://scholar.google.com/scholar?q=David+Reitter+and+Manfred+Stede%2E+2003%2E+Step+by+step%3A+underspecified+markup+in+incremental+rhetorical+analysis%2E+In+Proc%2E+of+the+4th+International+Workshop+on+Linguistically+Interpreted+Corpora+%28LINC%2D03)
19. Arndt Riester, Lorena Killmann, David Lorenz, and Melanie Portz, 2008. Richtlinien zur Annotation von Gegebenheit und Kontrast in Projekt A1 SFB 732 (draft version as of March 2008). University of Stuttgart, Germany. Manuscript. [find](https://scholar.google.com/scholar?q=Arndt+Riester%2C+Lorena+Killmann%2C+David+Lorenz%2C+and+Melanie+Portz%2C+2008%2E+Richtlinien+zur+Annotation+von+Gegebenheit+und+Kontrast+in+Projekt+A1+SFB+732+%28draft+version+as+of+March+2008%29%2E+University+of+Stu)
20. Arndt Riester. 2008. A semantic explication of Information Status and the underspecification of the recipients’ knowledge. In A. Gronn, editor, Proceedings of Sinn und Bedeutung 12, Oslo. [find](https://scholar.google.com/scholar?q=Arndt+Riester%2E+2008%2E+A+semantic+explication+of+Information+Status+and+the+underspecification+of+the+recipients%E2%80%99+knowledge%2E+In+A%2E+Gronn%2C+editor%2C+Proceedings+of+Sinn+und+Bedeutung+12%2C+Oslo%2E)
21. Susanne Salmon-Alt and Renata Vieira. 2002. Nominal Expressions in Multilingual Corpora. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC 2002), pages 1627–1634, Las Palmas, Canary Islands, Spain. [find](https://scholar.google.com/scholar?q=Susanne+Salmon%2DAlt+and+Renata+Vieira%2E+2002%2E+Nominal+Expressions+in+Multilingual+Corpora%2E+In+Proceedings+of+the+3rd+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2002%29%2C+pages+1627)
22. Thomas Schmidt. 2004. EXMARaLDA - ein System zur computergestützten Diskurstranskription. In Alexander Mehler and Henning Lobin, editors, Automatische Tex-tanalyse - Systeme und Methoden zur Annotation und Analyse natürlichsprachlicher Texte, pages 203–218. Wiesbaden: VS Verlag für Sozialwissensc… [find](https://scholar.google.com/scholar?q=Thomas+Schmidt%2E+2004%2E+EXMARaLDA+%2D+ein+System+zur+computergestu%CC%88tzten+Diskurstranskription%2E+In+Alexander+Mehler+and+Henning+Lobin%2C+editors%2C+Automatische+Tex%2Dtanalyse+%2D+Systeme+und+Methoden+zur+Annotati)
23. Petr Sgall, Eva Hajic̆ová, and Jarmila Panevová. 1986. The Meaning of the Sentence in Its Semantic and Pragmatic Aspects. Reidel, Dordrecht. [find](https://scholar.google.com/scholar?q=Petr+Sgall%2C+Eva+Hajic%CC%86ova%CC%81%2C+and+Jarmila+Panevova%CC%81%2E+1986%2E+The+Meaning+of+the+Sentence+in+Its+Semantic+and+Pragmatic+Aspects%2E+Reidel%2C+Dordrecht%2E)
24. Stavros Skopeteas, Ines Fiedler, Sam Hellmuth, Anne Schwarz, Ruben Stoel, Gisbert Fanselow, and Manfred Krifka. 2006. Questionnaire on Information Structure: Reference Manual. Interdisciplinary Studies on Information Structure (ISIS), Vol. 4, Universitätsverlag Potsdam, Potsdam, Germany. [find](https://scholar.google.com/scholar?q=Stavros+Skopeteas%2C+Ines+Fiedler%2C+Sam+Hellmuth%2C+Anne+Schwarz%2C+Ruben+Stoel%2C+Gisbert+Fanselow%2C+and+Manfred+Krifka%2E+2006%2E+Questionnaire+on+Information+Structure%3A+Reference+Manual%2E+Interdisciplinary+Studie)
25. Jennifer Spenader. 2003. Between binding and accomodation. In Peter Kuehnlein, Hannes Rieser, and Henk Zeevat, editors, Perspectives on Dialogue in the New Millennium, pages 79–110. John Benjamins. [find](https://scholar.google.com/scholar?q=Jennifer+Spenader%2E+2003%2E+Between+binding+and+accomodation%2E+In+Peter+Kuehnlein%2C+Hannes+Rieser%2C+and+Henk+Zeevat%2C+editors%2C+Perspectives+on+Dialogue+in+the+New+Millennium%2C+pages+79%E2%80%93110%2E+John+Benjamins%2E)
26. Manfred Stede. 2004. The Potsdam Commentary Corpus. In Bonnie Webber and Donna K. Byron, editors, Association for Computational Linguistics (ACL) 2004 Workshop on Discourse Annotation, pages 96–102, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Manfred+Stede%2E+2004%2E+The+Potsdam+Commentary+Corpus%2E+In+Bonnie+Webber+and+Donna+K%2E+Byron%2C+editors%2C+Association+for+Computational+Linguistics+%28ACL%29+2004+Workshop+on+Discourse+Annotation%2C+pages+96%E2%80%93102%2C+B)
27. Michael Strube. 1998. Never Look Back: An Alternative to Centering. In Proceedings of COLING-ACL ’98, pages 1251–1257, Montréal, Québec, Canada. [find](https://scholar.google.com/scholar?q=Michael+Strube%2E+1998%2E+Never+Look+Back%3A+An+Alternative+to+Centering%2E+In+Proceedings+of+COLING%2DACL+%E2%80%9998%2C+pages+1251%E2%80%931257%2C+Montre%CC%81al%2C+Que%CC%81bec%2C+Canada%2E)
28. Kater̆ina Veselá, Jirı Havelka, and Eva Hajic̆ová. 2004. Annotators’ agreement: The Case of Topic-Focus Articulation. In Proceedings of the 4th Language Resources and Evaluation Conference (LREC 2004), pages 2191– 2194, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Kater%CC%86ina+Vesela%CC%81%2C+Jir%C4%B1+Havelka%2C+and+Eva+Hajic%CC%86ova%CC%81%2E+2004%2E+Annotators%E2%80%99+agreement%3A+The+Case+of+Topic%2DFocus+Articulation%2E+In+Proceedings+of+the+4th+Language+Resources+and+Evaluation+Conference+%28LREC+200)