---
title: "Beyond semantics — paper by Dipper et al. (2013)"
description: "The editors argue that such phenomena are rarely marked explicitly in text. 27 references."
authors: ["Stefanie Dipper", "Heike Zinsmeister", "Bonnie Webber"]
year: 2013
venue: "Dialogue & Discourse Volume 4"
source: "https://aclanthology.org/2013.dnd-4.14/"
doi: "10.5210/dad.2013.200"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dipper-2013-beyond-semantics-challenges-annotating-pragmatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dipper-2013-beyond-semantics-challenges-annotating-pragmatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/) / Beyond semantics: the challenges of annotating pragmatic and discourse phenomena

# Beyond semantics: the challenges of annotating pragmatic and discourse phenomena

**Authors:** [Stefanie Dipper](https://md.research.elamur.ai/free/en/authors/stefanie-dipper/), [Heike Zinsmeister](https://md.research.elamur.ai/free/en/authors/heike-zinsmeister/) and [Bonnie Webber](https://md.research.elamur.ai/free/en/authors/bonnie-webber/) · **Year:** 2013 · **Venue:** [Dialogue & Discourse Volume 4](https://md.research.elamur.ai/free/en/venues/dnd/) ([Dialogue & Discourse 2013](https://md.research.elamur.ai/free/en/venues/dnd/2013/)) · **Publisher:** [Linguistic Society of America](https://md.research.elamur.ai/free/en/publishers/linguistic-society-of-america/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/2013.dnd-4.14/) · [PDF](https://aclanthology.org/2013.dnd-4.14.pdf) · [DOI 10.5210/dad.2013.200](https://doi.org/10.5210/dad.2013.200) · [OpenAlex](https://openalex.org/W4411021050)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This piece is the editors' introduction to a Dialogue & Discourse special issue on annotating meaning that lies beyond sentence semantics: discourse relations, coherence, dialogue acts, specificity, inference-triggering verb relations and information structure. The editors argue that such phenomena are rarely marked explicitly in text. Annotators must reconstruct them from context, so agreement is often weak, and theories built on invented examples transfer poorly to real corpora. They survey a common remedy found across the collected papers: replacing an abstract judgement with a more tangible proxy. Examples include inserting a connective, rephrasing with a test frame, using translation equivalents, following morphological cues, or answering a question-based decision tree. The issue gathers twelve papers on corpora for German, English, Turkish, Danish and other languages, together with classifiers trained on them. The editors' broader aim is to bring theoretical linguists and computational annotators closer together. Operationalized concepts should yield more reliable annotation, corpus-based testing of hypotheses, and better use of pragmatic knowledge in NLP.

## Key points
- Earlier work (Ritz et al., 2008) found that annotating information structure in German often produces Cohen's κ well below 0.6. Artstein and Poesio (2008) report similarly low κ for dialogue acts, discourse segmentation and word sense tagging.
- Penn Discourse Treebank practice serves as the model proxy: for implicit relations, annotators supply an overt connective. Versley & Gastel extend this in TüBa-D/Z with substitution and insertion tests, paraphrases, nominalizations and explicit questions under discussion. Their scheme organizes relations into five classes: temporal, comparison, contingency, expansion and reporting.
- Cartoni, Zufferey & Meyer approach connective senses through "translation spotting" in a parallel corpus. They train a Maximum Entropy classifier that separates six senses of English *while*.
- Several resources adapt or build on PDTB-style or existing annotation:
  - The Turkish Discourse Bank adapts PDTB-style annotation and treats nominalizations as discourse arguments.
  - The Copenhagen Dependency Treebanks place discourse annotation on top of dependency syntax instead of raw strings.
  - Taboada & Das add signalling cues to the existing RST Discourse Treebank.
- Louis & Nenkova found that 64% of sentences received the same general/specific label from at least four judges. Their classifier's confidence tracked human agreement. In total, the call drew 32 abstracts and 16 submissions, of which 12 papers were accepted (10 regular papers and 2 short notes).

## Interesting ideas
- Swap a vague label for an operational test that annotators can actually perform, such as inserting a connective or rewriting a sentence as "Concerning X, …" to check whether X is the topic; this makes hidden judgements checkable and repeatable.
- Use a cross-lingual translation of an ambiguous word as evidence of its sense, since another language often splits senses that English merges (concessive versus contrastive versus temporal *while*).
- Separate lexical givenness (a word repeated) from referential givenness (the same entity mentioned again), as RefLex does; the two layers do not always coincide and both bear on focus.
- Anchor discourse relations to dependency structure rather than to character spans, as the Copenhagen Dependency Treebanks do; argument boundaries then follow syntactic heads.
- Treat classifier confidence as a signal of how prototypical an item is, since items on which humans agree tend to be the ones a model is sure about.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): prag could decide implicit discourse relations and aboutness topics by trying candidate connectives or topic frames on the parsed clause pair, and record which test succeeded as the explanation.
- **I2** (learn): translation equivalents from parallel text could act as an additional witness when building silver sense labels for connectives, with disagreements flagged for the LLM teacher.
- **I3** (world): entity tracking could keep the two layers apart, so that "a man… the man" marks coreference while a mere repeated noun does not wrongly merge two entities.
- **I4** (tree): discourse links could be attached to clause heads in the UD tree, letting relation arguments be read straight off subtrees and keeping the tree-to-frames handoff explainable.
- **I5** (learn): when evaluating, low-confidence or witness-disagreement items can be split out as a likely non-prototypical set, rather than being counted as plain errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 255 works · 68,771 citations
- [Universität Hamburg](https://md.research.elamur.ai/free/en/universities/universitat-hamburg/) — University, Hamburg, Germany · 39 works · 4,669 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 394 works

## Cite

Cite as: Stefanie Dipper, Heike Zinsmeister and Bonnie Webber. 2013. Beyond semantics: the challenges of annotating pragmatic and discourse phenomena. Dialogue & Discourse Volume 4.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dipper2013beyond,
  title = {{Beyond semantics: the challenges of annotating pragmatic and discourse phenomena}},
  author = {Stefanie Dipper, Heike Zinsmeister and Bonnie Webber},
  booktitle = {Dialogue \& Discourse Volume 4},
  year = {2013},
  doi = {10.5210/dad.2013.200},
  url = {https://doi.org/10.5210/dad.2013.200}
}
```

- [← previous Beowulf and the Origins of the Written Old English Vernacular](https://md.research.elamur.ai/free/en/papers/brown-2019-beowulf-origins-written-old-english/)
- [next → Calle-Martín, Javier & Miguel Ángel Castaño- Gil 2013: A Late Middle English Remedy-book (MS Wellcome 542, ff . 1r–20v). A Scholarly Edition](https://md.research.elamur.ai/free/en/papers/barrera-2019-calle-martin-javier-miguel-angel/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/page/2/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ACE-EDT. Annotation guidelines for entity detection and tracking (EDT). Linguistic Data Consortium, 2004. URL [http://catalog.ldc.upenn.edu/docs/LDC2005T09/guidelines/EnglishEDTV4-2-6.PDF](http://catalog.ldc.upenn.edu/docs/LDC2005T09/guidelines/EnglishEDTV4-2-6.PDF). Version 4.2.6 200400401. [↗](http://catalog.ldc.upenn.edu/docs/LDC2005T09/guidelines/EnglishEDTV4-2-6.PDF)
2. Jan Alexandersson, Bianka Buschbeck-Wolf, Tsutomu Fujinami, Michael Kipp, Stephan Koch, Elisabeth Maier, Norbert Reithinger, Birte Schmitz, and Melanie Siegel. Dialogue acts in VERBMOBIL-2 (Second edition). Verbmobil Report 226, Saarland University, Saarbrücken, Germany, 1998. URL [http://www.coli.uni-sb.de/publikationen/softcopies/Alexandersson:1998:DAV.pdf](http://www.coli.uni-sb.de/publikationen/softcopies/Alexandersson:1998:DAV.pdf). [↗](http://www.coli.uni-sb.de/publikationen/softcopies/Alexandersson:1998:DAV.pdf)
3. James Allen and Mark Core. Draft of DAMSL: Dialogue act markup in several layers. Technical report, University of Rochester, 1997. URL [http://www.cs.rochester.edu/research/speech/damsl/RevisedManual/](http://www.cs.rochester.edu/research/speech/damsl/RevisedManual/). [↗](http://www.cs.rochester.edu/research/speech/damsl/RevisedManual/)
4. Anne H. Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jaqueline C. Kowtko, Jan McAllister, Jim Miller, Cathy Sotillo, Henry Thompson, and Regina Weinert. The HCRC Map Task Corpus. Language and Speech, 34(4): 351–366, 1991. [↗](https://doi.org/10.1177/002383099103400404)
5. [Ron Artstein and Massimo Poesio. Inter-coder agreement for computational linguistics (survey article). Computational Linguistics, 34(4):555–596, 2008.](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/) [↗](https://doi.org/10.1162/coli.07-034-r2)
6. Ada Brunstein. Annotation guidelines for answer types. Technical report, BBN Technologies, 2002. URL [http://catalog.ldc.upenn.edu/docs/LDC2005T33/BBN-Types-Subtypes.html](http://catalog.ldc.upenn.edu/docs/LDC2005T33/BBN-Types-Subtypes.html). [↗](http://catalog.ldc.upenn.edu/docs/LDC2005T33/BBN-Types-Subtypes.html)
7. Jean Carletta, Amy Isard, Stephen Isard, Jacqueline C. Kowtko, Gwyneth Doherty-Sneddon, and Anne H. Anderson. [The reliability of a dialogue structure coding scheme](https://md.research.elamur.ai/free/en/papers/carletta-1997-reliability-dialogue-structure-coding-scheme/). Computational Linguistics, 23(1):13–31, 1997. [↗](https://doi.org/10.5555/972684.972686)
8. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. [Building a discourse-tagged corpus in the framework of Rhetorical Structure Theory](https://md.research.elamur.ai/free/en/papers/carlson-2001-building-discourse-tagged-corpus-framework/). In Jan van Kuppevelt and Ronnie Smith, editors, Current and New Directions in Discourse and Dialogue, pages 85–112. Kluwer Academic Publishers, 2003. [↗](https://doi.org/10.1007/978-94-010-0019-2_5)
9. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. [↗](https://doi.org/10.1177/001316446002000104)
10. VIII Stefanie Dipper and Heike Zinsmeister, editors. Beyond Semantics: Corpus-based Investigations of Pragmatic and Discourse Phenomena. Proceedings of the DGfS Workshop, Göttingen, volume 3 of BLA (Bochumer Linguistische Arbeiten). Institute of Linguistics; Ruhr-University Bochum, 2011. URL [http://www.linguistics.ruhr-uni-bochum.de/bla/](http://www.linguistics.ruhr-uni-bochum.de/bla/). [↗](http://www.linguistics.ruhr-uni-bochum.de/bla/)
11. Charles Fillmore, Josef Ruppenhofer, and Collin F. Baker. FrameNet and representing the link between semantic and syntactic relations. In Chu-Ren Huang and Winfried Lenders, editors, Computational Linguistics and Beyond, Language and Linguistics Monographs Series B. Frontiers in Linguistics I, pages 19–62. Institute of Linguistics, Academia Sinica, Taipei, 2004. [find](https://scholar.google.com/scholar?q=Charles+Fillmore%2C+Josef+Ruppenhofer%2C+and+Collin+F%2E+Baker%2E+FrameNet+and+representing+the+link+between+semantic+and+syntactic+relations%2E+In+Chu%2DRen+Huang+and+Winfried+Lenders%2C+editors%2C+Computational+Lin)
12. Michael Götze, Thomas Weskott, Cornelia Endriss, Ines Fiedler, Stefan Hinterwimmer, Svetlana Petrova, Anne Schwarz, Stavros Skopeteas, and Ruben Stoel. Information Structure. In Stefanie Dipper, Michael Götze, and Stavros Skopeteas, editors, Information Structure in Cross-Linguistic Corpora, number 7 in Interdisciplinary Studies on Information Structure (ISIS), pages 147–187. Universitätsverlag Potsdam, 2007. [find](https://scholar.google.com/scholar?q=Michael+Go%CC%88tze%2C+Thomas+Weskott%2C+Cornelia+Endriss%2C+Ines+Fiedler%2C+Stefan+Hinterwimmer%2C+Svetlana+Petrova%2C+Anne+Schwarz%2C+Stavros+Skopeteas%2C+and+Ruben+Stoel%2E+Information+Structure%2E+In+Stefanie+Dipper%2C+Mich)
13. [Lynette Hirschman and Nancy Chinchor. MUC-7 coreference task definition—version 3.0. In Proceedings of the 7th Message Understanding Conference (MUC-7), Fairfax, Virginia, 1998.](https://md.research.elamur.ai/free/en/papers/hirschman-1998-appendix-f-muc-7-coreference/)
14. Dan Jurafsky, Liz Shriberg, and Debra Biasca. Switchboard SWBD-DAMSL shallow-discourse-function annotation. Coders manual, draft 13. Technical Report RT 97-02, University of Colorado at Boulder & SRI International, 1997. URL [http://www.stanford.edu/˜jurafsky/ws97/manual.august1.html](http://www.stanford.edu/˜jurafsky/ws97/manual.august1.html). [↗](http://www.stanford.edu/˜jurafsky/ws97/manual.august1.html)
15. J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174, 1977. [↗](https://doi.org/10.2307/2529310)
16. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. [The NomBank project: An interim report](https://md.research.elamur.ai/free/en/papers/meyers-2004-nombank-project-interim-report/). In Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 24–31, Boston, Massachusetts, 2004.
17. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. [A semantic concordance](https://md.research.elamur.ai/free/en/papers/miller-1993-semantic-concordance/). In Proceedings of the Workshop on Human Language Technology (HLT-93), pages 303–308, Stroudsburg, Pennsylvania, 1993.
18. MUC-6. Named Entity task definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 317–322, Columbia, Maryland, 1995. URL http://aclweb.org/anthology/M/M95/M95-1024.pdf. Version 2.1. [↗](https://aclanthology.org/M95-1024/)
19. Martha Palmer, Daniel Gildea, and Paul Kingsbury. [The Proposition Bank: An annotated corpus of semantic roles](https://md.research.elamur.ai/free/en/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/). Computational Linguistics, 31(1):71–105, 2005. [↗](https://doi.org/10.1162/0891201053630264)
20. Rebecca J. Passonneau. Instructions for Applying Discourse Reference Annotation for Multiple Applications (DRAMA). Unpublished. Department of Computer Science, Columbia University, 1996. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau%2E+Instructions+for+Applying+Discourse+Reference+Annotation+for+Multiple+Applications+%28DRAMA%29%2E+Unpublished%2E+Department+of+Computer+Science%2C+Columbia+University%2C+1996%2E)
21. Massimo Poesio. MATE dialogue annotation guidelines: Coreference. MATE Deliverable D2.1, pages 134–187, 2000. URL [http://www.andreasmengel.de/pubs/mdag.pdf](http://www.andreasmengel.de/pubs/mdag.pdf). [↗](http://www.andreasmengel.de/pubs/mdag.pdf)
22. IX Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. [The Penn Discourse Treebank 2.0](https://md.research.elamur.ai/free/en/papers/prasad-2008-penn-discourse-treebank-2-0/). In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC’08), pages 2961–2968, Marrakech, Morocco, 2008.
23. James Pustejovsky, José Castaño, Robert Ingria, Roser Saurı́, Robert Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir Radev. TimeML: Robust specification of event and temporal expressions in text. In Mark T. Maybury, editor, New directions in Question Answering. Papers from the 2003 AAAI Spring Symposium, pages 28–34. [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Jose%CC%81+Castan%CC%83o%2C+Robert+Ingria%2C+Roser+Saur%C4%B1%CC%81%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+Graham+Katz%2C+and+Dragomir+Radev%2E+TimeML%3A+Robust+specification+of+event+and+temporal+expressions+in+tex)
24. AAAI Press, Menlo Park, California, 2003a. [find](https://scholar.google.com/scholar?q=AAAI+Press%2C+Menlo+Park%2C+California%2C+2003a%2E)
25. James Pustejovsky, Patrick Hanks, Roser Saurı́, Andrew See, Robert Gaizauskas, Andrea Setzer, Dragomir Radev, Beth Sundheim, David Day, Lisa Ferro, and Marcia Lazo. The TIMEBANK Corpus. In Proceedings of Corpus Linguistics, pages 647–656, 2003b. [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Patrick+Hanks%2C+Roser+Saur%C4%B1%CC%81%2C+Andrew+See%2C+Robert+Gaizauskas%2C+Andrea+Setzer%2C+Dragomir+Radev%2C+Beth+Sundheim%2C+David+Day%2C+Lisa+Ferro%2C+and+Marcia+Lazo%2E+The+TIMEBANK+Corpus%2E+In+Proceedings)
26. Julia Ritz, Stefanie Dipper, and Michael Götze. [Annotation of information structure: An evaluation across different types of texts](https://md.research.elamur.ai/free/en/papers/ritz-2008-annotation-information-structure-evaluation-across/). In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC’08), pages 2137–2142, Marrakech, Morocco, 2008.
27. Yuping Zhou and Nianwen Xue. PDTB-style discourse annotation of Chinese text. In Proceedings the 50th Annual Meeting of the Association for Computational Linguistics (ACL’12), pages 69– 77, Jeju, South Korea, 2012. [find](https://scholar.google.com/scholar?q=Yuping+Zhou+and+Nianwen+Xue%2E+PDTB%2Dstyle+discourse+annotation+of+Chinese+text%2E+In+Proceedings+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%E2%80%9912%29%2C+pages+69%E2%80%93+77%2C+Jeju%2C+Sout)

*Updated 24 October 2026*