---
title: "Intonational Boundaries, Speech Repairs, and Discourse"
description: "They build a statistical language model that predicts the next word together with its POS… 77% recall. 28 references, cited by 7 papers here."
authors: ["Peter A. Heeman", "James F. Allen"]
year: 1997
venue: "35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/P97-1033/"
doi: "10.3115/976909.979650"
arxiv: "cmp-lg/9704008"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/heeman-1997-intonational-boundaries-speech-repairs-discourse/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/heeman-1997-intonational-boundaries-speech-repairs-discourse/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1997](https://md.research.elamur.ai/free/en/papers/1997/) / Intonational Boundaries, Speech Repairs, and Discourse Markers: Modeling Spoken Dialog

# Intonational Boundaries, Speech Repairs, and Discourse Markers: Modeling Spoken Dialog

**Authors:** [Peter A. Heeman](https://md.research.elamur.ai/free/en/authors/peter-a-heeman/) and [James F. Allen](https://md.research.elamur.ai/free/en/authors/james-f-allen/) · **Year:** 1997 · **Venue:** [35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 1997](https://md.research.elamur.ai/free/en/venues/acl/1997/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P97-1033/) · [PDF](https://aclanthology.org/P97-1033.pdf) · [DOI 10.3115/976909.979650](https://doi.org/10.3115/976909.979650) · [arXiv cmp-lg/9704008](https://arxiv.org/abs/cmp-lg/9704008) · [OpenAlex](https://openalex.org/W2143179123)

**Citations:** 32 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/heeman-1997-intonational-boundaries-speech-repairs-discourse/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Spoken dialog is hard for understanding systems because a speaker's turn must first be split into intonational phrases, cleared of speech repairs (stretches the speaker retracts or repeats), and checked for discourse markers such as "well" or "okay". Earlier work handled these problems one at a time. Heeman and Allen argue that they depend on each other and on syntax, so they should be solved jointly and early, inside the speech recognizer's language model. They build a statistical language model that predicts the next word together with its POS tag (Penn Treebank-based tags extended with discourse-marker tags), boundary tones, and repair events. Probabilities are estimated with decision trees that query clustered word and POS encodings, and the model also uses pause durations. Training and testing use the hand-annotated Trains corpus of task-oriented problem-solving dialogs. Modeling the phenomena together improves each task. POS tagging improves by 8.6%, and perplexity is about 14% lower than with word-based models. The paper shows that disfluency handling, segmentation and tagging work better as a single process than as separate stages.

## Key points
- **Data.** The Trains corpus has 98 dialogs, 34 speakers, about 58.3k words and 6163 turns. It contains 8278 discourse markers, 10,947 boundary tones (5535 of them turn-internal), and 1128 editing terms.
- **Repair types.** Repairs fall into three hand-annotated classes, with 671 fresh starts, 1302 modification repairs and 423 abridged repairs. The annotation marks word matches (m), replacements (r), editing terms (et) and the interruption point (ip:can / ip:mod / ip:abr).
- **Model structure.** The model factors Pr(W,P) into a POS-prediction part and a word-given-POS part. Both parts are estimated by decision trees with held-out smoothing. Brown et al. clustering gives a binary encoding for tags, and a separate word tree is built under each POS tag.
- **Interactions between tasks.**
  - Silence clearly helps boundary-tone detection.
  - Adding repair modeling improves tone detection further.
  - Modeling repair correction improves repair detection.
  - Adding tone detection improves both detection and correction of repairs.
  - In the full model, repair detection reaches roughly 77% recall and 85% precision.
- **Comparison with prior work.** Other systems are hard to compare because they use different corpora and input assumptions. Wang and Hirschberg classify boundaries point by point, whereas this model searches for the best interpretation of the whole turn.

## Interesting ideas
- Treating boundaries, repairs and discourse markers as hidden events placed between words lets one joint search decide them together with tags, so errors in one decision no longer cascade into the next.
- Words are modeled as a refinement *inside* their POS class, with one word cluster tree per tag, which avoids splitting sparse data across two independent sources.
- Repair annotation records explicit match and replacement links between the reparandum and the alteration, so a correction carries its own justification through parallel structure.
- Discourse markers get their own tags, which separates their pragmatic use (acknowledging, opening a new utterance, editing) from their literal lexical use.
- The paper evaluates by adding one component at a time and measuring every other task each time, which exposes cross-task effects.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The tagger could predict "sentence break / disfluency / discourse-marker use" labels on the same lattice as POS tags, rather than running a separate preprocessing splitter.
- **I2** (tokens): Lexical features in morphology and lemma models could be backed off hierarchically through the tag, which yields interpretable binary questions instead of opaque embeddings.
- **I3** (tree): When the parser meets self-corrections in dialog or forum text, it could attach the reparandum with an explainable "replaced-by" relation and parse only the alteration.
- **I4** (frames): The prag crate could take the tagger's discourse-marker decision as a speech-act cue, for example reading "okay" as an acknowledgment act rather than an adjective.
- **I5** (learn): Evaluation could run the same ablation grid across stages, checking whether a new repair rule helps or hurts parsing and frame accuracy as well as its own metric.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Rochester](https://md.research.elamur.ai/free/en/universities/university-of-rochester/) — University, Rochester, United States · 36 works · 7,398 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🔊 Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) · 452 works

🧭 Topics (2): [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works · [🧵 Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/) · 466 works

## Cite

Cite as: Peter A. Heeman and James F. Allen. 1997. Intonational Boundaries, Speech Repairs, and Discourse Markers: Modeling Spoken Dialog. 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{heeman1997intonational,
  title = {{Intonational Boundaries, Speech Repairs, and Discourse Markers: Modeling Spoken Dialog}},
  author = {Peter A. Heeman and James F. Allen},
  booktitle = {35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {1997},
  doi = {10.3115/976909.979650},
  eprint = {cmp-lg/9704008},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3115/976909.979650}
}
```

- [← previous Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages](https://md.research.elamur.ai/free/en/papers/hearst-1997-text-tiling-segmenting-text-into/)
- [next → Can pitch accent type convey information status in yes-no questions?](https://md.research.elamur.ai/free/en/papers/grice-1997-can-pitch-accent-type-convey/)
- [↑ back to papers of 1997](https://md.research.elamur.ai/free/en/papers/1997/)

## References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allen, J. F., L. Schubert, G. Ferguson, P. Heeman, C. Hwang, T. Kato, M. Light, N. Martin, B. Miller, M. Poesio, and D. Traum. 1995. The Trains project: A case study in building a conversational planning agent. Journal of Experimental and Theoretical AI, 7:7-48. [↗](https://doi.org/10.1080/09528139508953799)
2. Bahl, L. R., P. F. Brown, P. V. deSouza, and R. L. Mercer. 1989. A tree-based statistical language model for natural lnaguage speech recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 36(7):1001-1008. [find](https://scholar.google.com/scholar?q=Bahl%2C+L%2E+R%2E%2C+P%2E+F%2E+Brown%2C+P%2E+V%2E+deSouza%2C+and+R%2E+L%2E+Mercer%2E+1989%2E+A+tree%2Dbased+statistical+language+model+for+natural+lnaguage+speech+recognition%2E+IEEE+Transactions+on+Acoustics%2C+Speech%2C+and+Signal+Pro)
3. Bear, J., J. Dowding, and E. Shriberg. 1992. Integrating multiple knowledge sources for detection and correction of repairs in human-computer dialog. In Proceedings of the 3 0 th Annual Meeting of the Association for Computational Linguistics, pages 56-63. [find](https://scholar.google.com/scholar?q=Bear%2C+J%2E%2C+J%2E+Dowding%2C+and+E%2E+Shriberg%2E+1992%2E+Integrating+multiple+knowledge+sources+for+detection+and+correction+of+repairs+in+human%2Dcomputer+dialog%2E+In+Proceedings+of+the+3+0+th+Annual+Meeting+of+the)
4. Black, E., F. Jelinek, J. Lafferty, R. Mercer, and S. Roukos. 1992. Decision tree models applied to the labeling of text with parts-of-speech. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 117-121. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Black%2C+E%2E%2C+F%2E+Jelinek%2C+J%2E+Lafferty%2C+R%2E+Mercer%2C+and+S%2E+Roukos%2E+1992%2E+Decision+tree+models+applied+to+the+labeling+of+text+with+parts%2Dof%2Dspeech%2E+In+Proceedings+of+the+DARPA+Speech+and+Natural+Language+W)
5. Breiman, L., J. H. Friedman, R. A. Olshen, and C. J. Stone. 1984. Classification and Regression Trees. Monterrey, CA: Wadsworth & Brooks. [find](https://scholar.google.com/scholar?q=Breiman%2C+L%2E%2C+J%2E+H%2E+Friedman%2C+R%2E+A%2E+Olshen%2C+and+C%2E+J%2E+Stone%2E+1984%2E+Classification+and+Regression+Trees%2E+Monterrey%2C+CA%3A+Wadsworth+%26+Brooks%2E)
6. Brown, P. F., V. J. Della Pietra, P. V. deSouza, J. C. Lai, and R. L. Mercer. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467-479. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E+F%2E%2C+V%2E+J%2E+Della+Pietra%2C+P%2E+V%2E+deSouza%2C+J%2E+C%2E+Lai%2C+and+R%2E+L%2E+Mercer%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+Linguistics%2C+18%284%29%3A467%2D479%2E)
7. Byron, D. K. and P. A. Heeman. 1997. Discourse marker use in task-oriented spoken dialog. In Proceedings of the 5 th European Conference on Speech Communication and Technology (Eurospeech), Rhodes, Greece. [find](https://scholar.google.com/scholar?q=Byron%2C+D%2E+K%2E+and+P%2E+A%2E+Heeman%2E+1997%2E+Discourse+marker+use+in+task%2Doriented+spoken+dialog%2E+In+Proceedings+of+the+5+th+European+Conference+on+Speech+Communication+and+Technology+%28Eurospeech%29%2C+Rhodes%2C+Gr)
8. Entropic Research Laboratory, Inc., 1994. Aligner Reference Manual. Version 1.3. [find](https://scholar.google.com/scholar?q=Entropic+Research+Laboratory%2C+Inc%2E%2C+1994%2E+Aligner+Reference+Manual%2E+Version+1%2E3%2E)
9. Heeman, P. and J. Allen. 1994. Detecting and correcting speech repairs. In Proceedings of the 3 2 th Annual Meeting of the Association for Computational Linguistics, pages 295-302, Las Cruces, New Mexico, June. [find](https://scholar.google.com/scholar?q=Heeman%2C+P%2E+and+J%2E+Allen%2E+1994%2E+Detecting+and+correcting+speech+repairs%2E+In+Proceedings+of+the+3+2+th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+295%2D302%2C+Las+Cruces%2C+New+Mex)
10. Heeman, P. A. 1997. Speech repairs, intonational boundaries and discourse markers: Modeling speakers' utterances in spoken dialog. Doctoral dissertation. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9712009)
11. Heeman, P. A. and J. F. Allen. 1995. The Trains spoken dialog corpus. CD-ROM, Linguistics Data Consortium. [find](https://scholar.google.com/scholar?q=Heeman%2C+P%2E+A%2E+and+J%2E+F%2E+Allen%2E+1995%2E+The+Trains+spoken+dialog+corpus%2E+CD%2DROM%2C+Linguistics+Data+Consortium%2E)
12. Heeman, P. A. and J. F. Allen. 1997. Incorporating POS tagging into language modeling. In Proceedings of the 5 th European Conference on Speech Communication and Technology (Eurospeech), Rhodes, Greece. [find](https://scholar.google.com/scholar?q=Heeman%2C+P%2E+A%2E+and+J%2E+F%2E+Allen%2E+1997%2E+Incorporating+POS+tagging+into+language+modeling%2E+In+Proceedings+of+the+5+th+European+Conference+on+Speech+Communication+and+Technology+%28Eurospeech%29%2C+Rhodes%2C+Greec)
13. Heeman, P. A., K. Loken-Kim, and J. F. Allen. 1996. Combining the detection and correction of speech repairs. In Proceedings of the 4rd International Conference on Spoken Language Processing (ICSLP-96), pages 358-361, Philadephia, October. ttindle, D. 1983. Deterministic parsing of syntactic nonfluencies. In Proceedings of the 21 st Annual Meeting of the Association for Computational Linguistics, pages 123-128. [find](https://scholar.google.com/scholar?q=Heeman%2C+P%2E+A%2E%2C+K%2E+Loken%2DKim%2C+and+J%2E+F%2E+Allen%2E+1996%2E+Combining+the+detection+and+correction+of+speech+repairs%2E+In+Proceedings+of+the+4rd+International+Conference+on+Spoken+Language+Processing+%28ICSLP%2D96)
14. Hirschberg, J. and D. Litman. 1993. [Empirical studies on the disambiguation of cue phrases](https://md.research.elamur.ai/free/en/papers/hirschberg-1993-empirical-studies-disambiguation-cue-phrases/). Computational Linguistics, 19(3):501-530.
15. Katz, S. M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, pages 400-401, March. [↗](https://doi.org/10.1109/tassp.1987.1165125)
16. Kompe, R., A. Battiner, A. Kiefling, U. Kilian, H. Niemann, E. NSth, and P. Regel-Brietzmann. 1994. Automatic classification of prosodically marked phrase boundaries in german. In Proceedings of the International Conference on Audio, Speech and Signal Processing (ICASSP), pages 173-176, Adelaide. [↗](https://doi.org/10.22028/d291-25093)
17. Levelt, W. J. M. 1983. Monitoring and self-repair in speech. Cognition, 14:41-104. [↗](https://doi.org/10.1016/0010-0277(83)90026-4)
18. Magerman, D. M. 1995. Statistical decision trees fol parsing. In Proceedings of the 3 3 th Annual Meeting of the Association for Computational Linguistics, pages 7-14, Cambridge, MA, June. [find](https://scholar.google.com/scholar?q=Magerman%2C+D%2E+M%2E+1995%2E+Statistical+decision+trees+fol+parsing%2E+In+Proceedings+of+the+3+3+th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+7%2D14%2C+Cambridge%2C+MA%2C+June%2E)
19. Marcus, M. P., B. Santorini, and M. A. Marcinkiewicz. 1993. [Building a large annotated corpus of english: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational Linguistics, 19(2):313-330. [↗](https://doi.org/10.21236/ada273556)
20. Mast, M., R. Kompe, S. Harbeck, A. Kieflling, H. Niemann, E. NSth, E. G. Schukat-Taiamazzini, and V. Warnke. 1996. Dialog act classification with the help of prosody. In Proceedings of the 4rd International Conference on Spoken Language Processing (ICSLP-96), Philadephia, October. [find](https://scholar.google.com/scholar?q=Mast%2C+M%2E%2C+R%2E+Kompe%2C+S%2E+Harbeck%2C+A%2E+Kieflling%2C+H%2E+Niemann%2C+E%2E+NSth%2C+E%2E+G%2E+Schukat%2DTaiamazzini%2C+and+V%2E+Warnke%2E+1996%2E+Dialog+act+classification+with+the+help+of+prosody%2E+In+Proceedings+of+the+4rd+Interna)
21. Nakatani, C. H. and J. Hirschberg. 1994. A corpus-based study of repair cues in spontaneous speech. Journal of the Acoustical Society of America, 95(3):1603-1616. [↗](https://doi.org/10.1121/1.408547)
22. Rosenfeld, R. 1995. The CMU statistical language modeling toolkit and its use in the 1994 ARPA CSR evai-uation. In Proceedings of the ARPA Spoken Language Systems Technology Workshop, San Mateo, California, 1995. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Rosenfeld%2C+R%2E+1995%2E+The+CMU+statistical+language+modeling+toolkit+and+its+use+in+the+1994+ARPA+CSR+evai%2Duation%2E+In+Proceedings+of+the+ARPA+Spoken+Language+Systems+Technology+Workshop%2C+San+Mateo%2C+Calif)
23. Schiffrin, D. 1987. Discourse Markers. New York: Cambridge University Press. [find](https://scholar.google.com/scholar?q=Schiffrin%2C+D%2E+1987%2E+Discourse+Markers%2E+New+York%3A+Cambridge+University+Press%2E)
24. Silverman, K., M. Beckman, J. Pitrelli, M. Ostendorf, C. Wightman, P. Price, J. Pierrehumbert, and J. Hirschberg. 1992. ToBI: A standard for labelling English prosody. In Proceedings of the 2nd International Conference on Spoken Language Processing (ICSLP-92), pages 867-870. [find](https://scholar.google.com/scholar?q=Silverman%2C+K%2E%2C+M%2E+Beckman%2C+J%2E+Pitrelli%2C+M%2E+Ostendorf%2C+C%2E+Wightman%2C+P%2E+Price%2C+J%2E+Pierrehumbert%2C+and+J%2E+Hirschberg%2E+1992%2E+ToBI%3A+A+standard+for+labelling+English+prosody%2E+In+Proceedings+of+the+2nd+Intern)
25. Stolcke, A. and E. Shriberg. 1996a. Automatic linguistic segmentation of conversational speech. In Proceedings of the 4rd International Conference on Spoken Language Processing (1CSLP-96), October. [find](https://scholar.google.com/scholar?q=Stolcke%2C+A%2E+and+E%2E+Shriberg%2E+1996a%2E+Automatic+linguistic+segmentation+of+conversational+speech%2E+In+Proceedings+of+the+4rd+International+Conference+on+Spoken+Language+Processing+%281CSLP%2D96%29%2C+October%2E)
26. Stolcke, A. and E. Shriberg. 1996b. Statistical language modeling for speech disfluencies. In Proceedings of the International Conference on Audio, Speech and Signal Processing (1CASSP), May. [find](https://scholar.google.com/scholar?q=Stolcke%2C+A%2E+and+E%2E+Shriberg%2E+1996b%2E+Statistical+language+modeling+for+speech+disfluencies%2E+In+Proceedings+of+the+International+Conference+on+Audio%2C+Speech+and+Signal+Processing+%281CASSP%29%2C+May%2E)
27. Wang, M. Q. and J. Hirschberg. 1992. Automatic classification of intonational phrase boundaries. Computer Speech and Language, 6:175-196. [find](https://scholar.google.com/scholar?q=Wang%2C+M%2E+Q%2E+and+J%2E+Hirschberg%2E+1992%2E+Automatic+classification+of+intonational+phrase+boundaries%2E+Computer+Speech+and+Language%2C+6%3A175%2D196%2E)
28. Wightman, C. W. and M. Ostendorf. 1994. Automatic labeling of prosodic patterns. IEEE Transactions on speech and audio processing, October. [↗](https://doi.org/10.1109/89.326607)

*Updated 1 November 2026*