--- title: "Adaptive Sentence Boundary Disambiguation — Palmer 1994" description: "The authors decide whether a period, question mark or exclamation mark ends a sentence. 13 references, cited by 14 papers here." authors: ["David D. Palmer", "Marti A. Hearst"] year: 1994 venue: "Fourth Conference on Applied Natural Language Processing" source: "https://aclanthology.org/A94-1013/" doi: "10.3115/974358.974376" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/palmer-1994-adaptive-sentence-boundary-disambiguation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/palmer-1994-adaptive-sentence-boundary-disambiguation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1994](https://md.research.elamur.ai/free/en/papers/1994/) / Adaptive Sentence Boundary Disambiguation # Adaptive Sentence Boundary Disambiguation **Authors:** David D. Palmer and [Marti A. Hearst](https://md.research.elamur.ai/free/en/authors/marti-a-hearst/) · **Year:** 1994 · **Venue:** Fourth Conference on Applied Natural Language Processing **Link:** [aclanthology.org/A94-1013](https://aclanthology.org/A94-1013/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**77** citations](https://openalex.org/works?filter=cites:W2133065881) OpenAlex, 2026-10-14 · [**14** citations in our library](https://md.research.elamur.ai/free/en/papers/palmer-1994-adaptive-sentence-boundary-disambiguation/cited-by/) of the 140,884 papers whose bibliographies we read · [Fourth Conference on Applied Natural Language Processing](https://md.research.elamur.ai/free/en/venues/anlp/) · published by [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/A94-1013/) - [PDF](https://aclanthology.org/A94-1013.pdf) - [DOI 10.3115/974358.974376](https://doi.org/10.3115/974358.974376) - [OpenAlex](https://openalex.org/W2133065881) **Authors (2)** 1. [David D. Palmer](https://openalex.org/A5037867022) [Semantic Scholar](https://www.semanticscholar.org/author/33611948) · h 12 · 674 citations · 25 works 2. [Marti A. Hearst](https://openalex.org/A5019933387) [ORCID](https://orcid.org/0000-0002-4346-1603) · [Semantic Scholar](https://www.semanticscholar.org/author/1716902) · h 62 · 28,774 citations · 279 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about The authors decide whether a period, question mark or exclamation mark ends a sentence. Their method is trainable and does not use hand-made regular expressions. A lexicon gives the part-of-speech probabilities of the tokens around the punctuation mark, and a small neural network reads those probabilities over a window of context. Trained on a corpus of over 27,000 boundary marks, the method labels 98.5% correctly, which is 409 errors out of 27,294 marks, with both sensitivity thresholds at 0.5. Tested on uppercase legal text, it reaches 99.71% and 98.24% on two sets of 5,305 and 9,396 periods. The authors note that the results would probably be lower on lowercase data, and they analyse the errors by cause, such as abbreviations and quotation marks. ## Ideas - I3. Test the thresholds: sweep the two decision thresholds on the dev part of EWT and report how many errors each threshold gives, as the paper does with its two cut-offs. ## For Mova - I1. `en/src/tok.rs` with `en/src/tag.rs`: replace the abbreviation list by the tag distribution of the token before the period, taken from the TnT prior, and a small table-driven score. Measure the boundary errors on EWT test, then count the errors per cause as the paper does. - I2. Error analysis by cause: group the boundary errors into abbreviation, quotation-mark and other cases, and print the groups with `en explain`, so each decision says which cause it belongs to. The code is `en/src/tok.rs` and `en/src/tag.rs`, and the metric is `en tok-eval`. The chapter is [languages/en/07-punctuation-capitals-numbers.md](https://mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/). The paper uses a 1990s corpus and a neural net, so its numbers are not Mova numbers and the method only gives the idea. Measure first on EWT test, twice, to check that the error count is stable. ## Institutions [🎓**University of California, Berkeley** Berkeley · United States 107 works · 205,102 citations](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) · [🔬**Palo Alto Research Center** Palo Alto · United States 9 works · 4,797 citations](https://md.research.elamur.ai/free/en/labs/palo-alto-research-center/) 📚 Shelves 4[Computer Science · 7,249 works](https://md.research.elamur.ai/free/en/shelves/computer-science/) · [Artificial Intelligence · 7,084 works](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [Arts and Humanities · 7,398 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · [Language and Linguistics · 7,340 works](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) 🧭 Topics 2[English · 976 works](https://md.research.elamur.ai/free/en/topics/english/) · [Tokenization & segmentation · 68 works](https://md.research.elamur.ai/free/en/topics/tokenization/) ## Cite Cite as David D. Palmer and Marti A. Hearst. 1994. Adaptive Sentence Boundary Disambiguation. Fourth Conference on Applied Natural Language Processing. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/974358.974376](https://doi.org/10.3115/974358.974376) ```bibtex @inproceedings{palmer1994adaptive, title = {{Adaptive Sentence Boundary Disambiguation}}, author = {David D. Palmer and Marti A. Hearst}, booktitle = {Fourth Conference on Applied Natural Language Processing}, year = {1994}, doi = {10.3115/974358.974376}, url = {https://doi.org/10.3115/974358.974376} } ``` - [← previous Abstract Generation Based on Rhetorical Structure Extraction](https://md.research.elamur.ai/free/en/papers/ono-1994-abstract-generation-based-rhetorical-structure/) - [next → Multi-Tape Two-Level Morphology: A Case Study in Semitic Non-Linear Morphology](https://md.research.elamur.ai/free/en/papers/kiraz-1994-multi-tape-two-level-morphology/) - [↑ back to papers of 1994](https://md.research.elamur.ai/free/en/papers/1994/) ## References 13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Leo Breiman, Jerome H. Friedman, Richard Olshen, and Charles J. Stone. 1984. Classification and regression trees. Wadsworth International Group, Belmont, CA. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2C+Jerome+H%2E+Friedman%2C+Richard+Olshen%2C+and+Charles+J%2E+Stone%2E+1984%2E+Classification+and+regression+trees%2E+Wadsworth+International+Group%2C+Belmont%2C+CA%2E) 2. Kenneth W. Church and Mark Y. Liberman. 1991. A status report on the ACL/DCI. In The Proceedings of the 7th Annual Conference of the UW Centre for the New OED and Tezt Research: Using Corpora, pages 84-91, Oxford. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church+and+Mark+Y%2E+Liberman%2E+1991%2E+A+status+report+on+the+ACL%2FDCI%2E+In+The+Proceedings+of+the+7th+Annual+Conference+of+the+UW+Centre+for+the+New+OED+and+Tezt+Research%3A+Using+Corpora%2C+pages+8) 3. Kenneth W. Church. 1988. A stochastic parts program and noun phrase parser for unrestricted text. In Second Conference on Applied Natural Language Processing, pages 136-143, Austin, TX. [our card](https://md.research.elamur.ai/free/en/papers/church-1988-stochastic-parts-program-noun-phrase/) 4. Doug Cutting, Julian Kupiec, Jan Pedersen, and Penelope Sibun. 1991. A practical part-of-speech tagger. [our card](https://md.research.elamur.ai/free/en/papers/cutting-1992-practical-part-speech-tagger/) 5. Training Error 1.52 0.75 0.043 Cross Error 2.36 2.01 1.88 Testing Errors 1424 409 877 Testing Error (%) 5.22% 1.50% 3.21% Testing Error (%) Training Error 1.05 1.08 0.39 0.27 Cross Error 1.61 2.18 2.27 1.42 Testing Errors 721 409 435 1343 2.64% 1.50% 1.59% 4.92% In The 3rd Conference on Applied Natu… [find](https://scholar.google.com/scholar?q=Training+Error+1%2E52+0%2E75+0%2E043+Cross+Error+2%2E36+2%2E01+1%2E88+Testing+Errors+1424+409+877+Testing+Error+%28%25%29+5%2E22%25+1%2E50%25+3%2E21%25+Testing+Error+%28%25%29+Training+Error+1%2E05+1%2E08+0%2E39+0%2E27+Cross+Error+1%2E61+2%2E18+2%2E2) 6. W. Francis and H. Kucera. 1982. Frequency Analysis of English Usage. Houghton Mifl]in Co., New York. [find](https://scholar.google.com/scholar?q=W%2E+Francis+and+H%2E+Kucera%2E+1982%2E+Frequency+Analysis+of+English+Usage%2E+Houghton+Mifl%5Din+Co%2E%2C+New+York%2E) 7. William A. Gale and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75-102. [our card](https://md.research.elamur.ai/free/en/papers/gale-1993-program-aligning-sentences-bilingual-corpora/) 8. John Hertz, Anders Krogh, and Richard G. Palmer. 1991. Introduction to the theory of neural computation. Santa Fe Institute studies in the sciences of complexity. Addison-Wesley Pub. Co., Redwood City, CA. [find](https://scholar.google.com/scholar?q=John+Hertz%2C+Anders+Krogh%2C+and+Richard+G%2E+Palmer%2E+1991%2E+Introduction+to+the+theory+of+neural+computation%2E+Santa+Fe+Institute+studies+in+the+sciences+of+complexity%2E+Addison%2DWesley+Pub%2E+Co%2E%2C+Redwood+City) 9. Susanne M. Humphrey. 1989. Research on interactive knowledge-based indexing: The medindex prototype. In Symposium on Computer Applications in Medical Care, pages 527-533. [find](https://scholar.google.com/scholar?q=Susanne+M%2E+Humphrey%2E+1989%2E+Research+on+interactive+knowledge%2Dbased+indexing%3A+The+medindex+prototype%2E+In+Symposium+on+Computer+Applications+in+Medical+Care%2C+pages+527%2D533%2E) 10. Martin Kay and Martin R6schelsen. 1993. Text-translation alignment. Computational Linguistics, 19(i):121-142. [our card](https://md.research.elamur.ai/free/en/papers/kay-1993-text-translation-alignment/) 11. Mark Y. Liberman and Kenneth W. Church. 1992. Text analysis and word pronunciation in text-to-speech synthesis. In Sadaoki Furui and Man Mohan Sondhi, editors, Advances in Speech Signal Processing, pages 791-831. Marcel Dekker, Inc. [find](https://scholar.google.com/scholar?q=Mark+Y%2E+Liberman+and+Kenneth+W%2E+Church%2E+1992%2E+Text+analysis+and+word+pronunciation+in+text%2Dto%2Dspeech+synthesis%2E+In+Sadaoki+Furui+and+Man+Mohan+Sondhi%2C+editors%2C+Advances+in+Speech+Signal+Processing%2C+pa) 12. Hans Miiller, V. Amerl, and G. Natalls. 1980. Worterkennungsverfahren als Grundlage einer Unl-versalmethode zur automatlschen Segmentierung yon Texten in Sgtze. Ein Verfahren zur maschineUen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung, 1. [find](https://scholar.google.com/scholar?q=Hans+Miiller%2C+V%2E+Amerl%2C+and+G%2E+Natalls%2E+1980%2E+Worterkennungsverfahren+als+Grundlage+einer+Unl%2Dversalmethode+zur+automatlschen+Segmentierung+yon+Texten+in+Sgtze%2E+Ein+Verfahren+zur+maschineUen+Satzgrenz) 13. Michael D. Riley. 1989. Some applications of tree-based modelling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, pages 339-352. Morgan Kaufrnann. [find](https://scholar.google.com/scholar?q=Michael+D%2E+Riley%2E+1989%2E+Some+applications+of+tree%2Dbased+modelling+to+speech+and+language+indexing%2E+In+Proceedings+of+the+DARPA+Speech+and+Natural+Language+Workshop%2C+pages+339%2D352%2E+Morgan+Kaufrnann%2E) Updated 23 October 2026