--- title: "A Maximum Entropy Approach to Identifying — Reynar 1997" description: "Reynar and Ratnaparkhi address sentence boundary detection: deciding whether each." authors: ["Jeffrey C. Reynar", "Adwait Ratnaparkhi"] year: 1997 venue: "Fifth Conference on Applied Natural Language Processing" source: "https://aclanthology.org/A97-1004/" doi: "10.3115/974557.974561" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/reynar-1997-maximum-entropy-approach-identifying-sentence/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/reynar-1997-maximum-entropy-approach-identifying-sentence/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1997](https://md.research.elamur.ai/free/en/papers/1997/) / A Maximum Entropy Approach to Identifying Sentence Boundaries # A Maximum Entropy Approach to Identifying Sentence Boundaries **Authors:** Jeffrey C. Reynar, [Adwait Ratnaparkhi](https://md.research.elamur.ai/free/en/authors/adwait-ratnaparkhi/) · **Year:** 1997 · **Venue:** Fifth Conference on Applied Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/A97-1004/) · [PDF](https://aclanthology.org/A97-1004.pdf) · [DOI 10.3115/974557.974561](https://doi.org/10.3115/974557.974561) · [OpenAlex](https://openalex.org/W2018560257) **Citations:** 391 citations (openalex-snapshot · 2026-10-14) · [83](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Reynar and Ratnaparkhi address sentence boundary detection: deciding whether each `.`, `?` or `!` in raw text actually ends a sentence. This is harder than it looks, because periods also appear in abbreviations, decimals, e-mail addresses and ellipses. Hand-written rules and exception lists never cover every case. The authors instead frame the task as binary classification with a maximum entropy model trained by Generalized Iterative Scaling. The model uses only the candidate token (split into the part before the mark and the part after it) and the words immediately to its left and right. They built two variants: - a higher-accuracy system with hand-made lists of honorifics and corporate designators; - a "highly portable" system whose only resource is an abbreviation list induced automatically from the annotated training data. Trained on Penn Treebank Wall Street Journal text, the systems reach 98.8% and 98.0% accuracy on Palmer and Hearst's WSJ test set. This matches systems that need POS lexica or far more data. The approach matters because a small boundary-annotated corpus is enough to retrain it for a new genre or Roman-alphabet language. ## Key points - Training data: 39,441 sentences (898,737 words) from the WSJ part of the Penn Treebank (release 2). Sections 03–06 were excluded because they overlap the test data. On a Sun Ultra Sparc, the model needed 18 minutes to train, and processing a single WSJ article takes roughly 1.4 seconds. - Best system results: - WSJ: 98.8% accuracy over 32,173 candidate marks. - Brown corpus: 97.9% over 61,282 candidates, with 750 false positives and 506 false negatives. - Highly portable system results: 98.0% on WSJ and 97.5% on Brown. The drop on Brown shows that training and test genre should match. - Learning curve: with only 500 training sentences, accuracy is 97.6% (best) and 96.5% (portable). Two naive baselines score far lower: - calling every candidate a boundary: 64.0%; - calling only token-final marks boundaries: 78.4%. - Comparison with earlier work: - Palmer & Hearst's SATZ reaches 98.5% on WSJ, but it relies on POS information. - Riley's decision tree scores 99.8% on Brown, but it was trained on 25 million words of Brown itself. ## Interesting ideas - **Inducing the abbreviation lexicon from annotations.** Any whitespace-delimited token in the training data that contains a period not marking a boundary is added to an abbreviation list. This replaces a hand-built resource with an automatically derived one, which helps when adapting tools to languages such as Ukrainian that lack curated lists. - **Splitting the candidate token at the punctuation mark.** Features are computed separately for the part before the mark and the part after it. This handles tokens with internal punctuation (e.g. "D.C.") cleanly and could carry over to tokenization or clitic splitting. - **Generating features from templates.** The designer only specifies the form of the information, such as "word to the left" or "prefix is an induced abbreviation". Concrete features are then produced by scanning the corpus, which keeps the model language-agnostic and cheap to retrain. - **Reporting a learning curve and naive baselines.** Measuring accuracy against training set size, alongside trivial baselines, tells practitioners how much annotation a new domain needs. The authors estimate that producing enough data would take only a few hours of work. - **Running both a resource-rich and a resource-free variant.** Reporting both systems side by side quantifies exactly what the hand-made lists add, which is a useful ablation pattern for any pipeline component. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Treebanks](https://md.research.elamur.ai/free/en/topics/treebanks/) ## Cited by Cited by 83 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 72 not yet. - [Supersense Tagging of Unknown Nouns using Semantic Similarity](https://md.research.elamur.ai/free/en/papers/aclbulk-2005-curran-supersense-tagging-unknown-nouns-semantic/) - [Sentence Boundary Detection and the Problem with the U.S.](https://md.research.elamur.ai/free/en/papers/aclbulk-2009-gillick-sentence-boundary-detection-problem-u/) - [Sentence Boundary Detection: A Long Solved Problem?](https://md.research.elamur.ai/free/en/papers/aclbulk-2012-read-sentence-boundary-detection-long-solved/) - [A Dependency Parser for Tweets](https://md.research.elamur.ai/free/en/papers/aclbulk-2014-kong-dependency-parser-tweets/) - [Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization](https://md.research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/) - [Advances in domain independent linear text segmentation](https://md.research.elamur.ai/free/en/papers/choi-2000-advances-domain-independent-linear-text/) - [Unsupervised Multilingual Sentence Boundary Detection](https://md.research.elamur.ai/free/en/papers/cl-2006-kiss-unsupervised-multilingual-sentence-boundary-detection/) - [Semantic Inference at the Lexical-Syntactic Level for Textual Entailment Recognition](https://md.research.elamur.ai/free/en/papers/haim-2007-semantic-inference-lexical-syntactic-level/) - [Dependency-based paraphrasing for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/marsi-2007-dependency-based-paraphrasing-recognizing-textual/) - [Coreference Resolution with and without Linguistic Knowledge](https://md.research.elamur.ai/free/en/papers/uryupina-2006-coreference-resolution-without-linguistic-knowledge/) - [Error Analysis for Learning-based Coreference Resolution](https://md.research.elamur.ai/free/en/papers/uryupina-2008-error-analysis-learning-based-coreference/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{reynar1997maximum, title = {{A Maximum Entropy Approach to Identifying Sentence Boundaries}}, author = {Reynar, Jeffrey C. and Ratnaparkhi, Adwait}, booktitle = {Fifth Conference on Applied Natural Language Processing}, year = {1997}, doi = {10.3115/974557.974561}, url = {https://doi.org/10.3115/974557.974561} } ``` - [← previous A Frame-Semantic Approach to Semantic Annotation](https://md.research.elamur.ai/free/en/papers/lowe-1997-frame-semantic-approach-semantic-annotation/) - [next → A non-projective dependency parser](https://md.research.elamur.ai/free/en/papers/tapanainen-1997-non-projective-dependency-parser/) - [↑ back to papers of 1997](https://md.research.elamur.ai/free/en/papers/1997/) ## References 12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Brill, Eric. 1994. Some advances in transformation-based part-of-speech tagging. In Proceedings of the Twelfth National Conference on Artificial Intelligence, volume 1, pages 722-727. [find](https://scholar.google.com/scholar?q=Brill%2C+Eric%2E+1994%2E+Some+advances+in+transformation%2Dbased+part%2Dof%2Dspeech+tagging%2E+In+Proceedings+of+the+Twelfth+National+Conference+on+Artificial+Intelligence%2C+volume+1%2C+pages+722%2D727%2E) 2. Collins, Michael. 1996. A new statistical parser Imsed on bigram lexical dependencies. In Proceed-Number of sentences in training corpus 1000 2000 4000 8000 16000 98.4% 98.0% 98.4% 98.3% 98.3% 97.3% 97.3% 97.6% 97.6% 97.8% 39441 98.8~Z~, 98.0% ings of the 34 tl' Annual :Uce ting of th~ Association f… [find](https://scholar.google.com/scholar?q=Collins%2C+Michael%2E+1996%2E+A+new+statistical+parser+Imsed+on+bigram+lexical+dependencies%2E+In+Proceed%2DNumber+of+sentences+in+training+corpus+1000+2000+4000+8000+16000+98%2E4%25+98%2E0%25+98%2E4%25+98%2E3%25+98%2E3%25+97%2E3%25+9) 3. Cutting, Doug, ,lulian Kupiee, Jan Pedersen, and Penelope Sibun. 1992. A practical part-of-speech tagger. In Proceedings of the Third Conference on Applied Natural Language Processing, pages 133-140, Trento, Italy, April. [find](https://scholar.google.com/scholar?q=Cutting%2C+Doug%2C+%2Clulian+Kupiee%2C+Jan+Pedersen%2C+and+Penelope+Sibun%2E+1992%2E+A+practical+part%2Dof%2Dspeech+tagger%2E+In+Proceedings+of+the+Third+Conference+on+Applied+Natural+Language+Processing%2C+pages+133%2D140%2C+) [our card](https://md.research.elamur.ai/free/en/papers/cutting-1992-practical-part-speech-tagger/) 4. Darroch, J. N. and D. Ratcliff. 1972. Generalized Iterative Scaling for Log-Linear Models. The Annals of Mathematical ,';tatistics, 43 (5) : 1470-1480. [find](https://scholar.google.com/scholar?q=Darroch%2C+J%2E+N%2E+and+D%2E+Ratcliff%2E+1972%2E+Generalized+Iterative+Scaling+for+Log%2DLinear+Models%2E+The+Annals+of+Mathematical+%2C%27%3Btatistics%2C+43+%285%29+%3A+1470%2D1480%2E) 5. Liberman, Mark Y. and Kenneth W. Church. 1992. Text analysis and word pronunciation in text-to-speech synthesis. In Sadaoki Furui and M. Mohan Sondi, editors, Advances in ,5'peech Signal Processing. Marcel Dekker, Incorporated, New York. [find](https://scholar.google.com/scholar?q=Liberman%2C+Mark+Y%2E+and+Kenneth+W%2E+Church%2E+1992%2E+Text+analysis+and+word+pronunciation+in+text%2Dto%2Dspeech+synthesis%2E+In+Sadaoki+Furui+and+M%2E+Mohan+Sondi%2C+editors%2C+Advances+in+%2C5%27peech+Signal+Processing%2E+M) 6. Marcus, Mitchell, Beatrice Sa.ntorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, lq(2):313-330. [find](https://scholar.google.com/scholar?q=Marcus%2C+Mitchell%2C+Beatrice+Sa%2Entorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+lq%282%29%3A313%2D330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/) 7. Nunberg, Geoffrey. 1990. The Linguistics of Punctuation. Number 18 in C, SLI Lecture Notes. University of Chicago Press. [find](https://scholar.google.com/scholar?q=Nunberg%2C+Geoffrey%2E+1990%2E+The+Linguistics+of+Punctuation%2E+Number+18+in+C%2C+SLI+Lecture+Notes%2E+University+of+Chicago+Press%2E) 8. Pahner, David D. and Marti A. Hearst. 1994. Adaptive sentence boundary disambiguation. In Pro-ceedmgs of the 199/~ conference on Applied Natural Language Processing (ANLP), Stuttgart, Germany, October. [find](https://scholar.google.com/scholar?q=Pahner%2C+David+D%2E+and+Marti+A%2E+Hearst%2E+1994%2E+Adaptive+sentence+boundary+disambiguation%2E+In+Pro%2Dceedmgs+of+the+199%2F%7E+conference+on+Applied+Natural+Language+Processing+%28ANLP%29%2C+Stuttgart%2C+Germany%2C+October) 9. Palmer, David D. and Marti A. Hearst. To appear. Adaptive multilingual sentence boundary disambiguation. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Palmer%2C+David+D%2E+and+Marti+A%2E+Hearst%2E+To+appear%2E+Adaptive+multilingual+sentence+boundary+disambiguation%2E+Computational+Linguistics%2E) 10. Rutnaparkhi, Adwait. 1996. A maxinmnl entropy moclel for p a r t - o f speech tagging, in Conference on Empirical 'Method.s in Natural Language Processing, pages 133 142, (h~iversit, y of Pennsylvania, May 17-18. [find](https://scholar.google.com/scholar?q=Rutnaparkhi%2C+Adwait%2E+1996%2E+A+maxinmnl+entropy+moclel+for+p+a+r+t+%2D+o+f+speech+tagging%2C+in+Conference+on+Empirical+%27Method%2Es+in+Natural+Language+Processing%2C+pages+133+142%2C+%28h%7Eiversit%2C+y+of+Pennsylvania) 11. Riley, Michael D. 1989. Some applications of tree-based modelling to speech and language. In DARPA ,5'peech and Language Technology Workshop, pages 339-352, Cape Cod, Massachusetts. [find](https://scholar.google.com/scholar?q=Riley%2C+Michael+D%2E+1989%2E+Some+applications+of+tree%2Dbased+modelling+to+speech+and+language%2E+In+DARPA+%2C5%27peech+and+Language+Technology+Workshop%2C+pages+339%2D352%2C+Cape+Cod%2C+Massachusetts%2E) 12. White, Michael. 1995. Presenting punctuation. In Proceedings of the Fifth £'urop+_an Workshop on Natural Language Generation, pages 107-125, Leiden, The Netherlands. [find](https://scholar.google.com/scholar?q=White%2C+Michael%2E+1995%2E+Presenting+punctuation%2E+In+Proceedings+of+the+Fifth+%C2%A3%27urop%2B%5Fan+Workshop+on+Natural+Language+Generation%2C+pages+107%2D125%2C+Leiden%2C+The+Netherlands%2E)