--- title: "Sentence Boundary Detection and the Problem with the U.S. — Gillick 2009" description: "The author builds a fully statistical system. 8 references, cited by 25 papers here." authors: ["Dan Gillick"] year: 2009 venue: "Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers" source: "https://aclanthology.org/N09-2061/" doi: "10.3115/1620853.1620920" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gillick-2009-sentence-boundary-detection-problem-u/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gillick-2009-sentence-boundary-detection-problem-u/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2009](https://md.research.elamur.ai/free/en/papers/2009/) / Sentence Boundary Detection and the Problem with the U.S. # Sentence Boundary Detection and the Problem with the U.S. **Authors:** [Dan Gillick](https://md.research.elamur.ai/free/en/authors/dan-gillick/) · **Year:** 2009 · **Venue:** Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers **Link:** [aclanthology.org/N09-2061](https://aclanthology.org/N09-2061/) · **Licence:** [CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) — summary only [**84** citations](https://openalex.org/works?filter=cites:W2143230354) OpenAlex, 2026-10-14 · [**25** citations in our library](https://md.research.elamur.ai/free/en/papers/gillick-2009-sentence-boundary-detection-problem-u/cited-by/) of the 140,884 papers whose bibliographies we read · [Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers](https://aclanthology.org/volumes/N09-2/) · [licence](https://creativecommons.org/licenses/by-nc-sa/3.0/) - [Publisher's page](https://aclanthology.org/N09-2061/) - [PDF](https://aclanthology.org/N09-2061.pdf) - [DOI 10.3115/1620853.1620920](https://doi.org/10.3115/1620853.1620920) - [OpenAlex](https://openalex.org/W2143230354) **Authors (1)** 1. [Gillick, Dan](https://openalex.org/A5110207594) [Semantic Scholar](https://www.semanticscholar.org/author/2396669) · h 16 · 1,352 citations · 26 works Citations and author metrics: OpenAlex, 2026-10-14; authors with the same name are not always told apart. ## What it is about Sentence boundary detection is hard because the period is ambiguous, and most practical tools rely on hand-made abbreviation lists. The author builds a fully statistical system. For each period, the features look only at one token on each side of it, and a linear support vector machine classifies the boundary. On a standard news corpus of about 27,000 examples the system makes 67 errors, which is an error rate of about 0.25%. Of these, 23 involve the word "U.S.". The single feature of the token that ends with the period gives surprisingly good results on its own, and a simple Naive Bayes classifier is also good, so the choice of classifier matters less than the features. ## Ideas - I2. Test the paper's ablation first: keep only the feature of the token ending with the period, then add the next-token feature. Count the errors each step removes on EWT test, to see which feature carries the weight in English. - I3. Store the abbreviation decisions as a table of learned weights, not a hand list, and print for each decision the feature that drove it, so every boundary can be explained. ## For Mova - I1. `en/src/tok.rs`: a period classifier over the token before the period and the token after it, with binary features and a linear score written in Rust. Measure the error count on EWT test per candidate period, and report the errors on "U.S.", "Mr." and similar tokens separately. The code is `en/src/tok.rs`, and the metric is `en tok-eval`. The chapter is [languages/en/07-punctuation-capitals-numbers.md](https://mova.elamur.ai/free/en/docs/languages/en/07-punctuation-capitals-numbers/). The paper's corpus is news text, so the 0.25% figure is not a Mova figure. The paper's data is not in Mova's build. Measure first on EWT test: errors per candidate period, with a two-fold check (`en folds`). ## Institutions [🎓**University of California, Berkeley** Berkeley · United States 107 works · 205,102 citations](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) Topics[Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) ## Cite Cite as Dan Gillick. 2009. Sentence Boundary Detection and the Problem with the U.S. Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers. The paper itself, not this card. BibTeX from the library's record of the work. [DOI 10.3115/1620853.1620920](https://doi.org/10.3115/1620853.1620920) ```bibtex @inproceedings{gillick2009sentence, title = {{Sentence Boundary Detection and the Problem with the U.S.}}, author = {Dan Gillick}, booktitle = {Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers}, year = {2009}, doi = {10.3115/1620853.1620920}, url = {https://doi.org/10.3115/1620853.1620920} } ``` - [← previous Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk](https://md.research.elamur.ai/free/en/papers/burch-2009-fast-cheap-creative-evaluating-translation/) - [next → SemEval-2010 Task 1: Coreference Resolution in Multiple Languages](https://md.research.elamur.ai/free/en/papers/recasens-2009-semeval-2010-task-1-coreference/) - [↑ back to papers of 2009](https://md.research.elamur.ai/free/en/papers/2009/) ## References 8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. J. Aberdeen, J. Burger, D. Day, L. Hirschman, P. Robinson, and M. Vilain. 1995. MITRE: description of the Alembic system used for MUC-6. In Proceedings of the 6th conference on Message understanding, pages 141–155. Association for Computational Linguistics Morristown, NJ, USA. [find](https://scholar.google.com/scholar?q=J%2E+Aberdeen%2C+J%2E+Burger%2C+D%2E+Day%2C+L%2E+Hirschman%2C+P%2E+Robinson%2C+and+M%2E+Vilain%2E+1995%2E+MITRE%3A+description+of+the+Alembic+system+used+for+MUC%2D6%2E+In+Proceedings+of+the+6th+conference+on+Message+understanding%2C+) 2. T. Joachims. 1999. Making large-scale support vector machine learning practical, Advances in kernel methods: support vector learning. [find](https://scholar.google.com/scholar?q=T%2E+Joachims%2E+1999%2E+Making+large%2Dscale+support+vector+machine+learning+practical%2C+Advances+in+kernel+methods%3A+support+vector+learning%2E) 3. T. Kiss and J. Strunk. 2006. Unsupervised Multilingual Sentence Boundary Detection. Computational Linguistics, 32(4):485–525. [our card](https://md.research.elamur.ai/free/en/papers/kiss-2006-unsupervised-multilingual-sentence-boundary-detection/) 4. E. Loper and S. Bird. 2002. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 62–69. [find](https://scholar.google.com/scholar?q=E%2E+Loper+and+S%2E+Bird%2E+2002%2E+NLTK%3A+The+Natural+Language+Toolkit%2E+In+Proceedings+of+the+ACL+Workshop+on+Effective+Tools+and+Methodologies+for+Teaching+Natural+Language+Processing+and+Computational+Lingu) 5. M.P. Marcus, M.A. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: the penn treebank. Computational Linguistics, 19(2):313–330. [our card](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/) 6. A. Mikheev. 2002. Periods, Capitalized Words, etc. Computational Linguistics, 28(3):289–318. [find](https://scholar.google.com/scholar?q=A%2E+Mikheev%2E+2002%2E+Periods%2C+Capitalized+Words%2C+etc%2E+Computational+Linguistics%2C+28%283%29%3A289%E2%80%93318%2E) 7. D.D. Palmer and M.A. Hearst. 1997. Adaptive Multilingual Sentence Boundary Disambiguation. Computational Linguistics, 23(2):241–267. [find](https://scholar.google.com/scholar?q=D%2ED%2E+Palmer+and+M%2EA%2E+Hearst%2E+1997%2E+Adaptive+Multilingual+Sentence+Boundary+Disambiguation%2E+Computational+Linguistics%2C+23%282%29%3A241%E2%80%93267%2E) 8. J.C. Reynar and A. Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, pages 16–19. [our card](https://md.research.elamur.ai/free/en/papers/reynar-1997-maximum-entropy-approach-identifying-sentence/)