---
title: "A Statistical Model for Domain-Independent Text Segmentation"
description: "This matters because it offers a principled, training-free segmenter that works in any domain and can… 22 references, cited by 74 papers here."
authors: ["Masao Utiyama", "Hitoshi Isahara"]
year: 2001
venue: "Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P01-1064/"
doi: "10.3115/1073012.1073076"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/utiyama-2001-statistical-model-domain-independent-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/utiyama-2001-statistical-model-domain-independent-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2001](https://md.research.elamur.ai/free/en/papers/2001/) / A Statistical Model for Domain-Independent Text Segmentation

# A Statistical Model for Domain-Independent Text Segmentation

**Authors:** [Masao Utiyama](https://md.research.elamur.ai/free/en/authors/masao-utiyama/) and [Hitoshi Isahara](https://md.research.elamur.ai/free/en/authors/hitoshi-isahara/) · **Year:** 2001 · **Venue:** [Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2001](https://md.research.elamur.ai/free/en/venues/acl/2001/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P01-1064/) · [PDF](https://aclanthology.org/P01-1064.pdf) · [DOI 10.3115/1073012.1073076](https://doi.org/10.3115/1073012.1073076) · [OpenAlex](https://openalex.org/W2148818577)

**Citations:** 287 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/utiyama-2001-statistical-model-domain-independent-text/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Utiyama and Isahara address text segmentation, the task of splitting a document into topically coherent parts. They target settings with no labelled data, such as summarizing documents or speeches. Earlier unsupervised methods relied on lexical cohesion heuristics like cosine similarity or word repetition. This paper instead treats segmentation as a probabilistic search problem. Each segment is modelled as its own word distribution, estimated only from the text being segmented and smoothed with Laplace's law. A prior over segmentations comes from a minimum-description-length argument: the cost of transmitting the segment lengths. Together these give a cost for every candidate segment. Finding the best segmentation then reduces to a minimum-cost path through a graph of inter-word positions, solved with dynamic programming. The method also picks the number of segments automatically. On Choi's artificial benchmark, the system matched or beat Choi's (2000) state-of-the-art segmenter, with statistically significant gains. This matters because it offers a principled, training-free segmenter that works in any domain and can later absorb supervised features.

## Key points
- The model assumes topics are statistically independent and words are independent given their topic. Word probabilities within a segment use Laplace's law, so no external training corpus is needed.
- The prior is uninformative, based on description length. Each segment length costs log n bits for a text of n words, giving k·log n bits for k segments. This term counterbalances the likelihood, which on its own would over-segment.
- Segments correspond to edges in a graph of word-gap positions. The best segmentation is a minimum-cost path found by DP, with boundaries restricted here to sentence ends. A fixed number of segments can be enforced by limiting the number of edges in the path.
- The number of segments grows only slowly with text length. A ~27-sentence editorial yields roughly 4–6 segments, while a text of over 1000 sentences yields about 10–20.
- On 700 samples from Choi's artificial corpus, with the segment count given beforehand, the overall error was about 9% versus 11% for Choi's system (one-sided t-test, p ≈ 6.8E-9). The proposed system was better on every data subset reported. The implementation was publicly released by CRL.

## Interesting ideas
- **Segmentation as shortest path over word gaps.** If segment scores are additive, any segmentation-style problem becomes a DP over a position graph. This could apply to sentence chunking, dialogue turn grouping, or splitting long Ukrainian documents for retrieval.
- **MDL as a free prior.** Charging bits for encoding each segment length provides a parameter-free penalty against too many segments. This trick is reusable wherever a model needs an automatic stopping criterion without tuning.
- **Estimating from the input itself.** Fitting smoothed per-segment word distributions on the document alone makes the method language- and domain-agnostic. It is attractive for low-resource languages or niche domains like trade documents, where annotated data is scarce.
- **Pluggable segment-level priors.** Because whole segments are scored directly, features of a segment can be added as extra cost terms. The authors show one example: a normal-distribution prior on segment length. This is easier than in HMM topic-tagging approaches, which only model transitions.
- **By-product word densities.** The per-segment word probabilities could help locate passages where a term is discussed in depth. This might support glossary extraction or term-definition mining.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Masao Utiyama and Hitoshi Isahara. 2001. A Statistical Model for Domain-Independent Text Segmentation. Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{utiyama2001statistical,
  title = {{A Statistical Model for Domain-Independent Text Segmentation}},
  author = {Masao Utiyama and Hitoshi Isahara},
  booktitle = {Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073012.1073076},
  url = {https://doi.org/10.3115/1073012.1073076}
}
```

- [← previous Immediate-Head Parsing for Language Models](https://md.research.elamur.ai/free/en/papers/charniak-2001-immediate-head-parsing-language-models/)
- [next → Fast Decoding and Optimal Decoding for Machine Translation](https://md.research.elamur.ai/free/en/papers/germann-2001-fast-decoding-optimal-decoding-machine/)
- [↑ back to papers of 2001](https://md.research.elamur.ai/free/en/papers/2001/)

## References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. James Allan, Jaime Carbonell, George Doddington, Jonathan Yamron, and Yiming Yang. 1998. Topic detection and tracking pilot study final report. In Proc. of the DARPA Broadcast News Transcription and Understanding Workshop. [find](https://scholar.google.com/scholar?q=James+Allan%2C+Jaime+Carbonell%2C+George+Doddington%2C+Jonathan+Yamron%2C+and+Yiming+Yang%2E+1998%2E+Topic+detection+and+tracking+pilot+study+final+report%2E+In+Proc%2E+of+the+DARPA+Broadcast+News+Transcription+and+U)
2. Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Machine Learning, 34(1-3):177–210. [find](https://scholar.google.com/scholar?q=Doug+Beeferman%2C+Adam+Berger%2C+and+John+Lafferty%2E+1999%2E+Statistical+models+for+text+segmentation%2E+Machine+Learning%2C+34%281%2D3%29%3A177%E2%80%93210%2E)
3. Freddy Y. Y. Choi. 2000. [Advances in domain independent linear text segmentation](https://md.research.elamur.ai/free/en/papers/choi-2000-advances-domain-independent-linear-text/). In Proc. of NAACL-2000.
4. Marti A. Hearst and Christian Plaunt. 1993. Subtopic structuring for full-length document access. In Proc. of the Sixteenth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 59–68. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst+and+Christian+Plaunt%2E+1993%2E+Subtopic+structuring+for+full%2Dlength+document+access%2E+In+Proc%2E+of+the+Sixteenth+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Inf)
5. Marti A. Hearst. 1994. Multi-paragraph segmentation of expository text. In Proc. of ACL’94. [find](https://scholar.google.com/scholar?q=Marti+A%2E+Hearst%2E+1994%2E+Multi%2Dparagraph+segmentation+of+expository+text%2E+In+Proc%2E+of+ACL%E2%80%9994%2E)
6. Oskari Heinonen. 1998. Optimal multi-paragraph text segmentation by dynamic programming. In Proc. of COLING-ACL’98. [find](https://scholar.google.com/scholar?q=Oskari+Heinonen%2E+1998%2E+Optimal+multi%2Dparagraph+text+segmentation+by+dynamic+programming%2E+In+Proc%2E+of+COLING%2DACL%E2%80%9998%2E)
7. Min-Yen Kan, Judith L. Klavans, and Kathleen R. McKeown. 1998. Linear segmentation and segment significance. In Proc. of WVLC-6, pages 197– 205. [find](https://scholar.google.com/scholar?q=Min%2DYen+Kan%2C+Judith+L%2E+Klavans%2C+and+Kathleen+R%2E+McKeown%2E+1998%2E+Linear+segmentation+and+segment+significance%2E+In+Proc%2E+of+WVLC%2D6%2C+pages+197%E2%80%93+205%2E)
8. Judith L. Klavans, Kathleen R. McKeown, Min-Yen Kan, and Susan Lee. 1998. Resources for the evaluation of summarization techniques. In Proceedings of the 1st International Conference on Language Resources and Evaluation (LREC), pages 899–902. [find](https://scholar.google.com/scholar?q=Judith+L%2E+Klavans%2C+Kathleen+R%2E+McKeown%2C+Min%2DYen+Kan%2C+and+Susan+Lee%2E+1998%2E+Resources+for+the+evaluation+of+summarization+techniques%2E+In+Proceedings+of+the+1st+International+Conference+on+Language+Resou)
9. Hideki Kozima. 1993. Text segmentation based on similarity between words. In Proc. of ACL’93. [find](https://scholar.google.com/scholar?q=Hideki+Kozima%2E+1993%2E+Text+segmentation+based+on+similarity+between+words%2E+In+Proc%2E+of+ACL%E2%80%9993%2E)
10. Sadao Kurohashi, Nobuyuki Shiraki, and Makoto Nagao. 1997. A method for detecting important descriptions of a word based on its density distribution in text (in Japanese). IPSJ (Information Processing Society of Japan) Journal, 38(4):845–854. [find](https://scholar.google.com/scholar?q=Sadao+Kurohashi%2C+Nobuyuki+Shiraki%2C+and+Makoto+Nagao%2E+1997%2E+A+method+for+detecting+important+descriptions+of+a+word+based+on+its+density+distribution+in+text+%28in+Japanese%29%2E+IPSJ+%28Information+Processing)
11. Kikuo Maekawa and Hanae Koiso. 2000. Design of spontaneous speech corpus for Japanese. In Proc of International Symposium: Toward the Realization of Spontaneous Speech Engineering, pages 70–77. [find](https://scholar.google.com/scholar?q=Kikuo+Maekawa+and+Hanae+Koiso%2E+2000%2E+Design+of+spontaneous+speech+corpus+for+Japanese%2E+In+Proc+of+International+Symposium%3A+Toward+the+Realization+of+Spontaneous+Speech+Engineering%2C+pages+70%E2%80%9377%2E)
12. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. The MIT Press. [find](https://scholar.google.com/scholar?q=Christopher+D%2E+Manning+and+Hinrich+Schu%CC%88tze%2E+1999%2E+Foundations+of+Statistical+Natural+Language+Processing%2E+The+MIT+Press%2E)
13. Masaaki Nagata. 1994. A stochastic Japanese morphological analyzer using a forward-DP backward-A n-best search algorithm. In Proc. of COL-ING’94, pages 201–207. ì Yoshio Nakao. 2000. An algorithm for one-page summarization of a long text based on thematic hierarchy detection. In Proc. of ACL’2000, p… [find](https://scholar.google.com/scholar?q=Masaaki+Nagata%2E+1994%2E+A+stochastic+Japanese+morphological+analyzer+using+a+forward%2DDP+backward%2DA+n%2Dbest+search+algorithm%2E+In+Proc%2E+of+COL%2DING%E2%80%9994%2C+pages+201%E2%80%93207%2E+%C3%AC+Yoshio+Nakao%2E+2000%2E+An+algorithm+for+)
14. Manabu Okumura and Takeo Honda. 1994. Word sense disambiguation and text segmentation based on lexical cohesion. In Proc. of COLING-94. [find](https://scholar.google.com/scholar?q=Manabu+Okumura+and+Takeo+Honda%2E+1994%2E+Word+sense+disambiguation+and+text+segmentation+based+on+lexical+cohesion%2E+In+Proc%2E+of+COLING%2D94%2E)
15. Jay M. Ponte and W. Bruce Croft. 1997. Text segmentation by topic. In Proc. of the First European Conference on Research and Advanced Technology for Digital Libraries, pages 120–129. [find](https://scholar.google.com/scholar?q=Jay+M%2E+Ponte+and+W%2E+Bruce+Croft%2E+1997%2E+Text+segmentation+by+topic%2E+In+Proc%2E+of+the+First+European+Conference+on+Research+and+Advanced+Technology+for+Digital+Libraries%2C+pages+120%E2%80%93129%2E)
16. Jeffrey C. Reynar. 1994. An automatic method of finding topic boundaries. In Proc. of ACL-94. [find](https://scholar.google.com/scholar?q=Jeffrey+C%2E+Reynar%2E+1994%2E+An+automatic+method+of+finding+topic+boundaries%2E+In+Proc%2E+of+ACL%2D94%2E)
17. Jeffrey C. Reynar. 1998. Topic segmentation: Algorithms and applications. Ph.D. thesis, Computer and Information Science, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=Jeffrey+C%2E+Reynar%2E+1998%2E+Topic+segmentation%3A+Algorithms+and+applications%2E+Ph%2ED%2E+thesis%2C+Computer+and+Information+Science%2C+University+of+Pennsylvania%2E)
18. Jeffrey C. Reynar. 1999. Statistical models for topic segmentation. In Proc. of ACL-99, pages 357–364. [find](https://scholar.google.com/scholar?q=Jeffrey+C%2E+Reynar%2E+1999%2E+Statistical+models+for+topic+segmentation%2E+In+Proc%2E+of+ACL%2D99%2C+pages+357%E2%80%93364%2E)
19. Gerard Salton, Amit Singhal, Chris Buckley, and Mandar Mitra. 1996. Automatic text decomposition using text segments and text themes. In Proc. of Hypertext’96. [find](https://scholar.google.com/scholar?q=Gerard+Salton%2C+Amit+Singhal%2C+Chris+Buckley%2C+and+Mandar+Mitra%2E+1996%2E+Automatic+text+decomposition+using+text+segments+and+text+themes%2E+In+Proc%2E+of+Hypertext%E2%80%9996%2E)
20. Andreas Stolcke and Stephen M. Omohundro. 1994. Best-first model merging for hidden Markov model induction. Technical Report TR-94-003, ICSI, Berkeley, CA. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke+and+Stephen+M%2E+Omohundro%2E+1994%2E+Best%2Dfirst+model+merging+for+hidden+Markov+model+induction%2E+Technical+Report+TR%2D94%2D003%2C+ICSI%2C+Berkeley%2C+CA%2E)
21. Yaakov Yaari. 1997. Segmentation of expository texts by hierarchical agglomerative clustering. In Proc. of the Recent Advances in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Yaakov+Yaari%2E+1997%2E+Segmentation+of+expository+texts+by+hierarchical+agglomerative+clustering%2E+In+Proc%2E+of+the+Recent+Advances+in+Natural+Language+Processing%2E)
22. J. P. Yamron, I. Carp, S. Lowe, and P. van Mulbregt. 1998. A hidden Markov model approach to text segmentation and event tracking. In Proc. of ICASSP-98. [find](https://scholar.google.com/scholar?q=J%2E+P%2E+Yamron%2C+I%2E+Carp%2C+S%2E+Lowe%2C+and+P%2E+van+Mulbregt%2E+1998%2E+A+hidden+Markov+model+approach+to+text+segmentation+and+event+tracking%2E+In+Proc%2E+of+ICASSP%2D98%2E)

*Updated 24 October 2026*