---
title: "Catching the Drift — Barzilay 2004"
description: "The paper tackles the problem of modeling how texts in a narrow domain are organized in terms of content: which topics they cover and in what order."
authors: ["Regina Barzilay", "Lillian Lee"]
year: 2004
venue: "Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004"
source: "https://aclanthology.org/N04-1015/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

# Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization

**Authors:** [Regina Barzilay](https://md.research.elamur.ai/free/en/authors/regina-barzilay/), [Lillian Lee](https://md.research.elamur.ai/free/en/authors/lillian-lee/) · **Year:** 2004 · **Venue:** Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N04-1015/) · [PDF](https://aclanthology.org/N04-1015.pdf) · [OpenAlex](https://openalex.org/W2118370253)

**Citations:** 295 citations (openalex-snapshot · 2026-10-14) · [116](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper tackles the problem of modeling how texts in a narrow domain are organized in terms of content: which topics they cover and in what order. Instead of hand-built schemas or knowledge bases, Barzilay and Lee learn this structure from unannotated documents. They model it as a Hidden Markov Model, called a content model. Each state stands for a domain topic and emits sentences through its own bigram language model. Transitions between states capture typical topic orderings.

Training starts by clustering sentences by lexical similarity and then refines the clusters with Viterbi-style re-estimation, so that sentence position also shapes topic assignments. The model is tested on two tasks:
- **Information ordering:** it beats Lapata's (2003) method by a wide margin.
- **Extractive summarization:** on the Earthquakes data, its sentence selection agrees with human summaries 88% of the time.

The work matters because it gives empirical support to the old idea that word distribution patterns reflect discourse structure, at least within one domain. It also yields a cheap, reusable representation that transfers across generation and summarization.

## Key points
- Content models are HMMs whose states are topics with state-specific bigram language models. A plain n-gram model is a degenerate, single-state case.
- Initial topics come from complete-link clustering of sentences, using cosine similarity over word-bigram features. During clustering, generic tokens stand in for proper names, numbers and dates, so that clusters group event types rather than specific events.
- On information ordering, content models outperform Lapata's (2003) approach, with a gap of 78 percentage points for one domain and metric.
- On summarization (Earthquakes), extraction accuracy is 88%, compared with 76% for a sentence classifier using words plus location and 69% for the leading-sentences baseline. Trained on only one third of the data, the content-model method still beats the classifier trained on the full set.
- When the number of states is varied on Earthquakes, the same size (64 states) gives the best result on both tasks: 72% OSO prediction rate for ordering and 88% extraction accuracy for summarization.

## Interesting ideas
- **Catch-all insertion state.** Small clusters are merged into an "etcetera" cluster. Its state gets a language model that is the complement of all the other states, so it absorbs digressions and unseen content. Any unsupervised segmenter or tagger that needs a principled "other" class could reuse this trick.
- **Transitions from document-level co-occurrence.** Transition probabilities are estimated from how many documents contain one cluster's sentence directly followed by another's, rather than from raw counts. This keeps one long article from dominating the ordering statistics.
- **Order-aware re-clustering.** Sentences are reassigned to clusters using the Viterbi path, which lets position separate lexically similar sentences. For example, casualty mentions at the end of a quake report often describe an earlier quake. This idea suits any domain where similar wording means different things depending on where it appears.
- **One model, several tasks.** The authors check that the same model configuration ranks similarly on ordering and on summarization. This suggests tuning a structural model on a task with cheap evaluation and then reusing it where labeled data is scarce.
- **Masking entities before clustering.** Replacing names, numbers and dates with placeholders pushes clusters toward abstract information types. This is a simple preprocessing step for topic induction on news-like corpora, including Ukrainian ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

116 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 104 not yet.

- [A Simple Domain-Independent Probabilistic Approach to Generation](https://md.research.elamur.ai/free/en/papers/angeli-2010-simple-domain-independent-probabilistic-approach/)
- [Modeling Local Coherence: An Entity-Based Approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/)
- [Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints](https://md.research.elamur.ai/free/en/papers/durrett-2016-learning-based-single-document-summarization/)
- [A Unified Local and Global Model for Discourse Coherence](https://md.research.elamur.ai/free/en/papers/elsner-2007-unified-local-global-model-discourse/)
- [Coreference-inspired Coherence Modeling](https://md.research.elamur.ai/free/en/papers/elsner-2008-coreference-inspired-coherence-modeling/)
- [Global Features for Shallow Discourse Parsing](https://md.research.elamur.ai/free/en/papers/ghosh-2012-global-features-shallow-discourse-parsing/)
- [Exploring Content Models for Multi-Document Summarization](https://md.research.elamur.ai/free/en/papers/haghighi-2009-exploring-content-models-multi-document/)
- [Neural Net Models of Open-domain Discourse Coherence](https://md.research.elamur.ai/free/en/papers/li-2017-neural-net-models-open-domain/)
- [Automatically Evaluating Text Coherence Using Discourse Relations](https://md.research.elamur.ai/free/en/papers/lin-2011-automatically-evaluating-text-coherence-discourse/)
- [What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment](https://md.research.elamur.ai/free/en/papers/mei-2016-what-talk-about-how-selective/)
- [Unsupervised Modeling of Twitter Conversations](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{barzilay2004catching,
  title = {{Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization}},
  author = {Barzilay, Regina and Lee, Lillian},
  booktitle = {Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004},
  year = {2004},
  url = {https://aclanthology.org/N04-1015/}
}
```

- [← previous BootCaT: Bootstrapping Corpora and Terms from the Web](https://md.research.elamur.ai/free/en/papers/baroni-2004-bootcat-bootstrapping-corpora-terms-web/)
- [next → Chapter 3: Minimalist assumptions on negation in Biblical Hebrew syntax](https://md.research.elamur.ai/free/en/papers/hbo-2004-snyman-chapter-3-minimalist-assumptions-negation/)
- [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/)

## References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. F. C. Bartlett. 1932. Remembering: a study in experimental and social psychology. Cambridge University Press. [find](https://scholar.google.com/scholar?q=F%2E+C%2E+Bartlett%2E+1932%2E+Remembering%3A+a+study+in+experimental+and+social+psychology%2E+Cambridge+University+Press%2E)
2. R. Barzilay, L. Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. In HLT-NAACL 2003: Main Proceedings, 16– 23. [find](https://scholar.google.com/scholar?q=R%2E+Barzilay%2C+L%2E+Lee%2E+2003%2E+Learning+to+paraphrase%3A+An+unsupervised+approach+using+multiple%2Dsequence+alignment%2E+In+HLT%2DNAACL+2003%3A+Main+Proceedings%2C+16%E2%80%93+23%2E) [our card](https://md.research.elamur.ai/free/en/papers/barzilay-2003-learning-paraphrase-unsupervised-approach-multiple/)
3. D. Beeferman, A. Berger, J. Lafferty. 1997. Text segmentation using exponential models. In Proceedings of EMNLP, 35–46. [find](https://scholar.google.com/scholar?q=D%2E+Beeferman%2C+A%2E+Berger%2C+J%2E+Lafferty%2E+1997%2E+Text+segmentation+using+exponential+models%2E+In+Proceedings+of+EMNLP%2C+35%E2%80%9346%2E)
4. S. F. Chen, K. Seymore, R. Rosenfeld. 1998. Topic adaptation for language modeling using unnormalized exponential models. In Proceedings of ICASSP, volume 2, 681–684. [find](https://scholar.google.com/scholar?q=S%2E+F%2E+Chen%2C+K%2E+Seymore%2C+R%2E+Rosenfeld%2E+1998%2E+Topic+adaptation+for+language+modeling+using+unnormalized+exponential+models%2E+In+Proceedings+of+ICASSP%2C+volume+2%2C+681%E2%80%93684%2E)
5. G. DeJong. 1982. An overview of the FRUMP system. In W. G. Lehnert, M. H. Ringle, eds., Strategies for Natural Language Processing, 149–176. Lawrence Erlbaum Associates, Hillsdale, New Jersey. [find](https://scholar.google.com/scholar?q=G%2E+DeJong%2E+1982%2E+An+overview+of+the+FRUMP+system%2E+In+W%2E+G%2E+Lehnert%2C+M%2E+H%2E+Ringle%2C+eds%2E%2C+Strategies+for+Natural+Language+Processing%2C+149%E2%80%93176%2E+Lawrence+Erlbaum+Associates%2C+Hillsdale%2C+New+Jersey%2E)
6. P. A. Duboue, K. R. McKeown. 2003. Statistical acquisition of content selection rules for natural language generation. In Proceedings of EMNLP, 121–128. [find](https://scholar.google.com/scholar?q=P%2E+A%2E+Duboue%2C+K%2E+R%2E+McKeown%2E+2003%2E+Statistical+acquisition+of+content+selection+rules+for+natural+language+generation%2E+In+Proceedings+of+EMNLP%2C+121%E2%80%93128%2E)
7. S. Fine, Y. Singer, N. Tishby. 1998. The hierarchical hidden Markov model: Analysis and applications. Machine Learning, 32(1):41–62. [find](https://scholar.google.com/scholar?q=S%2E+Fine%2C+Y%2E+Singer%2C+N%2E+Tishby%2E+1998%2E+The+hierarchical+hidden+Markov+model%3A+Analysis+and+applications%2E+Machine+Learning%2C+32%281%29%3A41%E2%80%9362%2E)
8. R. Florian, D. Yarowsky. 1999. Dynamic non-local language modeling via hierarchical topic-based adaptation. In Proceedings of the ACL, 167–174. [find](https://scholar.google.com/scholar?q=R%2E+Florian%2C+D%2E+Yarowsky%2E+1999%2E+Dynamic+non%2Dlocal+language+modeling+via+hierarchical+topic%2Dbased+adaptation%2E+In+Proceedings+of+the+ACL%2C+167%E2%80%93174%2E)
9. D. Gildea, T. Hofmann. 1999. Topic-based language models using EM. In Proceedings of EUROSPEECH, 2167–2170. [find](https://scholar.google.com/scholar?q=D%2E+Gildea%2C+T%2E+Hofmann%2E+1999%2E+Topic%2Dbased+language+models+using+EM%2E+In+Proceedings+of+EUROSPEECH%2C+2167%E2%80%932170%2E)
10. Z. Harris. 1982. Discourse and sublanguage. In R. Kittredge, J. Lehrberger, eds., Sublanguage: Studies of Language in Restricted Semantic Domains, 231–236. Walter de Gruyter, Berlin; New York. [find](https://scholar.google.com/scholar?q=Z%2E+Harris%2E+1982%2E+Discourse+and+sublanguage%2E+In+R%2E+Kittredge%2C+J%2E+Lehrberger%2C+eds%2E%2C+Sublanguage%3A+Studies+of+Language+in+Restricted+Semantic+Domains%2C+231%E2%80%93236%2E+Walter+de+Gruyter%2C+Berlin%3B+New+York%2E)
11. M. Hearst. 1994. Multi-paragraph segmentation of expository text. In Proceedings of the ACL, 9–16. [find](https://scholar.google.com/scholar?q=M%2E+Hearst%2E+1994%2E+Multi%2Dparagraph+segmentation+of+expository+text%2E+In+Proceedings+of+the+ACL%2C+9%E2%80%9316%2E)
12. R. Iyer, M. Ostendorf. 1996. Modeling long distance dependence in language: Topic mixtures vs. dynamic cache models. In Proceedings of ICSLP, 236–239. [find](https://scholar.google.com/scholar?q=R%2E+Iyer%2C+M%2E+Ostendorf%2E+1996%2E+Modeling+long+distance+dependence+in+language%3A+Topic+mixtures+vs%2E+dynamic+cache+models%2E+In+Proceedings+of+ICSLP%2C+236%E2%80%93239%2E)
13. D. R. Jones, C. A. Thompson. 2003. Identifying events using similarity and context. In Proceedings of CoNLL, 135–141. [find](https://scholar.google.com/scholar?q=D%2E+R%2E+Jones%2C+C%2E+A%2E+Thompson%2E+2003%2E+Identifying+events+using+similarity+and+context%2E+In+Proceedings+of+CoNLL%2C+135%E2%80%93141%2E)
14. R. Kittredge, T. Korelsky, O. Rambow. 1991. On the need for domain communication language. Computational Intelligence, 7(4):305–314. [find](https://scholar.google.com/scholar?q=R%2E+Kittredge%2C+T%2E+Korelsky%2C+O%2E+Rambow%2E+1991%2E+On+the+need+for+domain+communication+language%2E+Computational+Intelligence%2C+7%284%29%3A305%E2%80%93314%2E)
15. J. Kupiec, J. Pedersen, F. Chen. 1999. A trainable document summarizer. In I. Mani, M. T. Maybury, eds., Advances in Automatic Summarization, 55–60. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=J%2E+Kupiec%2C+J%2E+Pedersen%2C+F%2E+Chen%2E+1999%2E+A+trainable+document+summarizer%2E+In+I%2E+Mani%2C+M%2E+T%2E+Maybury%2C+eds%2E%2C+Advances+in+Automatic+Summarization%2C+55%E2%80%9360%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
16. M. Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In Proceeding of the ACL, 545–552. [find](https://scholar.google.com/scholar?q=M%2E+Lapata%2E+2003%2E+Probabilistic+text+structuring%3A+Experiments+with+sentence+ordering%2E+In+Proceeding+of+the+ACL%2C+545%E2%80%93552%2E) [our card](https://md.research.elamur.ai/free/en/papers/lapata-2003-probabilistic-text-structuring-experiments-sentence/)
17. W. C. Mann, S. A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. TEXT, 8(3):243–281. [find](https://scholar.google.com/scholar?q=W%2E+C%2E+Mann%2C+S%2E+A%2E+Thompson%2E+1988%2E+Rhetorical+structure+theory%3A+Toward+a+functional+theory+of+text+organization%2E+TEXT%2C+8%283%29%3A243%E2%80%93281%2E) [our card](https://md.research.elamur.ai/free/en/papers/enud-1988-mann-rhetorical-structure-theory-functional/)
18. D. Marcu. 1997. The rhetorical parsing of natural language texts. In Proceedings of the ACL/EACL, 96– 103. [find](https://scholar.google.com/scholar?q=D%2E+Marcu%2E+1997%2E+The+rhetorical+parsing+of+natural+language+texts%2E+In+Proceedings+of+the+ACL%2FEACL%2C+96%E2%80%93+103%2E)
19. K. R. McKeown. 1985. Text Generation: Using Discourse Strategies and Focus Constraints to Generate Natural Language Text. Cambridge University Press, Cambridge, UK. [find](https://scholar.google.com/scholar?q=K%2E+R%2E+McKeown%2E+1985%2E+Text+Generation%3A+Using+Discourse+Strategies+and+Focus+Constraints+to+Generate+Natural+Language+Text%2E+Cambridge+University+Press%2C+Cambridge%2C+UK%2E)
20. W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery. 1997. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, second edition. [find](https://scholar.google.com/scholar?q=W%2E+H%2E+Press%2C+S%2E+A%2E+Teukolsky%2C+W%2E+T%2E+Vetterling%2C+B%2E+P%2E+Flannery%2E+1997%2E+Numerical+Recipes+in+C%3A+The+Art+of+Scientific+Computing%2E+Cambridge+University+Press%2C+second+edition%2E)
21. L. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257–286. [find](https://scholar.google.com/scholar?q=L%2E+Rabiner%2E+1989%2E+A+tutorial+on+hidden+Markov+models+and+selected+applications+in+speech+recognition%2E+Proceedings+of+the+IEEE%2C+77%282%29%3A257%E2%80%93286%2E)
22. D. R. Radev, K. R. McKeown. 1998. Generating natural language summaries from multiple on-line sources. Computational Linguistics, 24(3):469–500. [find](https://scholar.google.com/scholar?q=D%2E+R%2E+Radev%2C+K%2E+R%2E+McKeown%2E+1998%2E+Generating+natural+language+summaries+from+multiple+on%2Dline+sources%2E+Computational+Linguistics%2C+24%283%29%3A469%E2%80%93500%2E)
23. O. Rambow. 1990. Domain communication knowledge. In Fifth International Workshop on Natural Language Generation, 87–94. [find](https://scholar.google.com/scholar?q=O%2E+Rambow%2E+1990%2E+Domain+communication+knowledge%2E+In+Fifth+International+Workshop+on+Natural+Language+Generation%2C+87%E2%80%9394%2E)
24. J. Reynar, A. Ratnaparkhi. 1997. A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing, 16–19. [find](https://scholar.google.com/scholar?q=J%2E+Reynar%2C+A%2E+Ratnaparkhi%2E+1997%2E+A+maximum+entropy+approach+to+identifying+sentence+boundaries%2E+In+Proceedings+of+the+Fifth+Conference+on+Applied+Natural+Language+Processing%2C+16%E2%80%9319%2E) [our card](https://md.research.elamur.ai/free/en/papers/reynar-1997-maximum-entropy-approach-identifying-sentence/)
25. R. E. Schapire, Y. Singer. 2000. BoosTexter: A boosting-based system for text categorization. Machine Learning, 2/3:135–168. [find](https://scholar.google.com/scholar?q=R%2E+E%2E+Schapire%2C+Y%2E+Singer%2E+2000%2E+BoosTexter%3A+A+boosting%2Dbased+system+for+text+categorization%2E+Machine+Learning%2C+2%2F3%3A135%E2%80%93168%2E)
26. R. Soricut, D. Marcu. 2003. Sentence level discourse parsing using syntactic and lexical information. In Proceedings of the HLT/NAACL, 228–235. [find](https://scholar.google.com/scholar?q=R%2E+Soricut%2C+D%2E+Marcu%2E+2003%2E+Sentence+level+discourse+parsing+using+syntactic+and+lexical+information%2E+In+Proceedings+of+the+HLT%2FNAACL%2C+228%E2%80%93235%2E) [our card](https://md.research.elamur.ai/free/en/papers/soricut-2003-sentence-level-discourse-parsing-syntactic/)
27. M. White, T. Korelsky, C. Cardie, V. Ng, D. Pierce, K. Wagstaff. 2001. Multi-document summarization via information extraction. In Proceedings of the HLT Conference, 263–269. [find](https://scholar.google.com/scholar?q=M%2E+White%2C+T%2E+Korelsky%2C+C%2E+Cardie%2C+V%2E+Ng%2C+D%2E+Pierce%2C+K%2E+Wagstaff%2E+2001%2E+Multi%2Ddocument+summarization+via+information+extraction%2E+In+Proceedings+of+the+HLT+Conference%2C+263%E2%80%93269%2E)
28. A. Wray. 2002. Formulaic Language and the Lexicon. Cambridge University Press, Cambridge. [find](https://scholar.google.com/scholar?q=A%2E+Wray%2E+2002%2E+Formulaic+Language+and+the+Lexicon%2E+Cambridge+University+Press%2C+Cambridge%2E)
29. J. Wu, S. Khudanpur. 2002. Building a topic-dependent maximum entropy language model for very large corpora. In Proceedings of ICASSP, volume 1, 777–780. [find](https://scholar.google.com/scholar?q=J%2E+Wu%2C+S%2E+Khudanpur%2E+2002%2E+Building+a+topic%2Ddependent+maximum+entropy+language+model+for+very+large+corpora%2E+In+Proceedings+of+ICASSP%2C+volume+1%2C+777%E2%80%93780%2E)