---
title: "Optimizing Semantic Coherence in Topic Models — Mimno 2011"
description: "Topic models such as LDA often produce topics that domain experts immediately see as flawed, and this undermines trust."
authors: ["David Mimno", "Hanna Wallach", "Edmund Talley", "Miriam Leenders", "Andrew McCallum"]
year: 2011
venue: "EMNLP 2011"
source: "https://aclanthology.org/D11-1024/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mimno-2011-optimizing-semantic-coherence-topic-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mimno-2011-optimizing-semantic-coherence-topic-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Optimizing Semantic Coherence in Topic Models

# Optimizing Semantic Coherence in Topic Models

**Authors:** [David Mimno](https://md.research.elamur.ai/free/en/authors/david-mimno/), [Hanna Wallach](https://md.research.elamur.ai/free/en/authors/hanna-wallach/), Edmund Talley, Miriam Leenders, [Andrew McCallum](https://md.research.elamur.ai/free/en/authors/andrew-mccallum/) · **Year:** 2011 · **Venue:** EMNLP 2011 · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D11-1024/) · [PDF](https://aclanthology.org/D11-1024.pdf) · [OpenAlex](https://openalex.org/W2130339025)

**Citations:** 1189 citations (openalex-snapshot · 2026-10-14) · [145](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Topic models such as LDA often produce topics that domain experts immediately see as flawed, and this undermines trust. The problem is worst for the smallest topics in models with many topics. The authors worked with two NINDS experts on about 300,000 NIH grant and journal abstracts. The experts rated 148 of 500 LDA topics and labeled their failure types (chained, intruded, random, unbalanced). The authors then propose an intrinsic coherence score based on how often a topic's top words co-occur in documents of the training corpus itself. It needs neither human judges nor external resources such as Wikipedia, and it matches expert ratings well. Building on this score, they design a topic model that replaces LDA's standard Pólya (DCM) word distribution with a generalized Pólya urn, so that drawing one word can raise the probability of related words. The new model increases average coherence, improves the weakest topics and converges in far fewer Gibbs iterations. However, its final held-out likelihood falls below LDA's, and its reduction in bad topics is not statistically significant. The work offers a cheap, self-contained way to detect and reduce junk topics.

## Key points
- Expert annotation of 148 NIH topics: 90 good, 21 intermediate, 37 bad. Among flawed topics, 23 were chained, 21 intruded, 3 random and 15 unbalanced (multiple labels were allowed). The two experts never split good versus bad.
- The coherence metric uses only document frequency and co-document frequency of top words from the modeled corpus. It flags problems, such as chained topics, that word-intrusion tests miss.
- The generalized Pólya urn model reaches its final coherence level within about 50 Gibbs iterations. Each iteration costs roughly 2–3× LDA's time. Results held for T ∈ {100, 200, 300}, using 5-fold cross-validation with 3 initializations each.
- In a blind expert re-evaluation at T = 200, coherence predicted bad topics with AUC 0.83 for the new model and 0.80 for LDA.
- Bad topics made up 13.5% of the new model's topics versus 16.5% for LDA. This difference is not significant at p = 0.05.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

## Cited by

Cited by

145 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 140 not yet.

- [Evaluating Topic Coherence Using Distributional Semantics](https://md.research.elamur.ai/free/en/papers/aletras-2013-evaluating-topic-coherence-distributional-semantics/)
- [Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality](https://md.research.elamur.ai/free/en/papers/lau-2014-machine-reading-tea-leaves-automatically/)
- [When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation](https://md.research.elamur.ai/free/en/papers/prouteau-2026-when-numbers-tell-half-story/)
- [Exploring the Space of Topic Coherence Measures](https://md.research.elamur.ai/free/en/papers/roder-2015-exploring-space-topic-coherence-measures/)
- [Modeling Changing Scientific Concepts with Complex Networks: A Case Study on the Chemical Revolution](https://md.research.elamur.ai/free/en/papers/valdez-2026-modeling-changing-scientific-concepts-complex/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mimno2011optimizing,
  title = {{Optimizing Semantic Coherence in Topic Models}},
  author = {Mimno, David and Wallach, Hanna and Talley, Edmund and Leenders, Miriam and McCallum, Andrew},
  booktitle = {EMNLP 2011},
  year = {2011},
  url = {https://aclanthology.org/D11-1024/}
}
```

- [← previous Opinion Word Expansion and Target Extraction through Double Propagation](https://md.research.elamur.ai/free/en/papers/qiu-2011-opinion-word-expansion-target-extraction/)
- [next → Overview of BioNLP 2011 Protein Coreference Shared Task](https://md.research.elamur.ai/free/en/papers/nguyen-2011-overview-bionlp-2011-protein-coreference/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/)

## References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loulwah AlSumait, Daniel Barbara, James Gentle, and Carlotta Domeniconi. 2009. Topic significance ranking of LDA generative models. In ECML. [find](https://scholar.google.com/scholar?q=Loulwah+AlSumait%2C+Daniel+Barbara%2C+James+Gentle%2C+and+Carlotta+Domeniconi%2E+2009%2E+Topic+significance+ranking+of+LDA+generative+models%2E+In+ECML%2E)
2. David Andrzejewski, Xiaojin Zhu, and Mark Craven. 2009. Incorporating domain knowledge into topic modeling via Dirichlet forest priors. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 25–32. [find](https://scholar.google.com/scholar?q=David+Andrzejewski%2C+Xiaojin+Zhu%2C+and+Mark+Craven%2E+2009%2E+Incorporating+domain+knowledge+into+topic+modeling+via+Dirichlet+forest+priors%2E+In+Proceedings+of+the+26th+Annual+International+Conference+on+Ma)
3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, January. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei%2C+Andrew+Y%2E+Ng%2C+and+Michael+I%2E+Jordan%2E+2003%2E+Latent+Dirichlet+allocation%2E+Journal+of+Machine+Learning+Research%2C+3%3A993%E2%80%931022%2C+January%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
4. K.R. Canini, L. Shi, and T.L. Griffiths. 2009. Online inference of topics with latent Dirichlet allocation. In Proceedings of the 12th International Conference on Artificial Intelligence and Statistics. [find](https://scholar.google.com/scholar?q=K%2ER%2E+Canini%2C+L%2E+Shi%2C+and+T%2EL%2E+Griffiths%2E+2009%2E+Online+inference+of+topics+with+latent+Dirichlet+allocation%2E+In+Proceedings+of+the+12th+International+Conference+on+Artificial+Intelligence+and+Statistic)
5. Jonathan Chang, Jordan Boyd-Graber, Chong Wang, Sean Gerrish, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22, pages 288–296. [find](https://scholar.google.com/scholar?q=Jonathan+Chang%2C+Jordan+Boyd%2DGraber%2C+Chong+Wang%2C+Sean+Gerrish%2C+and+David+M%2E+Blei%2E+2009%2E+Reading+tea+leaves%3A+How+humans+interpret+topic+models%2E+In+Advances+in+Neural+Information+Processing+Systems+22%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/)
6. Kenneth Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 6(1):22–29. [find](https://scholar.google.com/scholar?q=Kenneth+Church+and+Patrick+Hanks%2E+1990%2E+Word+association+norms%2C+mutual+information%2C+and+lexicography%2E+Computational+Linguistics%2C+6%281%29%3A22%E2%80%9329%2E) [our card](https://md.research.elamur.ai/free/en/papers/church-1990-word-association-norms-mutual-information/)
7. Gabriel Doyle and Charles Elkan. 2009. Accounting for burstiness in topic models. In ICML. [find](https://scholar.google.com/scholar?q=Gabriel+Doyle+and+Charles+Elkan%2E+2009%2E+Accounting+for+burstiness+in+topic+models%2E+In+ICML%2E)
8. S. Geman and D. Geman. 1984. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transaction on Pattern Analysis and Machine Intelligence 6, pages 721–741. [find](https://scholar.google.com/scholar?q=S%2E+Geman+and+D%2E+Geman%2E+1984%2E+Stochastic+relaxation%2C+Gibbs+distributions%2C+and+the+Bayesian+restoration+of+images%2E+IEEE+Transaction+on+Pattern+Analysis+and+Machine+Intelligence+6%2C+pages+721%E2%80%93741%2E)
9. Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences, 101(suppl. 1):5228–5235. [find](https://scholar.google.com/scholar?q=Thomas+L%2E+Griffiths+and+Mark+Steyvers%2E+2004%2E+Finding+scientific+topics%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+101%28suppl%2E+1%29%3A5228%E2%80%935235%2E)
10. Matthew Hoffman, David Blei, and Francis Bach. 2010. Online learning for latent dirichlet allocation. In NIPS. [find](https://scholar.google.com/scholar?q=Matthew+Hoffman%2C+David+Blei%2C+and+Francis+Bach%2E+2010%2E+Online+learning+for+latent+dirichlet+allocation%2E+In+NIPS%2E)
11. Hosan Mahmoud. 2008. Pólya Urn Models. Chapman & Hall/CRC Texts in Statistical Science. [find](https://scholar.google.com/scholar?q=Hosan+Mahmoud%2E+2008%2E+Po%CC%81lya+Urn+Models%2E+Chapman+%26+Hall%2FCRC+Texts+in+Statistical+Science%2E)
12. Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007. Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 490–499. [find](https://scholar.google.com/scholar?q=Qiaozhu+Mei%2C+Xuehua+Shen%2C+and+ChengXiang+Zhai%2E+2007%2E+Automatic+labeling+of+multinomial+topic+models%2E+In+Proceedings+of+the+13th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+and+Data+Mini)
13. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Newman%2C+Jey+Han+Lau%2C+Karl+Grieser%2C+and+Timothy+Baldwin%2E+2010%2E+Automatic+evaluation+of+topic+coherence%2E+In+Human+Language+Technologies%3A+The+Annual+Conference+of+the+North+American+Chapter+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/)
14. Yee Whye Teh, Dave Newman, and Max Welling. 2006. A collapsed variational Bayesian inference algorithm for lat ent Dirichlet allocation. In Advances in Neural Information Processing Systems 18. [find](https://scholar.google.com/scholar?q=Yee+Whye+Teh%2C+Dave+Newman%2C+and+Max+Welling%2E+2006%2E+A+collapsed+variational+Bayesian+inference+algorithm+for+lat+ent+Dirichlet+allocation%2E+In+Advances+in+Neural+Information+Processing+Systems+18%2E)
15. Hanna Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. Evaluation methods for topic models. In Proceedings of the 26th Interational Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Hanna+Wallach%2C+Iain+Murray%2C+Ruslan+Salakhutdinov%2C+and+David+Mimno%2E+2009%2E+Evaluation+methods+for+topic+models%2E+In+Proceedings+of+the+26th+Interational+Conference+on+Machine+Learning%2E)
16. Xing Wei and Bruce Croft. 2006. LDA-based document models for ad-hoc retrival. In Proceedings of the 29th Annual International SIGIR Conference. [find](https://scholar.google.com/scholar?q=Xing+Wei+and+Bruce+Croft%2E+2006%2E+LDA%2Dbased+document+models+for+ad%2Dhoc+retrival%2E+In+Proceedings+of+the+29th+Annual+International+SIGIR+Conference%2E)