---
title: "Evaluating Topic Coherence Using — Aletras 2013"
description: "The paper addresses how to automatically judge whether the top words of a topic produced by a topic model such as LDA form an interpretable, coherent set."
authors: ["Nikolaos Aletras", "Mark Stevenson"]
year: 2013
venue: "Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers"
source: "https://aclanthology.org/W13-0102/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aletras-2013-evaluating-topic-coherence-distributional-semantics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aletras-2013-evaluating-topic-coherence-distributional-semantics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2013](https://md.research.elamur.ai/free/en/papers/2013/) / Evaluating Topic Coherence Using Distributional Semantics

# Evaluating Topic Coherence Using Distributional Semantics

**Authors:** [Nikolaos Aletras](https://md.research.elamur.ai/free/en/authors/nikolaos-aletras/), [Mark Stevenson](https://md.research.elamur.ai/free/en/authors/mark-stevenson/) · **Year:** 2013 · **Venue:** Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W13-0102/) · [PDF](https://aclanthology.org/W13-0102.pdf)

**Citations:** [63](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses how to automatically judge whether the top words of a topic produced by a topic model such as LDA form an interpretable, coherent set. Earlier work by Newman et al. (2010b) averaged pairwise PMI between topic words. The authors instead represent each topic word as a context vector built from co-occurrence statistics in Wikipedia, using a ±5-word window. Vectors are weighted with PMI or NPMI, raised to a power γ that boosts strongly associated features. Coherence is then the average pairwise similarity between vectors (cosine, Dice or Jaccard) or the mean cosine to the topic centroid. They test three vector spaces: a full space, a Reduced Semantic Space that keeps the most related features per word, and a Topic Word Space that uses only the topic's own words as features. To evaluate, they built a new public data set of 300 LDA topics with crowdsourced human ratings. The Topic Word Space correlated best with humans and consistently beat earlier methods. This offers a more reliable automatic check of topic quality.

## Key points
- The data set has 300 topics, each shown as its 10 most probable words, from three collections (100 per collection):
  - NYT: 47,229 GigaWord articles.
  - 20NG: 20 Newsgroups.
  - Genomics: 30,000 MEDLINE articles from TREC-Genomics.
- Topics were generated with gensim LDA, and coherence ratings were collected via CrowdFlower.
- Under Spearman correlation with human ratings, average pairwise PMI (Newman et al.) scored about 0.71 on NYT, 0.73 on 20NG and 0.75 on Genomics. Averaging NPMI instead added roughly 0.03 on every set.
- Cosine similarity in the Topic Word Space with PMI weighting reached 0.76 (NYT), 0.79 (20NG) and 0.80 (Genomics), which is 0.05–0.06 above average PMI.
- The co-document frequency measure of Mimno et al. (2011) did worst: −0.39 on NYT, 0.34 on 20NG and −0.4 on Genomics.
- Cosine generally outperformed Dice and Jaccard, and centroid similarity was about as good as cosine. Polysemous context features hurt the Reduced Semantic Space on NYT and Genomics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 60 not yet.

- [Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality](https://md.research.elamur.ai/free/en/papers/lau-2014-machine-reading-tea-leaves-automatically/)
- [When Numbers Tell Half the Story: Human-Metric Alignment in Topic Model Evaluation](https://md.research.elamur.ai/free/en/papers/prouteau-2026-when-numbers-tell-half-story/)
- [Exploring the Space of Topic Coherence Measures](https://md.research.elamur.ai/free/en/papers/roder-2015-exploring-space-topic-coherence-measures/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aletras2013evaluating,
  title = {{Evaluating Topic Coherence Using Distributional Semantics}},
  author = {Aletras, Nikolaos and Stevenson, Mark},
  booktitle = {Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers},
  year = {2013},
  url = {https://aclanthology.org/W13-0102/}
}
```

- [← previous Error-Driven Analysis of Challenges in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/kummerfeld-2013-error-driven-analysis-challenges-coreference/)
- [next → Experimenting with the king of France: Topics, verifiability and definite descriptions](https://md.research.elamur.ai/free/en/papers/abrusan-2013-experimenting-king-france-topics-verifiability/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/en/papers/2013/)

## References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loulwah AlSumait, Daniel Barbará, James Gentle, and Carlotta Domeniconi. Topic Significance Ranking of LDA Generative Models. Machine Learning and Knowledge Discovery in Databases, pages 67–82, 2009. [find](https://scholar.google.com/scholar?q=Loulwah+AlSumait%2C+Daniel+Barbara%CC%81%2C+James+Gentle%2C+and+Carlotta+Domeniconi%2E+Topic+Significance+Ranking+of+LDA+Generative+Models%2E+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+pages+67%E2%80%9382%2C+2009%2E)
2. David Andrzejewski, Xiaojin Zhu, and Mark Craven. Incorporating Domain Knowledge into Topic Modeling via Dirichlet Forest Priors. In Proceedings of the International Conference on Machine Learning, pages 25–32, 2009. [find](https://scholar.google.com/scholar?q=David+Andrzejewski%2C+Xiaojin+Zhu%2C+and+Mark+Craven%2E+Incorporating+Domain+Knowledge+into+Topic+Modeling+via+Dirichlet+Forest+Priors%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning%2C+pa)
3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, 2003. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei%2C+Andrew+Y%2E+Ng%2C+and+Michael+I%2E+Jordan%2E+Latent+Dirichlet+Allocation%2E+Journal+of+Machine+Learning+Research%2C+3%3A993%E2%80%931022%2C+2003%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
4. G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In Proceedings of the International Conference of the German Society for Computational Linguistics and Language Technology (GSCL ’09), Potsdam, Germany, 2009. [find](https://scholar.google.com/scholar?q=G%2E+Bouma%2E+Normalized+%28pointwise%29+mutual+information+in+collocation+extraction%2E+In+Proceedings+of+the+International+Conference+of+the+German+Society+for+Computational+Linguistics+and+Language+Technolog)
5. Jordan Boyd-Graber, David Blei, and Xiaojin Zhu. A topic model for word sense disambiguation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL ’07), pages 1024–1033, 2007. [find](https://scholar.google.com/scholar?q=Jordan+Boyd%2DGraber%2C+David+Blei%2C+and+Xiaojin+Zhu%2E+A+topic+model+for+word+sense+disambiguation%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+Comput)
6. Allison June-Barlow Chaney and David M. Blei. Visualizing topic models. In Proceedings of the 6th International AAAI Conference on Weblogs and Social Media (IWSCM), 2012. [find](https://scholar.google.com/scholar?q=Allison+June%2DBarlow+Chaney+and+David+M%2E+Blei%2E+Visualizing+topic+models%2E+In+Proceedings+of+the+6th+International+AAAI+Conference+on+Weblogs+and+Social+Media+%28IWSCM%29%2C+2012%2E)
7. Jonathan Chang, Jordan Boyd-Graber, and Sean Gerrish. Reading Tea Leaves: How Humans Interpret Topic Models. Neural Information, pages 1–9, 2009. [find](https://scholar.google.com/scholar?q=Jonathan+Chang%2C+Jordan+Boyd%2DGraber%2C+and+Sean+Gerrish%2E+Reading+Tea+Leaves%3A+How+Humans+Interpret+Topic+Models%2E+Neural+Information%2C+pages+1%E2%80%939%2C+2009%2E) [our card](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/)
8. James R. Curran. From distributional to semantic similarity. Ph.D. Thesis, University of Edinburgh, 2003. [find](https://scholar.google.com/scholar?q=James+R%2E+Curran%2E+From+distributional+to+semantic+similarity%2E+Ph%2ED%2E+Thesis%2C+University+of+Edinburgh%2C+2003%2E)
9. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6): 391–407, 1990. [find](https://scholar.google.com/scholar?q=Scott+Deerwester%2C+Susan+T%2E+Dumais%2C+George+W%2E+Furnas%2C+Thomas+K%2E+Landauer%2C+and+Richard+Harshman%2E+Indexing+by+latent+semantic+analysis%2E+Journal+of+the+American+Society+for+Information+Science%2C+41%286%29%3A+391)
10. Gregory Grefenstette. Explorations in Automatic Thesaurus Discovery. Springer, 1994. [find](https://scholar.google.com/scholar?q=Gregory+Grefenstette%2E+Explorations+in+Automatic+Thesaurus+Discovery%2E+Springer%2C+1994%2E)
11. Aria Haghighi and Lucy Vanderwende. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362–370, Boulder, Colorado, 2009. [find](https://scholar.google.com/scholar?q=Aria+Haghighi+and+Lucy+Vanderwende%2E+Exploring+content+models+for+multi%2Ddocument+summarization%2E+In+Proceedings+of+Human+Language+Technologies%3A+The+2009+Annual+Conference+of+the+North+American+Chapter+o) [our card](https://md.research.elamur.ai/free/en/papers/haghighi-2009-exploring-content-models-multi-document/)
12. Zellig Sabbettai Harris. Distributional structure. Word, 10:146–162, 1954. [find](https://scholar.google.com/scholar?q=Zellig+Sabbettai+Harris%2E+Distributional+structure%2E+Word%2C+10%3A146%E2%80%93162%2C+1954%2E)
13. Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’99), pages 50–57, Berkeley, California, United States, 1999. [find](https://scholar.google.com/scholar?q=Thomas+Hofmann%2E+Probabilistic+latent+semantic+indexing%2E+In+Proceedings+of+the+22nd+Annual+International+ACM+SIGIR+Conference+on+Research+and+Development+in+Information+Retrieval+%28SIGIR+%E2%80%9999%29%2C+pages+50%E2%80%93)
14. Aminul Md. Islam and Diana Inkpen. Second Order Co-occurrence PMI for Determining the Semantic Similarity of Words. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC ’06), pages 1033–1038, 2006. [find](https://scholar.google.com/scholar?q=Aminul+Md%2E+Islam+and+Diana+Inkpen%2E+Second+Order+Co%2Doccurrence+PMI+for+Determining+the+Semantic+Similarity+of+Words%2E+In+Proceedings+of+the+5th+International+Conference+on+Language+Resources+and+Evaluat)
15. Jey Han Lau, Karl Grieser, David Newman, and Timothy Baldwin. Automatic labelling of topic models. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1536–1545, Portland, Oregon, USA, June 2011. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Karl+Grieser%2C+David+Newman%2C+and+Timothy+Baldwin%2E+Automatic+labelling+of+topic+models%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Lan)
16. David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 262–272, Edinburgh, Scotland, UK, 2011. [find](https://scholar.google.com/scholar?q=David+Mimno%2C+Hanna+Wallach%2C+Edmund+Talley%2C+Miriam+Leenders%2C+and+Andrew+McCallum%2E+Optimizing+semantic+coherence+in+topic+models%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+La) [our card](https://md.research.elamur.ai/free/en/papers/mimno-2011-optimizing-semantic-coherence-topic-models/)
17. Claudiu C. Musat, Julien Velcin, Stefan Trausan-Matu, and Marian A. Rizoiu. Improving topic evaluation using conceptual knowledge. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence (IJCAI ’11), pages 1866–1871, Barcelona, Spain, 2011. [find](https://scholar.google.com/scholar?q=Claudiu+C%2E+Musat%2C+Julien+Velcin%2C+Stefan+Trausan%2DMatu%2C+and+Marian+A%2E+Rizoiu%2E+Improving+topic+evaluation+using+conceptual+knowledge%2E+In+Proceedings+of+the+Twenty%2DSecond+International+Joint+Conference+on)
18. David Newman, Timothy Baldwin, Lawrence Cavedon, Eric Huang, Sarvnaz Karimi, David Martinez, Falk Scholer, and Justin Zobel. Visualizing search results and document collections using topic maps. Web Semantics: Science, Services and Agents on the World Wide Web, 8(23):169–175, 2010a. [find](https://scholar.google.com/scholar?q=David+Newman%2C+Timothy+Baldwin%2C+Lawrence+Cavedon%2C+Eric+Huang%2C+Sarvnaz+Karimi%2C+David+Martinez%2C+Falk+Scholer%2C+and+Justin+Zobel%2E+Visualizing+search+results+and+document+collections+using+topic+maps%2E+Web+S)
19. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT ’10), pages 100–108, Los Angeles, California, 20… [find](https://scholar.google.com/scholar?q=David+Newman%2C+Jey+Han+Lau%2C+Karl+Grieser%2C+and+Timothy+Baldwin%2E+Automatic+evaluation+of+topic+coherence%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+Chapter+of+the+A) [our card](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/)
20. Sebastian Padó and Mirella Lapata. Constructing semantic space models from parsed corpora. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 128–135, Sapporo, Japan, 2003. [find](https://scholar.google.com/scholar?q=Sebastian+Pado%CC%81+and+Mirella+Lapata%2E+Constructing+semantic+space+models+from+parsed+corpora%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+128%E2%80%93135%2C+S)
21. Judita Preiss. Identifying comparable corpora using LDA. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 558–562, Montréal, Canada, 2012. [find](https://scholar.google.com/scholar?q=Judita+Preiss%2E+Identifying+comparable+corpora+using+LDA%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technologi)
22. Eduardo H. Ramirez, Ramon Brena, Davide Magatti, and Fabio Stella. Topic Model Validation. Neurocomputing, 76(1):125–133, 2012. [find](https://scholar.google.com/scholar?q=Eduardo+H%2E+Ramirez%2C+Ramon+Brena%2C+Davide+Magatti%2C+and+Fabio+Stella%2E+Topic+Model+Validation%2E+Neurocomputing%2C+76%281%29%3A125%E2%80%93133%2C+2012%2E)
23. Peter D. Turney and Patrick Pantel. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37:141–188, 2010. [find](https://scholar.google.com/scholar?q=Peter+D%2E+Turney+and+Patrick+Pantel%2E+From+frequency+to+meaning%3A+Vector+space+models+of+semantics%2E+Journal+of+Artificial+Intelligence+Research%2C+37%3A141%E2%80%93188%2C+2010%2E)
24. Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. Evaluation Methods for Topic Models. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML ’09), pages 1105–1112, Montreal, Quebec, Canada, 2009. [find](https://scholar.google.com/scholar?q=Hanna+M%2E+Wallach%2C+Iain+Murray%2C+Ruslan+Salakhutdinov%2C+and+David+Mimno%2E+Evaluation+Methods+for+Topic+Models%2E+In+Proceedings+of+the+26th+Annual+International+Conference+on+Machine+Learning+%28ICML+%E2%80%9909%29%2C+pa)
25. Xing Wei and W. Bruce Croft. LDA-based Document Models for Ad-hoc Retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR ’06), pages 178–185, 2006. [find](https://scholar.google.com/scholar?q=Xing+Wei+and+W%2E+Bruce+Croft%2E+LDA%2Dbased+Document+Models+for+Ad%2Dhoc+Retrieval%2E+In+Proceedings+of+the+29th+annual+international+ACM+SIGIR+conference+on+Research+and+Development+in+Information+Retrieval+%28)