---
title: "Pre-training is a Hot Topic — Bianchi 2021"
description: "Classic topic models take bag-of-words (BoW) input, which ignores word order and meaning."
authors: ["Federico Bianchi", "Silvia Terragni", "Dirk Hovy"]
year: 2021
venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)"
source: "https://aclanthology.org/2021.acl-short.96/"
doi: "10.18653/v1/2021.acl-short.96"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

# Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence

**Authors:** [Federico Bianchi](https://md.research.elamur.ai/free/en/authors/federico-bianchi/), Silvia Terragni, [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/) · **Year:** 2021 · **Venue:** Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.acl-short.96/) · [PDF](https://aclanthology.org/2021.acl-short.96.pdf) · [DOI 10.18653/v1/2021.acl-short.96](https://doi.org/10.18653/v1/2021.acl-short.96) · [OpenAlex](https://openalex.org/W3015202090)

**Citations:** 269 citations (OpenAlex, 2026-10-14) · [96](https://md.research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Classic topic models take bag-of-words (BoW) input, which ignores word order and meaning. The resulting topics are often hard to interpret. The authors ask whether pre-trained contextual representations can make topics more coherent.

They propose CombinedTM, which extends the neural ProdLDA model, a VAE-based topic model. Each document is encoded with SBERT, projected to vocabulary size and concatenated with the BoW vector before the inference network. The approach works with any autoencoder-style topic model and any document encoder. One limitation is that text beyond SBERT's length limit is dropped.

They tested it on five English corpora (20Newsgroups, Wiki20K, StackOverflow, Tweets2011, Google News) against ProdLDA, NVDM, ETM, MetaLDA and LDA. CombinedTM produced the most coherent topics on four of the five datasets while keeping topic diversity high. A stronger sentence encoder improved results further.

This matters because topic models can gain from progress in language models without architectural redesign. The code is released as an open Python library.

## Key points
- **Evaluation setup:** models were scored with NPMI (τ), word-embedding coherence (α, using word2vec) and inverted RBO diversity (ρ). Results cover 25, 50, 75, 100 and 150 topics, averaged over 30 runs each.
- **Average τ gains:** CombinedTM reached 0.1823 vs 0.1443 for MetaLDA on Wiki20K, and 0.1207 vs 0.0849 on Google News. On Tweets2011 it scored 0.1008 vs 0.0612 for ProdLDA. Only on 20Newsgroups was it beaten, by MetaLDA (0.1300 vs 0.1025).
- **Diversity:** CombinedTM's ρ stayed around 0.98–0.997. ETM had good α on some corpora but produced highly redundant topics (ρ ≈ 0.43–0.86).
- **Encoder comparison:** stsb-roberta-large matched or outperformed bert-base-nli-means on every dataset. For example, τ was 0.12 vs 0.08 on Google News.
- **Data and code:** dataset sizes ranged from 2,471 documents (Tweets2011) to 20,000 (Wiki20K). The implementation is available at github.com/MilaNLProc/contextualized-topic-models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Institutions

- [University of Milan](https://md.research.elamur.ai/free/en/universities/university-of-milan/) — University, Milan, Italy · 3 works · 38 686 citations
- [University of Milano-Bicocca](https://md.research.elamur.ai/free/en/universities/university-of-milano-bicocca/) — University, Milan, Italy · 9 works · 4 698 citations
- [Bocconi University](https://md.research.elamur.ai/free/en/universities/bocconi-university/) — University, Milan, Italy · 10 works · 2 044 citations

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bianchi2021pre,
  title = {{Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence}},
  author = {Bianchi, Federico and Terragni, Silvia and Hovy, Dirk},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.96},
  url = {https://doi.org/10.18653/v1/2021.acl-short.96}
}
```

- [← previous Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction](https://md.research.elamur.ai/free/en/papers/lu-2021-text2event-controllable-sequence-structure-generation/)
- [next → The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. 2021. Cross-lingual contextualized topic models with zero-shot learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1676–1683, O… [find](https://scholar.google.com/scholar?q=Federico+Bianchi%2C+Silvia+Terragni%2C+Dirk+Hovy%2C+Debora+Nozza%2C+and+Elisabetta+Fersini%2E+2021%2E+Cross%2Dlingual+contextualized+topic+models+with+zero%2Dshot+learning%2E+In+Proceedings+of+the+16th+Conference+of+th)
2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei%2C+Andrew+Y%2E+Ng%2C+and+Michael+I%2E+Jordan%2E+2003%2E+Latent+dirichlet+allocation%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+3%3A993%E2%80%931022%2E) [our card](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/)
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
4. Stephen Carrow. 2018. PyTorchAVITM: Open Source AVITM Implementation in PyTorch. Github. [find](https://scholar.google.com/scholar?q=Stephen+Carrow%2E+2018%2E+PyTorchAVITM%3A+Open+Source+AVITM+Implementation+in+PyTorch%2E+Github%2E)
5. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+In%CC%83igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+t) [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
6. Jonathan Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, and David M. Blei. 2009. Reading tea leaves: How humans interpret topic models. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009, pages 288–296. Curran Associ… [find](https://scholar.google.com/scholar?q=Jonathan+Chang%2C+Jordan+L%2E+Boyd%2DGraber%2C+Sean+Gerrish%2C+Chong+Wang%2C+and+David+M%2E+Blei%2E+2009%2E+Reading+tea+leaves%3A+How+humans+interpret+topic+models%2E+In+Advances+in+Neural+Information+Processing+Systems+22) [our card](https://md.research.elamur.ai/free/en/papers/chang-2009-reading-tea-leaves-humans-interpret-topic/)
7. Zhiyuan Chen, Arjun Mukherjee, Bing Liu, Meichun Hsu, Malú Castellanos, and Riddhiman Ghosh. 2013. Discovering coherent topics using general knowledge. In 22nd ACM International Conference on Information and Knowledge Management, CIKM’13, San Francisco, CA, USA, October 27 -November 1, 2013, pages… [find](https://scholar.google.com/scholar?q=Zhiyuan+Chen%2C+Arjun+Mukherjee%2C+Bing+Liu%2C+Meichun+Hsu%2C+Malu%CC%81+Castellanos%2C+and+Riddhiman+Ghosh%2E+2013%2E+Discovering+coherent+topics+using+general+knowledge%2E+In+22nd+ACM+International+Conference+on+Informa)
8. Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian LDA for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natur… [find](https://scholar.google.com/scholar?q=Rajarshi+Das%2C+Manzil+Zaheer%2C+and+Chris+Dyer%2E+2015%2E+Gaussian+LDA+for+topic+models+with+word+embeddings%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+th)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
10. Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439–453. [find](https://scholar.google.com/scholar?q=Adji+B%2E+Dieng%2C+Francisco+J%2E+R%2E+Ruiz%2C+and+David+M%2E+Blei%2E+2020%2E+Topic+modeling+in+embedding+spaces%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A439%E2%80%93453%2E) [our card](https://md.research.elamur.ai/free/en/papers/dieng-2020-topic-modeling-embedding-spaces/)
11. Ran Ding, Ramesh Nallapati, and Bing Xiang. 2018. Coherence-aware neural topic modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 830– 836. [find](https://scholar.google.com/scholar?q=Ran+Ding%2C+Ramesh+Nallapati%2C+and+Bing+Xiang%2E+2018%2E+Coherence%2Daware+neural+topic+modeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+Brussels%2C+Belgium%2C+O)
12. Pankaj Gupta, Yatin Chaudhary, Florian Buettner, and Hinrich Schütze. 2019. Document informed neural autoregressive topic models with distributional prior. In AAAI2019, pages 6505–6512. AAAI Press. [find](https://scholar.google.com/scholar?q=Pankaj+Gupta%2C+Yatin+Chaudhary%2C+Florian+Buettner%2C+and+Hinrich+Schu%CC%88tze%2E+2019%2E+Document+informed+neural+autoregressive+topic+models+with+distributional+prior%2E+In+AAAI2019%2C+pages+6505%E2%80%936512%2E+AAAI+Press%2E)
13. Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, and Hinrich Schuetze. 2020. Neural topic modeling with continual lifelong learning. In International Conference on Machine Learning, pages 3907–3917. PMLR. [find](https://scholar.google.com/scholar?q=Pankaj+Gupta%2C+Yatin+Chaudhary%2C+Thomas+Runkler%2C+and+Hinrich+Schuetze%2E+2020%2E+Neural+topic+modeling+with+continual+lifelong+learning%2E+In+International+Conference+on+Machine+Learning%2C+pages+3907%E2%80%933917%2E+PML)
14. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeffrey+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+In+NIPS+Deep+Learning+and+Representation+Learning+Workshop%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
15. Geoffrey E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771–1800. [find](https://scholar.google.com/scholar?q=Geoffrey+E%2E+Hinton%2E+2002%2E+Training+products+of+experts+by+minimizing+contrastive+divergence%2E+Neural+Computation%2C+14%288%29%3A1771%E2%80%931800%2E)
16. Alexander Miserlis Hoyle, Pranav Goel, and Philip Resnik. 2020. Improving Neural Topic Models using Knowledge Distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1752–1771, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+Miserlis+Hoyle%2C+Pranav+Goel%2C+and+Philip+Resnik%2E+2020%2E+Improving+Neural+Topic+Models+using+Knowledge+Distillation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langua)
17. Hugo Larochelle and Stanislas Lauly. 2012. A neural autoregressive topic model. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, pages 2717–2725. [find](https://scholar.google.com/scholar?q=Hugo+Larochelle+and+Stanislas+Lauly%2E+2012%2E+A+neural+autoregressive+topic+model%2E+In+Advances+in+Neural+Information+Processing+Systems+25%3A+26th+Annual+Conference+on+Neural+Information+Processing+Systems)
18. Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, pages 530–539. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+David+Newman%2C+and+Timothy+Baldwin%2E+2014%2E+Machine+reading+tea+leaves%3A+Automatically+evaluating+topic+coherence+and+topic+model+quality%2E+In+Proceedings+of+the+14th+Conference+of+the+Europea) [our card](https://md.research.elamur.ai/free/en/papers/lau-2014-machine-reading-tea-leaves-automatically/)
19. Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, volume 70 of Proceedings of Machine Learning Research, pages 2410–2419. PMLR. [find](https://scholar.google.com/scholar?q=Yishu+Miao%2C+Edward+Grefenstette%2C+and+Phil+Blunsom%2E+2017%2E+Discovering+discrete+latent+topics+with+neural+variational+inference%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+)
20. Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1727– 1736. JMLR.or… [find](https://scholar.google.com/scholar?q=Yishu+Miao%2C+Lei+Yu%2C+and+Phil+Blunsom%2E+2016%2E+Neural+variational+inference+for+text+processing%2E+In+Proceedings+of+the+33nd+International+Conference+on+Machine+Learning%2C+ICML+2016%2C+New+York+City%2C+NY%2C+USA)
21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
22. Andriy Mnih and Karol Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1791– 1799. JMLR.org. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Karol+Gregor%2E+2014%2E+Neural+variational+inference+and+learning+in+belief+networks%2E+In+Proceedings+of+the+31th+International+Conference+on+Machine+Learning%2C+ICML+2014%2C+volume+32+of+JMLR+)
23. Aaron Mueller and Mark Dredze. 2021. Fine-tuning encoders for improved monolingual and zero-shot polylingual neural topic modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3054–3068, Onli… [find](https://scholar.google.com/scholar?q=Aaron+Mueller+and+Mark+Dredze%2E+2021%2E+Fine%2Dtuning+encoders+for+improved+monolingual+and+zero%2Dshot+polylingual+neural+topic+modeling%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+)
24. Dat Quoc Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. 2015. Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics, 3:299–313. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Richard+Billingsley%2C+Lan+Du%2C+and+Mark+Johnson%2E+2015%2E+Improving+topic+models+with+latent+feature+word+representations%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A)
25. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making Sense of Language-Specific BERT Models. arXiv preprint arXiv:2003.02912. — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5BMASK%5D%3F+Making+Sense+of+Language%2DSpecific+BERT+Models%2E+arXiv+preprint+arXiv%3A2003%2E02912%2E+%E2%80%94+%5BarXiv%3A2003%2E02912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E02)
26. Jipeng Qiang, Qian Zhenyu, Yun Li, Yunhao Yuan, and Xindong Wu. 2019. Short text topic modeling techniques, applications, and performance: A survey. arXiv preprint arXiv:1904.07695. — [arXiv:1904.07695](https://arxiv.org/abs/1904.07695) [find](https://scholar.google.com/scholar?q=Jipeng+Qiang%2C+Qian+Zhenyu%2C+Yun+Li%2C+Yunhao+Yuan%2C+and+Xindong+Wu%2E+2019%2E+Short+text+topic+modeling+techniques%2C+applications%2C+and+performance%3A+A+survey%2E+arXiv+preprint+arXiv%3A1904%2E07695%2E+%E2%80%94+%5BarXiv%3A1904%2E0769)
27. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1–38. [find](https://scholar.google.com/scholar?q=William+Webber%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2010%2E+A+similarity+measure+for+indefinite+rankings%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C+28%284%29%3A1%E2%80%9338%2E)
28. Rajesh Ranganath, Sean Gerrish, and David M. Blei. 2014. Black box variational inference. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS 2014, 2014, volume 33 of JMLR Workshop and Conference Proceedings, pages 814–822. JMLR.org. [find](https://scholar.google.com/scholar?q=Rajesh+Ranganath%2C+Sean+Gerrish%2C+and+David+M%2E+Blei%2E+2014%2E+Black+box+variational+inference%2E+In+Proceedings+of+the+Seventeenth+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+AISTATS+)
29. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/)
31. Guangxu Xun, Yaliang Li, Wayne Xin Zhao, Jing Gao, and Aidong Zhang. 2017. A correlated topic model using word embeddings. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 4207–4213. ijcai.org. [find](https://scholar.google.com/scholar?q=Guangxu+Xun%2C+Yaliang+Li%2C+Wayne+Xin+Zhao%2C+Jing+Gao%2C+and+Aidong+Zhang%2E+2017%2E+A+correlated+topic+model+using+word+embeddings%2E+In+Proceedings+of+the+Twenty%2DSixth+International+Joint+Conference+on+Artifici)
32. Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, pages 399–408. ACM. [find](https://scholar.google.com/scholar?q=Michael+Ro%CC%88der%2C+Andreas+Both%2C+and+Alexander+Hinneburg%2E+2015%2E+Exploring+the+space+of+topic+coherence+measures%2E+In+Proceedings+of+the+Eighth+ACM+International+Conference+on+Web+Search+and+Data+Mining%2C+W) [our card](https://md.research.elamur.ai/free/en/papers/roder-2015-exploring-space-topic-coherence-measures/)
33. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842–866. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+and+Anna+Rumshisky%2E+2020%2E+A+primer+in+BERTology%3A+What+we+know+about+how+BERT+works%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A842%E2%80%93866%2E) [our card](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/)
34. Yi Yang, Doug Downey, and Jordan L. Boyd-Graber. 2015. Efficient methods for incorporating knowledge into topic models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 308–317. The Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yi+Yang%2C+Doug+Downey%2C+and+Jordan+L%2E+Boyd%2DGraber%2E+2015%2E+Efficient+methods+for+incorporating+knowledge+into+topic+models%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+P)
35. Ruslan Salakhutdinov and Geoffrey E. Hinton. 2009. Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009, pages 1607–1614. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Ruslan+Salakhutdinov+and+Geoffrey+E%2E+Hinton%2E+2009%2E+Replicated+softmax%3A+an+undirected+topic+model%2E+In+Advances+in+Neural+Information+Processing+Systems+22%3A+23rd+Annual+Conference+on+Neural+Information+)
36. He Zhao, Lan Du, Wray Buntine, and Gang Liu. 2017. Metalda: A topic model that efficiently incorporates meta information. In 2017 IEEE International Conference on Data Mining (ICDM), pages 635–644. IEEE. [find](https://scholar.google.com/scholar?q=He+Zhao%2C+Lan+Du%2C+Wray+Buntine%2C+and+Gang+Liu%2E+2017%2E+Metalda%3A+A+topic+model+that+efficiently+incorporates+meta+information%2E+In+2017+IEEE+International+Conference+on+Data+Mining+%28ICDM%29%2C+pages+635%E2%80%93644%2E+IE)
37. Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Akash+Srivastava+and+Charles+Sutton%2E+2017%2E+Autoencoding+variational+inference+for+topic+models%2E+In+5th+International+Conference+on+Learning+Representations%2C+ICLR+2017%2C+Toulon%2C+France%2C+April+24%2D26%2C+201)
38. Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021a. OCTIS: Comparing and optimizing topic models is simple! In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,… [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Elisabetta+Fersini%2C+Bruno+Giovanni+Galuzzi%2C+Pietro+Tropeano%2C+and+Antonio+Candelieri%2E+2021a%2E+OCTIS%3A+Comparing+and+optimizing+topic+models+is+simple%21+In+Proceedings+of+the+16th+Conferen)
39. Silvia Terragni, Elisabetta Fersini, and Enza Messina. 2020a. Constrained relational topic models. Information Sciences, 512:581–594. [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Elisabetta+Fersini%2C+and+Enza+Messina%2E+2020a%2E+Constrained+relational+topic+models%2E+Information+Sciences%2C+512%3A581%E2%80%93594%2E)
40. Silvia Terragni, Elisabetta Fersini, and Enza Messina. 2021b. Word embedding-based topic similarity measures. In Natural Language Processing and Information Systems - 26th International Conference on Applications of Natural Language to Information Systems, NLDB 2021. Springer. [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Elisabetta+Fersini%2C+and+Enza+Messina%2E+2021b%2E+Word+embedding%2Dbased+topic+similarity+measures%2E+In+Natural+Language+Processing+and+Information+Systems+%2D+26th+International+Conference+on+)
41. Silvia Terragni, Debora Nozza, Elisabetta Fersini, and Messina Enza. 2020b. Which matters most? comparing the impact of concept and document relationships in topic models. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 32–40. [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Debora+Nozza%2C+Elisabetta+Fersini%2C+and+Messina+Enza%2E+2020b%2E+Which+matters+most%3F+comparing+the+impact+of+concept+and+document+relationships+in+topic+models%2E+In+Proceedings+of+the+First+)
42. He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021. Topic modelling meets deep neural networks: A survey. arXiv preprint arXiv:2103.00498. — [arXiv:2103.00498](https://arxiv.org/abs/2103.00498) [find](https://scholar.google.com/scholar?q=He+Zhao%2C+Dinh+Phung%2C+Viet+Huynh%2C+Yuan+Jin%2C+Lan+Du%2C+and+Wray+Buntine%2E+2021%2E+Topic+modelling+meets+deep+neural+networks%3A+A+survey%2E+arXiv+preprint+arXiv%3A2103%2E00498%2E+%E2%80%94+%5BarXiv%3A2103%2E00498%5D%28https%3A%2F%2Farxiv%2Eorg)