---
title: "Cross-lingual Contextualized Topic Models with Zero-shot Learning"
description: "Bianchi et al. propose ZeroShotTM, a neural topic model built on Neural-ProdLDA (a VAE). 28 references, cited by 61 papers here."
authors: ["Federico Bianchi", "Silvia Terragni", "Dirk Hovy", "Debora Nozza", "Elisabetta Fersini"]
year: 2021
venue: "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume"
source: "https://aclanthology.org/2021.eacl-main.143/"
doi: "10.18653/v1/2021.eacl-main.143"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bianchi-2021-cross-lingual-contextualized-topic-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bianchi-2021-cross-lingual-contextualized-topic-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Cross-lingual Contextualized Topic Models with Zero-shot Learning

# Cross-lingual Contextualized Topic Models with Zero-shot Learning

**Authors:** [Federico Bianchi](https://md.research.elamur.ai/free/en/authors/federico-bianchi/), [Silvia Terragni](https://md.research.elamur.ai/free/en/authors/silvia-terragni/), [Dirk Hovy](https://md.research.elamur.ai/free/en/authors/dirk-hovy/), [Debora Nozza](https://md.research.elamur.ai/free/en/authors/debora-nozza/) and [Elisabetta Fersini](https://md.research.elamur.ai/free/en/authors/elisabetta-fersini/) · **Year:** 2021 · **Venue:** [Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2021](https://md.research.elamur.ai/free/en/venues/eacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.eacl-main.143/) · [PDF](https://aclanthology.org/2021.eacl-main.143.pdf) · [DOI 10.18653/v1/2021.eacl-main.143](https://doi.org/10.18653/v1/2021.eacl-main.143) · [OpenAlex](https://openalex.org/W3017299116)

**Citations:** 154 citations (OpenAlex, 2026-10-14) · [61](https://md.research.elamur.ai/free/en/papers/bianchi-2021-cross-lingual-contextualized-topic-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Classic bag-of-words topic models are bound to the vocabulary of their training language. They cannot process unseen words, and they cannot be reused for texts in other languages. Bianchi et al. propose ZeroShotTM, a neural topic model built on Neural-ProdLDA (a VAE). It takes multilingual SBERT document embeddings as input in place of the BoW. The BoW is still reconstructed by the decoder during training, which supplies the topic words, but it is not needed at inference. As a result, a model trained only on English can assign topics to documents in languages it never saw, with topic descriptors in English. The authors use DBpedia abstracts from English Wikipedia and test on 300 abstracts with counterparts in Italian, French, Portuguese and German. On English data, the contextual input keeps coherence on par with BoW-based models. Cross-lingually, predicted topics usually match those of the English version, roughly as well as a machine-translation pipeline. This makes topic modeling possible for languages with little data and requires no alignment resources.

## Key points
- **Data:** W1 contains 20,000 English abstracts. W2 contains 100,000 abstracts, split into 99,700 for training and 300 for testing, with comparable test documents in IT, FR, PT and DE. The BoW vocabulary uses the 2,000 most frequent non-stopwords.
- **Coherence on W1 (NPMI, 50/100 topics):** ZeroShotTM scores 0.1632/0.1381, comparable to Combined TM (0.1644/0.1409) and Neural-ProdLDA (0.1658/0.1285). LDA scores negative.
- **Zero-shot topic matches with English:** ZeroShotTM averages 78.00% with 25 topics and 64.41% with 50. The DeepL translation baseline (Ori) reaches 76.00% and 69.00%, while a uniform baseline reaches 4% and 2%. Centroid similarity is 0.85 and 0.77, and KL divergence is 0.15 and 0.20.
- **Manual evaluation:** 1,500 documents were rated, with an average topic quality of 2.26 out of 3 and inter-rater agreement of Gwet AC1 = 0.88.
- **Code release:** the model is available as the Python package `contextualized-topic-models` (MilaNLProc on GitHub). It uses the SBERT model distiluse-base-multilingual-cased.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Bocconi University](https://md.research.elamur.ai/free/en/universities/bocconi-university/) — University, Milan, Italy · 16 works · 2,850 citations
- [University of Milano-Bicocca](https://md.research.elamur.ai/free/en/universities/university-of-milano-bicocca/) — University, Milan, Italy · 10 works · 4,918 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 465 works

## Cite

Cite as: Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza and Elisabetta Fersini. 2021. Cross-lingual Contextualized Topic Models with Zero-shot Learning. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bianchi2021cross,
  title = {{Cross-lingual Contextualized Topic Models with Zero-shot Learning}},
  author = {Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza and Elisabetta Fersini},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.143},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.143}
}
```

- [← previous Annotating and Modeling Fine-grained Factuality in Summarization](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/)
- [next → Measuring Association Between Labels and Free-Text Rationales](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-measuring-association-between-labels-free/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2020. [Pre-training is a hot topic: Contextualized document embeddings improve topic coherence](https://md.research.elamur.ai/free/en/papers/bianchi-2021-pre-training-is-hot-topic/). arXiv preprint arXiv:2004.03974.
2. David M Blei. 2012. Probabilistic topic models. Communications of the ACM, 55(4):77–84. [find](https://scholar.google.com/scholar?q=David+M+Blei%2E+2012%2E+Probabilistic+topic+models%2E+Communications+of+the+ACM%2C+55%284%29%3A77%E2%80%9384%2E)
3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of Machine Learning Research, 3:993–1022.
4. Jordan L. Boyd-Graber and David M. Blei. 2009. Multilingual topic models for unaligned text. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, UAI 2009, pages 75–82. AUAI Press. [find](https://scholar.google.com/scholar?q=Jordan+L%2E+Boyd%2DGraber+and+David+M%2E+Blei%2E+2009%2E+Multilingual+topic+models+for+unaligned+text%2E+In+Proceedings+of+the+25th+Conference+on+Uncertainty+in+Artificial+Intelligence%2C+UAI+2009%2C+pages+75%E2%80%9382%2E+AUA)
5. Jordan L. Boyd-Graber, David Mimno, and David Newman. 2014. Care and Feeding of Topic Models: Problems, Diagnostics, and Improvements, CRC Handbooks of Modern Statistical Methods. CRC Press. [find](https://scholar.google.com/scholar?q=Jordan+L%2E+Boyd%2DGraber%2C+David+Mimno%2C+and+David+Newman%2E+2014%2E+Care+and+Feeding+of+Topic+Models%3A+Problems%2C+Diagnostics%2C+and+Improvements%2C+CRC+Handbooks+of+Modern+Statistical+Methods%2E+CRC+Press%2E)
6. Stephen Carrow. 2018. Pytorchavitm: Open source avitm implementation in pytorch. Github. [find](https://scholar.google.com/scholar?q=Stephen+Carrow%2E+2018%2E+Pytorchavitm%3A+Open+source+avitm+implementation+in+pytorch%2E+Github%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
8. Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. [Topic modeling in embedding spaces](https://md.research.elamur.ai/free/en/papers/dieng-2020-topic-modeling-embedding-spaces/). Transactions of the Association for Computational Linguistics, 8:439–453.
9. E.D. Gutiérrez, Ekaterina Shutova, Patricia Lichtenstein, Gerard de Melo, and Luca Gilardi. 2016. Detecting cross-cultural differences using a multilingual topic model. Transactions of the Association for Computational Linguistics, 4:47–60. [find](https://scholar.google.com/scholar?q=E%2ED%2E+Gutie%CC%81rrez%2C+Ekaterina+Shutova%2C+Patricia+Lichtenstein%2C+Gerard+de+Melo%2C+and+Luca+Gilardi%2E+2016%2E+Detecting+cross%2Dcultural+differences+using+a+multilingual+topic+model%2E+Transactions+of+the+Associatio)
10. Kilem L Gwet. 2014. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC. [find](https://scholar.google.com/scholar?q=Kilem+L+Gwet%2E+2014%2E+Handbook+of+inter%2Drater+reliability%3A+The+definitive+guide+to+measuring+the+extent+of+agreement+among+raters%2E+Advanced+Analytics%2C+LLC%2E)
11. Shudong Hao and Michael J. Paul. 2018. Learning multilingual topics from incomparable corpora. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, pages 2595–2609. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shudong+Hao+and+Michael+J%2E+Paul%2E+2018%2E+Learning+multilingual+topics+from+incomparable+corpora%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+COLING+2018%2C+pages+2595)
12. Geert Heyman, Ivan Vulic, and Marie-Francine Moens. 2016. C-BiLDA extracting cross-lingual topics from non-parallel texts by distinguishing shared from unshared content. Data Mining and Knowledge Discovery, 30(5):1299–1323. [find](https://scholar.google.com/scholar?q=Geert+Heyman%2C+Ivan+Vulic%2C+and+Marie%2DFrancine+Moens%2E+2016%2E+C%2DBiLDA+extracting+cross%2Dlingual+topics+from+non%2Dparallel+texts+by+distinguishing+shared+from+unshared+content%2E+Data+Mining+and+Knowledge+Disc)
13. Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari. 2020. “You sound just like your father” Commercial machine translation systems include stylistic biases. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1686–1690, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Federico+Bianchi%2C+and+Tommaso+Fornaciari%2E+2020%2E+%E2%80%9CYou+sound+just+like+your+father%E2%80%9D+Commercial+machine+translation+systems+include+stylistic+biases%2E+In+Proceedings+of+the+58th+Annual+Meeting+)
14. Dirk Hovy and Christoph Purschke. 2018. Capturing regional variation with distributed place representations and geographic retrofitting. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4383–4394, Brussels, Belgium. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Dirk+Hovy+and+Christoph+Purschke%2E+2018%2E+Capturing+regional+variation+with+distributed+place+representations+and+geographic+retrofitting%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+N)
15. Jey Han Lau, David Newman, and Timothy Baldwin. 2014. [Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality](https://md.research.elamur.ai/free/en/papers/lau-2014-machine-reading-tea-leaves-automatically/). In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg…
16. Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. 2016. Topic modeling for short texts with auxiliary word embeddings. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016, pa… [find](https://scholar.google.com/scholar?q=Chenliang+Li%2C+Haoran+Wang%2C+Zhiqian+Zhang%2C+Aixin+Sun%2C+and+Zongyang+Ma%2E+2016%2E+Topic+modeling+for+short+texts+with+auxiliary+word+embeddings%2E+In+Proceedings+of+the+39th+International+ACM+SIGIR+conference)
17. Xiaodong Liu, Kevin Duh, and Yuji Matsumoto. 2015. Multilingual topic models for bilingual dictionary extraction. ACM Transactions on Asian Language Information Processing, 14(3):11:1–11:22. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Kevin+Duh%2C+and+Yuji+Matsumoto%2E+2015%2E+Multilingual+topic+models+for+bilingual+dictionary+extraction%2E+ACM+Transactions+on+Asian+Language+Information+Processing%2C+14%283%29%3A11%3A1%E2%80%9311%3A22%2E)
18. Tengfei Ma and Tetsuya Nasukawa. 2017. Inverted bilingual topic models for lexicon extraction from non-parallel data. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI 2017, pages 4075–4081. IJCAI.org. [find](https://scholar.google.com/scholar?q=Tengfei+Ma+and+Tetsuya+Nasukawa%2E+2017%2E+Inverted+bilingual+topic+models+for+lexicon+extraction+from+non%2Dparallel+data%2E+In+Proceedings+of+the+26th+International+Joint+Conference+on+Artificial+Intelligen)
19. David M. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP 2009, pages 880– 889. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+M%2E+Mimno%2C+Hanna+M%2E+Wallach%2C+Jason+Naradowsky%2C+David+A%2E+Smith%2C+and+Andrew+McCallum%2E+2009%2E+Polylingual+topic+models%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+)
20. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making Sense of Language-Specific BERT Models. arXiv preprint arXiv:2003.02912. — [arXiv:2003.02912](https://arxiv.org/abs/2003.02912) [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Federico+Bianchi%2C+and+Dirk+Hovy%2E+2020%2E+What+the+%5BMASK%5D%3F+Making+Sense+of+Language%2DSpecific+BERT+Models%2E+arXiv+preprint+arXiv%3A2003%2E02912%2E+%E2%80%94+%5BarXiv%3A2003%2E02912%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2003%2E02)
21. Debora Nozza, Elisabetta Fersini, and Enza Messina. 2016. Unsupervised irony detection: a probabilistic model with word embeddings. In International Conference on Knowledge Discovery and Information Retrieval, volume 2, pages 68–76. SCITEPRESS. [find](https://scholar.google.com/scholar?q=Debora+Nozza%2C+Elisabetta+Fersini%2C+and+Enza+Messina%2E+2016%2E+Unsupervised+irony+detection%3A+a+probabilistic+model+with+word+embeddings%2E+In+International+Conference+on+Knowledge+Discovery+and+Information+R)
22. Jagadeesh Jagarlamudi and Hal Daumé. 2010. Extracting multilingual topics from unaligned comparable corpora. In Advances in Information Retrieval, pages 444–456, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Jagadeesh+Jagarlamudi+and+Hal+Daume%CC%81%2E+2010%2E+Extracting+multilingual+topics+from+unaligned+comparable+corpora%2E+In+Advances+in+Information+Retrieval%2C+pages+444%E2%80%93456%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+H)
23. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3973–3983.…
24. Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Max+Welling%2E+2014%2E+Autoencoding+variational+bayes%2E+In+Proceedings+of+the+2nd+International+Conference+on+Learning+Representations%2C+ICLR+2014%2E)
25. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). arXiv preprint arXiv:2002.12327.
26. Kriste Krstovski, David A. Smith, and Michael J. Kurtz. 2016. Online multilingual topic models with multi-level hyperpriors. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan-Akash Srivastava and Charles A. Sutton. 2017. A… [find](https://scholar.google.com/scholar?q=Kriste+Krstovski%2C+David+A%2E+Smith%2C+and+Michael+J%2E+Kurtz%2E+2016%2E+Online+multilingual+topic+models+with+multi%2Dlevel+hyperpriors%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+)
27. Silvia Terragni, Elisabetta Fersini, and Enza Messina. 2020a. Constrained relational topic models. Information Sciences, 512:581–594. [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Elisabetta+Fersini%2C+and+Enza+Messina%2E+2020a%2E+Constrained+relational+topic+models%2E+Information+Sciences%2C+512%3A581%E2%80%93594%2E)
28. Silvia Terragni, Debora Nozza, Elisabetta Fersini, and Messina Enza. 2020b. Which matters most? comparing the impact of concept and document relationships in topic models. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 32–40, Online. Association for Computationa… [find](https://scholar.google.com/scholar?q=Silvia+Terragni%2C+Debora+Nozza%2C+Elisabetta+Fersini%2C+and+Messina+Enza%2E+2020b%2E+Which+matters+most%3F+comparing+the+impact+of+concept+and+document+relationships+in+topic+models%2E+In+Proceedings+of+the+First+)

*Updated 24 October 2026*