---
title: "Benchmarking Text Embedding Models for South African"
description: "The work releases resources and gives practical guidance for model choice in under-resourced, morphologically rich… 0.60 F1. 39 references."
authors: ["Ockert de Villiers", "Roald Eiselen"]
year: 2026
venue: "Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026"
source: "https://aclanthology.org/2026.rail-1.5/"
doi: "10.63317/28nicfxsyx92"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/villiers-2026-benchmarking-text-embedding-models-south/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/villiers-2026-benchmarking-text-embedding-models-south/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Benchmarking Text Embedding Models for South African Languages

# Benchmarking Text Embedding Models for South African Languages

**Authors:** Ockert de Villiers and Roald Eiselen · **Year:** 2026 · **Venue:** [Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026](https://md.research.elamur.ai/free/en/venues/rail/) ([Resources for African Indigenous Languages (RAIL) @ LREC 2026](https://md.research.elamur.ai/free/en/venues/rail/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.rail-1.5/) · [PDF](https://aclanthology.org/2026.rail-1.5.pdf) · [DOI 10.63317/28nicfxsyx92](https://doi.org/10.63317/28nicfxsyx92) · [OpenAlex](https://openalex.org/W7166550645)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses a practical gap for South African languages: few monolingual embedding models exist, and it is unclear which embedding type works best under low-resource conditions. The authors built monolingual training corpora for ten languages. Sources included OPUS, the Leipzig Corpora Collection, CommonCrawl, NCHLT, Autshumato and licensed internal texts. The data was deduplicated, filtered with the NCHLT South African Language Identifier and cleaned, with casing deliberately kept. From these corpora they trained six embedding models in four architectures: GloVe, fastText (CBOW and skip-gram), Flair (forward and backward) and RoBERTa. XLM-R was added as a multilingual comparison. Each embedding was used as input to an otherwise identical BiLSTM-CRF tagger for POS tagging and NER, so differences reflect the representations alone. Results were analysed by orthographic type: conjunctive Nguni languages, disjunctive Sotho-Tswana and Tswa-Ronga languages, and Afrikaans. Flair performed best and most consistently, especially for conjunctive languages. fastText was a cheap, strong runner-up. GloVe and XLM-R trailed. The work releases resources and gives practical guidance for model choice in under-resourced, morphologically rich settings.

## Key points
- Language groups were conjunctive (isiNdebele, isiXhosa, isiZulu, Siswati), disjunctive (Sepedi, Sesotho, Setswana, Tshivenḓa, Xitsonga) and Afrikaans. The authors also evaluated on MasakhaNER/MasakhaPOS data for Setswana, isiXhosa and isiZulu.
- UPOS micro-F1 peaks at 0.9878 for Afrikaans with Flair-forward. GloVe drops sharply on conjunctive languages, scoring 0.8313 on Siswati against 0.9390 for fastText-CBOW, and 0.8646 on isiXhosa against 0.9733 for Flair-backward.
- NER is harder than POS tagging. Sesotho stays around 0.60 F1 for all models. XLM-R is usually weakest, for example 0.6250 on Sepedi against 0.8199 for Flair-backward.
- The monolingual RoBERTa models are small: 6 layers, 6 attention heads, hidden size 768 and a 30k vocabulary. They are competitive on most languages but underperform on Afrikaans POS (0.9349).
- Siswati has a sentence count similar to Sesotho's but fewer than half as many tokens. This illustrates how conjunctive orthography inflates vocabulary size and weakens word-level vectors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [North-West University](https://md.research.elamur.ai/free/en/universities/north-west-university/) — University, Potchefstroom, South Africa · 6 works · 5 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Ockert de Villiers and Roald Eiselen. 2026. Benchmarking Text Embedding Models for South African Languages. Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{villiers2026benchmarking,
  title = {{Benchmarking Text Embedding Models for South African Languages}},
  author = {Ockert {de Villiers} and Roald Eiselen},
  booktitle = {Proceedings of Resources for African Indigenous Languages (RAIL) 2026 @ LREC 2026},
  year = {2026},
  doi = {10.63317/28nicfxsyx92},
  url = {https://doi.org/10.63317/28nicfxsyx92}
}
```

- [← previous Benchmarking Temporal Reasoning and Alignment Across Chinese Dynasties](https://md.research.elamur.ai/free/en/papers/wang-2026-benchmarking-temporal-reasoning-alignment-across/)
- [next → Benchmarking the Fine-Grained Discriminability in Image-Text Retrieval via Controlled Contrastive Differences](https://md.research.elamur.ai/free/en/papers/wang-2026-benchmarking-fine-grained-discriminability-image/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’Souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. [Masakhaner: Named entity recognition for african languages](https://md.research.elamur.ai/free/en/papers/adelani-2021-masakhaner-named-entity-recognition-african/). Transactions of the Association for Computational Li…
2. Etienne Barnard, Marelie H. Davel, Charl van Heerden, Febe de Wet, and Jaco Badenhorst. 2014. The NCHLT speech corpus of the south african languages. In Proceedings of the 4th Workshop on Spoken Language Technologies for Under-resourced Languages (SLTU), St. Petersburg, Russia. ISCA. [find](https://scholar.google.com/scholar?q=Etienne+Barnard%2C+Marelie+H%2E+Davel%2C+Charl+van+Heerden%2C+Febe+de+Wet%2C+and+Jaco+Badenhorst%2E+2014%2E+The+NCHLT+speech+corpus+of+the+south+african+languages%2E+In+Proceedings+of+the+4th+Workshop+on+Spoken+Langu)
3. David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O. Alabi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and… — [A Neural Probabilistic Language Model](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/)
4. Astik Biswas, Emre Yılmaz, Ewald van der Westhuizen, Febe de Wet, and Thomas Niesler. 2022. Code-switched automatic speech recognition in five south african languages. Computer Speech & Language, 71:101262. [find](https://scholar.google.com/scholar?q=Astik+Biswas%2C+Emre+Y%C4%B1lmaz%2C+Ewald+van+der+Westhuizen%2C+Febe+de+Wet%2C+and+Thomas+Niesler%2E+2022%2E+Code%2Dswitched+automatic+speech+recognition+in+five+south+african+languages%2E+Computer+Speech+%26+Language%2C+71%3A1)
5. J. du Toit and M. Puttkammer. 2021. Neural approaches to core technologies for nguni languages. Information, 12(7):276. [find](https://scholar.google.com/scholar?q=J%2E+du+Toit+and+M%2E+Puttkammer%2E+2021%2E+Neural+approaches+to+core+technologies+for+nguni+languages%2E+Information%2C+12%287%29%3A276%2E)
6. Roald Eiselen. 2016. Government domain named entity recognition for South African languages. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3344–3348, Portorož, Slovenia. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Roald+Eiselen%2E+2016%2E+Government+domain+named+entity+recognition+for+South+African+languages%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916%29%2C+pages+)
7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146.
8. Roald Eiselen and Martin J. Puttkammer. 2014. Developing text resources for ten South African languages. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014), pages 3698– 3703, Reykjavik, Iceland. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Roald+Eiselen+and+Martin+J%2E+Puttkammer%2E+2014%2E+Developing+text+resources+for+ten+South+African+languages%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In Proceedings of the 58th Annual Meeting of the Association fo…
10. Tanja Gaustad and Cindy McKellar. 2024. Updated morphologically annotated corpora for 9 South African languages. Journal of Open Humanities Data, 10(38):1–5. [find](https://scholar.google.com/scholar?q=Tanja+Gaustad+and+Cindy+McKellar%2E+2024%2E+Updated+morphologically+annotated+corpora+for+9+South+African+languages%2E+Journal+of+Open+Humanities+Data%2C+10%2838%29%3A1%E2%80%935%2E)
11. Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. [Universal Dependencies](https://md.research.elamur.ai/free/en/papers/marneffe-2021-universal-dependencies-2/). Computational Linguistics, 47(2):255–308.
12. Tanja Gaustad, Cindy McKellar, and Martin Puttkammer. 2024a. Dataset for Siswati: Parallel textual data for English and Siswati and monolingual textual data for Siswati. Data in Brief, 54. [find](https://scholar.google.com/scholar?q=Tanja+Gaustad%2C+Cindy+McKellar%2C+and+Martin+Puttkammer%2E+2024a%2E+Dataset+for+Siswati%3A+Parallel+textual+data+for+English+and+Siswati+and+monolingual+textual+data+for+Siswati%2E+Data+in+Brief%2C+54%2E)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT 2019, pages 4171–4186.
14. Tanja Gaustad, Cindy A. McKellar, and Martin J. Puttkammer. 2024b. Machine translation training data for English–Tshiven�a. Data in Brief, 57. [find](https://scholar.google.com/scholar?q=Tanja+Gaustad%2C+Cindy+A%2E+McKellar%2C+and+Martin+J%2E+Puttkammer%2E+2024b%2E+Machine+translation+training+data+for+English%E2%80%93Tshiven%EF%BF%BDa%2E+Data+in+Brief%2C+57%2E)
15. Tanja Gaustad and Martin J. Puttkammer. 2022. Linguistically annotated dataset for four official South African languages with a conjunctive orthography: isiNdebele, isiXhosa, isiZulu, and Siswati. Data in Brief, 41. [find](https://scholar.google.com/scholar?q=Tanja+Gaustad+and+Martin+J%2E+Puttkammer%2E+2022%2E+Linguistically+annotated+dataset+for+four+official+South+African+languages+with+a+conjunctive+orthography%3A+isiNdebele%2C+isiXhosa%2C+isiZulu%2C+and+Siswati%2E+Dat)
16. Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Buku… [find](https://scholar.google.com/scholar?q=Cheikh+M%2E+Bamba+Dione%2C+David+Ifeoluwa+Adelani%2C+Peter+Nabende%2C+Jesujoba+Alabi%2C+Thapelo+Sindane%2C+Happy+Buzaaba%2C+Shamsuddeen+Hassan+Muhammad%2C+Chris+Chinenye+Emezue%2C+Perez+Ogayo%2C+Anuoluwapo+Aremu%2C+Catheri)
17. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. [Building large monolingual dictionaries at the Leipzig corpora collection: From 100 to 200 languages](https://md.research.elamur.ai/free/en/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/). In Proceedings of the 8th International Language Resources and Evaluation (LREC’12), pages 759–765, Istanbul, Turkey. European Language Resourc…
18. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. [Learning word vectors for 157 languages](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/). In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
19. Hendrik Johannes Groenewald and Wildrich Fourie. 2009. Introducing the autshumato integrated translation environment. In Proceedings of the 13th Annual Conference of the European Association for Machine Translation (EAMT), Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Hendrik+Johannes+Groenewald+and+Wildrich+Fourie%2E+2009%2E+Introducing+the+autshumato+integrated+translation+environment%2E+In+Proceedings+of+the+13th+Annual+Conference+of+the+European+Association+for+Machi)
20. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, and Claytone Sikasote. 2022. [Quality at a glance: An audit of web-crawled multilingual datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/). Transactions of the Association for Computational Linguis…
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. \#\# 10.
22. Language Resource References N.C.P. Golele and X.E. Mabaso and Roald Eiselen. 2016. NCHLT Xitsonga Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADiLaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/362](https://hdl.handle.net/20.500.12185/362). [↗](https://hdl.handle.net/20.500.12185/362)
23. Melinda Loubser and Martin J. Puttkammer. 2020. Viability of neural networks for core technologies for resource-scarce languages. Information, 11(1):41. [find](https://scholar.google.com/scholar?q=Melinda+Loubser+and+Martin+J%2E+Puttkammer%2E+2020%2E+Viability+of+neural+networks+for+core+technologies+for+resource%2Dscarce+languages%2E+Information%2C+11%281%29%3A41%2E)
24. K.S. Mahlangu and Roald Eiselen. 2016. NCHLT isiNdebele Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/306](https://hdl.handle.net/20.500.12185/306). [↗](https://hdl.handle.net/20.500.12185/306)
25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781.
26. B.B. Malangwane and M.N. Kekana and S.S. Sedibe and B.C. Ndhlovu and Roald Eiselen. 2016. NCHLT Siswati Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADiLaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/346](https://hdl.handle.net/20.500.12185/346). [↗](https://hdl.handle.net/20.500.12185/346)
27. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeffrey Dean. 2013b. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems (NeurIPS 2013), volume 26.
28. A.N. Manzini and Roald Eiselen. 2016. NCHLT isiZulu Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/319](https://hdl.handle.net/20.500.12185/319). [↗](https://hdl.handle.net/20.500.12185/319)
29. Visvaganthie Moodley. 2007. Codeswitching in the multilingual english first language classroom. International Journal of Bilingual Education and Bilingualism, 10(6):707–722. [find](https://scholar.google.com/scholar?q=Visvaganthie+Moodley%2E+2007%2E+Codeswitching+in+the+multilingual+english+first+language+classroom%2E+International+Journal+of+Bilingual+Education+and+Bilingualism%2C+10%286%29%3A707%E2%80%93722%2E)
30. S.S.B.M. Phakedi and Roald Eiselen. 2016. NCHLT Setswana Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/341](https://hdl.handle.net/20.500.12185/341). [↗](https://hdl.handle.net/20.500.12185/341)
31. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. [Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages](https://md.research.elamur.ai/free/en/papers/ogueji-2021-small-data-no-problem-exploring/). In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126.
32. K. Podile and Roald Eiselen. 2016. NCHLT isiXhosa Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/312](https://hdl.handle.net/20.500.12185/312). [↗](https://hdl.handle.net/20.500.12185/312)
33. Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, et al. 2020. Masakhane: Machine translation for africa. arXiv preprint arXiv:2003.11529. — [arXiv:2003.11529](https://arxiv.org/abs/2003.11529) [find](https://scholar.google.com/scholar?q=Iroro+Orife%2C+Julia+Kreutzer%2C+Blessing+Sibanda%2C+Daniel+Whitenack%2C+Kathleen+Siminyu%2C+Laura+Martinus%2C+Jamiil+Toure+Ali%2C+Jade+Abbott%2C+Vukosi+Marivate%2C+Salomon+Kabongo%2C+et+al%2E+2020%2E+Masakhane%3A+Machine+tran)
34. D.J. Prinsloo and Roald Eiselen. 2016. NCHLT Sepedi Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/328](https://hdl.handle.net/20.500.12185/328). [↗](https://hdl.handle.net/20.500.12185/328)
35. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. ACL.
36. Martin Puttkammer and Tanja Gaustad. 2021. Linguistically enriched corpora for conjunctively written South African languages. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/546](https://hdl.handle.net/20.500.12185/546). [↗](https://hdl.handle.net/20.500.12185/546)
37. Raivis Skadiņš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. 2014. Billions of parallel Martin Puttkammer and Justin Hocking and Roald Eiselen. 2016. NCHLT South African Language Identifier. Centre for Text Technology (CTexT). SADiLaR Language Resource Repository. PID [https://hdl.handle.net/20.5…](https://hdl.handle.net/20.5…) [↗](https://hdl.handle.net/20.5…)
38. Gerhard van Huyssteen and Martin Puttkammer and E.B. Trollip and J.C. Liversage and Roald Eiselen. 2016. NCHLT Afrikaans Named Entity Annotated Corpus. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/299](https://hdl.handle.net/20.500.12185/299). [↗](https://hdl.handle.net/20.500.12185/299)
39. Martin Puttkammer and Martin Schlemmer and Ruan Bekker. 2014a. NCHLT Afrikaans Annotated Text Corpora. Centre for Text Technology (CTexT). SADi-LaR Language Resource Repository. PID [https://hdl.handle.net/20.500.12185/296](https://hdl.handle.net/20.500.12185/296). [↗](https://hdl.handle.net/20.500.12185/296)

*Updated 2 November 2026*