---
title: "Comparative Analysis of Tokenizers in Tamil Text Classification"
description: "The work offers practical guidance for picking tokenizers for Tamil and related agglutinative languages. 30 references, cited by 1 paper here."
authors: ["Gokulan Sivakumaran", "Randil Pushpananda", "ERAD Bandara"]
year: 2026
venue: "Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)"
source: "https://aclanthology.org/2026.chipsal-1.19/"
doi: "10.63317/5p78kf96x2jw"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sivakumaran-2026-comparative-analysis-tokenizers-tamil-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sivakumaran-2026-comparative-analysis-tokenizers-tamil-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings

# Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings

**Authors:** Gokulan Sivakumaran, Randil Pushpananda and ERAD Bandara · **Year:** 2026 · **Venue:** [Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)](https://md.research.elamur.ai/free/en/venues/chipsal/) ([workshop on Challenges in Processing South Asian Languages 2026](https://md.research.elamur.ai/free/en/venues/chipsal/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.chipsal-1.19/) · [PDF](https://aclanthology.org/2026.chipsal-1.19.pdf) · [DOI 10.63317/5p78kf96x2jw](https://doi.org/10.63317/5p78kf96x2jw) · [OpenAlex](https://openalex.org/W7166539495)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/sivakumaran-2026-comparative-analysis-tokenizers-tamil-text/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Tamil is an agglutinative, low-resource Dravidian language. It is unclear how well common subword tokenizers, which were mostly designed with English-like text in mind, handle its heavy inflection and compounding. The authors run a unified comparison of three widely used schemes: WordPiece, SentencePiece and Byte-Level BPE (BBPE). They measure both intrinsic segmentation quality and the effect on a downstream task. Each tokenizer was trained on a 5.45 GB Tamil corpus from Kaggle, cleaned of metadata, emojis and non-Tamil scripts, while punctuation, digits and embedded English words were kept. The intrinsic assessment examined how many tokens each scheme produced, how heavily words were split, the share of out-of-vocabulary (OOV) items, and how well the text was compressed. For the extrinsic test, a compact custom BERT-style Transformer was fine-tuned to sort Tamil news headlines into categories. WordPiece and SentencePiece outperformed BBPE in both efficiency and classification accuracy. BBPE never yields unknown tokens, but it breaks Tamil words into so many pieces that the model learns poorly. Unlike the other two tokenizers, BBPE also gains nothing from a larger vocabulary. The work offers practical guidance for picking tokenizers for Tamil and related agglutinative languages.

## Key points
- Compared tokenizers: WordPiece, SentencePiece and BBPE.
- Data and setup:
  - Tokenizer training used the Tamil Language Corpus for NLP (Praveen Govi, 2020; Kaggle; 5.45 GB).
  - Classification was tested on the Kaggle Tamil News Classification Dataset (Tamil Murasu; Vijaya Bhaskar, 2020).
  - Experiments used a lightweight BERT-based model built with PyTorch and Hugging Face Transformers on an HPC cluster.
- Intrinsic measures included the number of tokens generated, the degree of word splitting, the out-of-vocabulary (OOV) rate, and the compression achieved. BBPE achieves a 0% OOV rate by construction but over-fragments Tamil, since each Tamil character takes several bytes in UTF-8.
- Enlarging the vocabulary improved WordPiece and SentencePiece but not BBPE. Error analysis linked misclassifications to over-fragmentation.
- Code and trained models are released at github.com/Gokul200089/Tokenizers. The paper appeared at CHiPSAL2026 @ LREC, pp. 198–208.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works · [🌱 Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) · 267 works

## Cite

Cite as: Gokulan Sivakumaran, Randil Pushpananda and ERAD Bandara. 2026. Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings. Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sivakumaran2026comparative,
  title = {{Comparative Analysis of Tokenizers in Tamil Text Classification in Low Resource Settings}},
  author = {Gokulan Sivakumaran, Randil Pushpananda and ERAD Bandara},
  booktitle = {Proceedings of the Second workshop on Challenges in Processing South Asian Languages (CHiPSAL2026)},
  year = {2026},
  doi = {10.63317/5p78kf96x2jw},
  url = {https://doi.org/10.63317/5p78kf96x2jw}
}
```

- [← previous Compact Multimodal Language Models as Robust OCR Alternatives for Noisy Textual Clinical Reports](https://md.research.elamur.ai/free/en/papers/neveditsin-2026-compact-multimodal-language-models-as/)
- [next → COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Context](https://md.research.elamur.ai/free/en/papers/wan-2026-compass-enhancing-agent-long-horizon/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali et al. 2021. Arabic pre-trained language models. In Proceedings of the Sixth Arabic NLP Workshop. [find](https://scholar.google.com/scholar?q=Ahmed+Abdelali+et+al%2E+2021%2E+Arabic+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+Sixth+Arabic+NLP+Workshop%2E)
2. Muhammad Abdul-Mageed et al. 2020. Camelbert: Surprising cross-lingual transfer results. In Proceedings of EMNLP 2020. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed+et+al%2E+2020%2E+Camelbert%3A+Surprising+cross%2Dlingual+transfer+results%2E+In+Proceedings+of+EMNLP+2020%2E)
3. E. Annamalai and R. E. Asher. 2002. Colloquial Tamil: The Complete Course for Beginners. Routledge. [find](https://scholar.google.com/scholar?q=E%2E+Annamalai+and+R%2E+E%2E+Asher%2E+2002%2E+Colloquial+Tamil%3A+The+Complete+Course+for+Beginners%2E+Routledge%2E)
4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the Fourth Workshop on Open-Source Arabic Corpora and Processing Tools.
5. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pre-training. In Findings of EMNLP 2020, pages 4145–4155. [find](https://scholar.google.com/scholar?q=Kaj+Bostrom+and+Greg+Durrett%2E+2020%2E+Byte+pair+encoding+is+suboptimal+for+language+model+pre%2Dtraining%2E+In+Findings+of+EMNLP+2020%2C+pages+4145%E2%80%934155%2E)
6. Guy Dagan et al. 2024. Tokenization from a compression and semantic trade-off perspective. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guy+Dagan+et+al%2E+2024%2E+Tokenization+from+a+compression+and+semantic+trade%2Doff+perspective%2E+Transactions+of+the+Association+for+Computational+Linguistics%2E)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT, pages 4171–4186.
8. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2020. Optimizing word segmentation for downstream task. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1341–1351, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tatsuya+Hiraoka%2C+Sho+Takase%2C+Kei+Uchiumi%2C+Atsushi+Keyaki%2C+and+Naoaki+Okazaki%2E+2020%2E+Optimizing+word+segmentation+for+downstream+task%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNL)
9. Taku Kudo. 2018. Subword regularization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 66–75. [find](https://scholar.google.com/scholar?q=Taku+Kudo%2E+2018%2E+Subword+regularization%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+66%E2%80%9375%2E)
10. Taku Kudo and John Richardson. 2018a. Sentencepiece: A simple and language independent subword tokenizer. In Proceedings of EMNLP 2018: System Demonstrations, pages 66–71. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018a%2E+Sentencepiece%3A+A+simple+and+language+independent+subword+tokenizer%2E+In+Proceedings+of+EMNLP+2018%3A+System+Demonstrations%2C+pages+66%E2%80%9371%2E)
11. Taku Kudo and John Richardson. 2018b. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66– 71, Brussels, Belgium. Asso…
12. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, et al. 2020. [Albert: A lite bert for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In International Conference on Learning Representations.
13. Yinhan Liu, Myle Ott, Naman Goyal, et al. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
14. Harish Tayyar Madabushi, Elena Kochkina, and Mathieu Castelle. 2019. Cost-sensitive bert for generalisable sentence classification on imbalanced data. In Proceedings of the Second Workshop on NLP for Internet Freedom, pages 125– 134. [find](https://scholar.google.com/scholar?q=Harish+Tayyar+Madabushi%2C+Elena+Kochkina%2C+and+Mathieu+Castelle%2E+2019%2E+Cost%2Dsensitive+bert+for+generalisable+sentence+classification+on+imbalanced+data%2E+In+Proceedings+of+the+Second+Workshop+on+NLP+for+)
15. Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Language+Processing+%28Volume+1%3A+Long+Papers%29%2C+pages+3118%E2%80%933135%2C+Online%2E+Association+for+Computational+Linguistics%2E)
16. Phillip Rust, Jonas Pfeiffer, Ivan Vulic, et al. 2020. How good is your tokenizer? arXiv preprint arXiv:2012.15613. — [arXiv:2012.15613](https://arxiv.org/abs/2012.15613) [find](https://scholar.google.com/scholar?q=Phillip+Rust%2C+Jonas+Pfeiffer%2C+Ivan+Vulic%2C+et+al%2E+2020%2E+How+good+is+your+tokenizer%3F+arXiv+preprint+arXiv%3A2012%2E15613%2E+%E2%80%94+%5BarXiv%3A2012%2E15613%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E15613%29)
17. K. Sarveswaran, G. Dias, and M. Butt. 2021. Thamizhi morph: A morphological parser for the tamil language. Machine Translation, 35(1):37– 70. [find](https://scholar.google.com/scholar?q=K%2E+Sarveswaran%2C+G%2E+Dias%2C+and+M%2E+Butt%2E+2021%2E+Thamizhi+morph%3A+A+morphological+parser+for+the+tamil+language%2E+Machine+Translation%2C+35%281%29%3A37%E2%80%93+70%2E)
18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1715–1725.
19. Harshit Singh and Rajiv Ratn Shah. 2022. Tamilnlp: Low resource language processing. [find](https://scholar.google.com/scholar?q=Harshit+Singh+and+Rajiv+Ratn+Shah%2E+2022%2E+Tamilnlp%3A+Low+resource+language+processing%2E)
20. M. Thangarasu and R. Manavalan. 2013. Stemmers for tamil language: Performance analysis. International Journal of Computer Trends and Technology, 4(8):2582–2584. [find](https://scholar.google.com/scholar?q=M%2E+Thangarasu+and+R%2E+Manavalan%2E+2013%2E+Stemmers+for+tamil+language%3A+Performance+analysis%2E+International+Journal+of+Computer+Trends+and+Technology%2C+4%288%29%3A2582%E2%80%932584%2E)
21. Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30.
22. K. T. Veeramanju. 2023. A systematic review on news headlines categorization in malayalam. Scope, 13. [find](https://scholar.google.com/scholar?q=K%2E+T%2E+Veeramanju%2E+2023%2E+A+systematic+review+on+news+headlines+categorization+in+malayalam%2E+Scope%2C+13%2E)
23. Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference o… [find](https://scholar.google.com/scholar?q=Kyubyong+Park%2C+Joohong+Lee%2C+Seongbo+Jang%2C+and+Dawoon+Jung%2E+2020%2E+An+empirical+study+of+tokenization+strategies+for+various+korean+nlp+tasks%2E+In+Proceedings+of+the+1st+Conference+of+the+Asia%2DPacific+Ch)
24. Jia Wei, Qiang Liu, Yiqun Guo, and Xia Jiang. 2021. Training multilingual pre-trained language models with byte-level subwords. arXiv preprint arXiv:2101.09469. — [arXiv:2101.09469](https://arxiv.org/abs/2101.09469) [find](https://scholar.google.com/scholar?q=Jia+Wei%2C+Qiang+Liu%2C+Yiqun+Guo%2C+and+Xia+Jiang%2E+2021%2E+Training+multilingual+pre%2Dtrained+language+models+with+byte%2Dlevel+subwords%2E+arXiv+preprint+arXiv%3A2101%2E09469%2E+%E2%80%94+%5BarXiv%3A2101%2E09469%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
25. Muhammad Qarah and Ibrahim Alsanoosay. 2024. A comparative study on tokenizers for arabic nlp tasks. In Proceedings of the Conference on Arabic Natural Language Processing. [find](https://scholar.google.com/scholar?q=Muhammad+Qarah+and+Ibrahim+Alsanoosay%2E+2024%2E+A+comparative+study+on+tokenizers+for+arabic+nlp+tasks%2E+In+Proceedings+of+the+Conference+on+Arabic+Natural+Language+Processing%2E)
26. Yonghui Wu, Mike Schuster, Zhifeng Chen, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+the+gap+between+human+and+machine+translation%2E+arXiv+preprint+arXiv%3A1609%2E08144%2E)
27. Alec Radford, Jeffrey Wu, Rewon Child, et al. 2019. Language models are unsupervised multitask learners. OpenAI Technical Report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Technical+Report%2E)
28. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. [How good is your tokenizer? on the monolingual performance of multilingual language models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
29. Language Resource References Praveen Govi. 2020. Tamil Language Corpus for NLP. Kaggle Datasets. Kaggle. A large collection of Tamil text corpora for natural language processing research. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Praveen+Govi%2E+2020%2E+Tamil+Language+Corpus+for+NLP%2E+Kaggle+Datasets%2E+Kaggle%2E+A+large+collection+of+Tamil+text+corpora+for+natural+language+processing+research%2E)
30. Vijaya Bhaskar. 2020. Tamil News Classification Dataset (Tamil Murasu). Kaggle Datasets. Kaggle. News articles collected from the Tamil Murasu newspaper and released for text classification research. [find](https://scholar.google.com/scholar?q=Vijaya+Bhaskar%2E+2020%2E+Tamil+News+Classification+Dataset+%28Tamil+Murasu%29%2E+Kaggle+Datasets%2E+Kaggle%2E+News+articles+collected+from+the+Tamil+Murasu+newspaper+and+released+for+text+classification+research%2E)

*Updated 31 October 2026*