---
title: "Dotless Arabic Text for Natural Language Processing (2025)"
description: "The authors test whether modern NLP models can handle Arabic with every dot removed, much as… 70 references, cited by 1 paper here."
authors: ["Maged S. Al-Shaibani", "Irfan Ahmad"]
year: 2025
venue: "Computational Linguistics"
source: "https://aclanthology.org/2025.cl-2.5/"
doi: "10.1162/coli_a_00535"
licence: "CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shaibani-2025-dotless-arabic-text-natural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shaibani-2025-dotless-arabic-text-natural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/) / Dotless Arabic Text for Natural Language Processing

# Dotless Arabic Text for Natural Language Processing

**Authors:** [Maged S. Al-Shaibani](https://md.research.elamur.ai/free/en/authors/maged-s-al-shaibani/) and [Irfan Ahmad](https://md.research.elamur.ai/free/en/authors/irfan-ahmad/) · **Year:** 2025 · **Venue:** [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) ([Computational Linguistics 2025](https://md.research.elamur.ai/free/en/venues/cl/2025/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY-NC-ND 4.0 (Computational Linguistics, MIT Press)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.cl-2.5/) · [PDF](https://aclanthology.org/2025.cl-2.5.pdf) · [DOI 10.1162/coli\_a\_00535](https://doi.org/10.1162/coli_a_00535) · [OpenAlex](https://openalex.org/W4402475606)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/shaibani-2025-dotless-arabic-text-natural-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Many Arabic letters share one base shape (rasm) and are told apart only by dots, which early Arabic writing often left out. The authors test whether modern NLP models can handle Arabic with every dot removed, much as native readers still understand undotted script from context. They convert text to its dotless form and compare it with standard text on five corpora of varying size and domain, at word, subword and character granularity. Statistical checks (Zipf's and Heap's laws, entropy, and a comparison with English via parallel corpora) are followed by n-gram, RNN and transformer language models and seven downstream tasks: sentiment, news topic and poetry meter classification, POS tagging, NER, and translation in both directions between Arabic and English. A dot-restoration system lets generative tasks still output normal Arabic. Across tasks the two representations score closely, with neither consistently ahead, while the dotless vocabulary shrinks by up to half. This makes the undotted script a compact, cheaper input option, also relevant to ancient manuscripts and to social-media posts written undotted to evade filters.

## Key points
- Dotless text was compared with dotted text on five corpora (including the Quran) under word, Farasa subword, disjoint-letters and SentencePiece tokenizations.
- Dotless text follows Zipf/Heap patterns similar to dotted text, and its entropy drops only slightly, so little information is lost.
- Vocabulary size falls at every tokenization level, by up to roughly 50%, which also yields smaller neural language models with similar perplexity trends.
- English→Arabic BLEU: with word tokens, dotted scored 12.57, undotted 13.55 and undotted plus restoration 12.48. With SentencePiece the same three scored 17.38, 17.35 and 16.01.
- Arabic→English BLEU: with word tokens, dotted scored 25.85 and dotless 27.10. With SentencePiece, dotted scored 31.48 and dotless 29.77, so finer segmentation appears more sensitive to the missing dots.

## Interesting ideas
- Deliberately collapsing the script into a lossy, coarser form shrinks the vocabulary, and context-aware models can often recover the lost distinctions themselves.
- The lossy encoding is paired with an explicit restoration model, and generative output is scored after restoration, so the evaluation reflects the full round trip.
- Cheap corpus statistics (Zipf, Heap, entropy) are run before any training to check that a new representation behaves like real language.
- The same comparison is repeated across several tokenization granularities, which shows the effect can flip depending on segmentation, as the MT results did.
- A parallel English corpus serves as a reference point for vocabulary growth, making a single-language statistic interpretable.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): An English analogue, such as a consonant skeleton or a case- and spelling-folded key, could act as a backoff lookup channel for rare or misspelled words in the tagger and lemmatizer.
- **I2** (back): The text → graph → text compressor could likewise be judged on the restored surface text, not only on the compact intermediate form.
- **I3** (learn): Such statistics could serve as a quick sanity gate on silver data or a proposed normalization before spending GPU time on gparse training.
- **I4** (learn): Evaluations of any input change should be reported per tokenization or segmentation setting, so that a gain at one level is not mistaken for a general win.
- **I5** (world): Comparing entity and lemma inventory growth on parallel fables or tales could show where Mova's compiled ontology coverage lags behind actual text diversity.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [King Fahd University of Petroleum and Minerals](https://md.research.elamur.ai/free/en/universities/king-fahd-university-of-petroleum-and-minerals/) — University, Dhahran, Saudi Arabia · 54 works · 4,666 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works

## Cite

Cite as: Maged S. Al-Shaibani and Irfan Ahmad. 2025. Dotless Arabic Text for Natural Language Processing. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{alshaibani2025dotless,
  title = {{Dotless Arabic Text for Natural Language Processing}},
  author = {Maged S. Al-Shaibani and Irfan Ahmad},
  journal = {Computational Linguistics},
  year = {2025},
  doi = {10.1162/coli\_a\_00535},
  url = {https://doi.org/10.1162/coli_a_00535}
}
```

- [← previous Disagreements in analyses of rhetorical text structure: A new dataset and first analyses](https://md.research.elamur.ai/free/en/papers/hewett-2025-disagreements-analyses-rhetorical-text-structure/)
- [next → Emotions, Morality, and Cognition: The Unprecedented Scope of the Concept of the Heart in Amharic](https://md.research.elamur.ai/free/en/papers/kacprzak-2025-emotions-morality-cognition-unprecedented-scope/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/13/)

## References

70 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. hplt-project/sacremoses: Python port of Moses tokenizer, truecaser and normalizer. [https://github.com/hplt-project/sacremoses](https://github.com/hplt-project/sacremoses). (Accessed on 04/12/2024). [↗](https://github.com/hplt-project/sacremoses)
2. Abandah, Gheith A., Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of Arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18:183–197. https://doi.org/10.1007/s10032-015-0242-2 [↗](https://doi.org/10.1007/s10032-015-0242-2)
3. Abdelali, Ahmed, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. [Farasa: A fast and furious segmenter for Arabic](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16. https://doi.org/10.18653/v1/N16-3003 [↗](https://doi.org/10.18653/v1/n16-3003)
4. Abushariah, Mohammad A. M., Raja N. Ainon, Roziati Zainuddin, Moustafa Elshafei, and Othman O. Khalifa. 2010. Natural speaker-independent Arabic speech recognition system based on hidden Markov models using sphinx tools. In International Conference on Computer and Communication Engineering (ICCCE’10), pages 1–6. https://doi.org/10.1109/ICCCE.2010.5556829 [↗](https://doi.org/10.1109/ICCCE.2010.5556829)
5. Al-Kadi, Ibrahim A. 1998. Study of information-theoretic properties of Arabic based on word entropy and Zipf’s law. Journal of King Saud University-Computer and Information Sciences, 10:1–14. https://doi.org/10.1016/S1319 -1578(98)80001-3 [↗](https://doi.org/10.1016/s1319)
6. Al-Shaibani, Maged and Irfan Ahmad. 2023. [Consonant is all you need: A compact representation of English text for efficient NLP](https://md.research.elamur.ai/free/en/papers/shaibani-2023-consonant-is-all-you-need/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11578–11588. https://doi.org/10.18653/v1/2023.findings-emnlp.775 [↗](https://doi.org/10.18653/v1/2023.findings-emnlp.775)
7. Al-Shaibani, Maged S., Zaid Alyafeai, and Irfan Ahmad. 2020. Meter classification of Arabic poems using deep bidirectional recurrent neural networks. Pattern Recognition Letters, 136:1–7. https://doi.org/10.1016/j.patrec.2020.05.028 [↗](https://doi.org/10.1016/j.patrec.2020.05.028)
8. Alajrami, Ahmed, Katerina Margatina, and Nikolaos Aletras. 2023. Understanding the role of input token characters in language models: How does information loss affect performance? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9085–9108. https://doi.org/10.18653/v1/2023.emnlp-main.563 [↗](https://aclanthology.org/2023.emnlp-main.563/)
9. Alarifi, Abdulrahman, Mansour Alghamdi, Mohammad Zarour, Batoul Aloqail, Heelah Alraqibah, Kholood Alsadhan, and Lamia Alkwai. 2012. Estimating the size of Arabic indexed web content. Scientific Research and Essays, 7(28):2472–2483. https://doi.org/10.5897/SRE11.1708 [↗](https://doi.org/10.5897/sre11.1708)
10. Alhathloul, Zainab and Irfan Ahmad. 2022. Automatic dottization of Arabic text (Rasms) using deep recurrent neural networks. Pattern Recognition Letters, 162:47–55. https://doi.org/10.1016/j.patrec.2022.09.001 [↗](https://doi.org/10.1016/j.patrec.2022.09.001)
11. Almeman, Khalid and Mark Lee. 2013. Automatic building of Arabic multi dialect text corpora by bootstrapping dialect words. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6. https://doi.org/10.1109/ICCSPA.2013.6487247 [↗](https://doi.org/10.1109/ICCSPA.2013.6487247)
12. Alotaiby, Fahad, Ibrahim Alkharashi, and Salah Foda. 2009. Processing large Arabic text corpora: Preliminary analysis and results. In Proceedings of the Second International Conference on Arabic Language Resources and Tools, pages 78–82. [find](https://scholar.google.com/scholar?q=Alotaiby%2C+Fahad%2C+Ibrahim+Alkharashi%2C+and+Salah+Foda%2E+2009%2E+Processing+large+Arabic+text+corpora%3A+Preliminary+analysis+and+results%2E+In+Proceedings+of+the+Second+International+Conference+on+Arabic+Langu)
13. Alotaiby, Fahad, Salah Foda, and Ibrahim Alkharashi. 2014. Arabic vs. English: Comparative statistical study. Arabian Journal for Science and Engineering, 39:809–820. https://doi.org/10.1007/s13369-013-0665-3 [↗](https://doi.org/10.1007/s13369-013-0665-3)
14. Aloufi, Khalid. 2019. Quran dataset. [find](https://scholar.google.com/scholar?q=Aloufi%2C+Khalid%2E+2019%2E+Quran+dataset%2E)
15. Aly, Mohamed and Amir Atiya. 2013. [LABR: A large scale Arabic book reviews dataset](https://md.research.elamur.ai/free/en/papers/aly-2013-labr-large-scale-arabic-book/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498. [↗](https://doi.org/10.13140/2.1.3960.5761)
16. Alyafeai, Zaid and Maged Al-Shaibani. 2020. [ARBML: Democritizing Arabic natural language processing tools](https://md.research.elamur.ai/free/en/papers/alyafeai-2020-arbml-democratizing-arabic-natural-language/). In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 8–13. https://doi.org/10.18653/v1/2020.nlposs-1.2 [↗](https://doi.org/10.18653/v1/2020.nlposs-1.2)
17. Alyafeai, Zaid, Maged S. Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2023. [Evaluating various tokenizers for Arabic text classification](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/). Neural Processing Letters, 55(3):2911–2933. https://doi.org/10.1007/s11063-022-10990-8 [↗](https://doi.org/10.1007/s11063-022-10990-8)
18. Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. [wav2vec 2.0: A framework for self-supervised learning of speech representations](https://md.research.elamur.ai/free/en/papers/baevski-2020-wav2vec-2-0-framework-self/). Advances in Neural Information Processing Systems, 33:12449–12460.
19. Bengio, Yoshua, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. In Advances in Neural Information Processing Systems, 13:932–938. [find](https://scholar.google.com/scholar?q=Bengio%2C+Yoshua%2C+Re%CC%81jean+Ducharme%2C+and+Pascal+Vincent%2E+2000%2E+A+neural+probabilistic+language+model%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+13%3A932%E2%80%93938%2E)
20. Bentouati, Abdelkader. 2020. Arabic language controls before the descent of the holy quran, points - shape - sizes. Bidayat, 2(1):18–28. [find](https://scholar.google.com/scholar?q=Bentouati%2C+Abdelkader%2E+2020%2E+Arabic+language+controls+before+the+descent+of+the+holy+quran%2C+points+%2D+shape+%2D+sizes%2E+Bidayat%2C+2%281%29%3A18%E2%80%9328%2E)
21. Boudad, Naaima, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb. 2018. Sentiment analysis in Arabic: A review of the literature. Ain Shams Engineering Journal, 9(4):2479–2490. https://doi.org/10.1016/j.asej .2017.04.007 [↗](https://doi.org/10.1016/j.asej)
22. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems, 33:1877–1901.
23. Cettolo, Mauro, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. [Overview of the IWSLT 2017 evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/). In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2–14.
24. Chernyavskiy, Anton, Dmitry Ilvovsky, and Preslav Nakov. 2021. Transformers: “The end of history” for natural language processing? In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Proceedings, Part III 21, pages 677–693. https://doi.org/10.1007/978-3-030-86523-8 41 [↗](https://doi.org/10.1007/978-3-030-86523-8)
25. Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412 .3555. [find](https://scholar.google.com/scholar?q=Chung%2C+Junyoung%2C+Caglar+Gulcehre%2C+KyungHyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+arXiv+preprint+arXiv%3A1412+%2E3555%2E)
26. Dehghani, Mostafa, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819. [↗](https://arxiv.org/abs/1807.03819)
27. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
28. Diab, Mona, Mahmoud Ghoneim, and Nizar Habash. 2007. [Arabic diacritization in the context of statistical machine translation](https://md.research.elamur.ai/free/en/papers/diab-2007-arabic-diacritization-context-statistical-machine/). In Proceedings of Machine Translation Summit XI: Papers.
29. Einea, Omar, Ashraf Elnagar, and Ridhwan Al Debsi. 2019. Sanad: Single-label Arabic news articles dataset for automatic text categorization. Data in Brief, 25:104076. https://doi.org/10.1016/j.dib .2019.104076, PubMed: 31440535 [↗](https://doi.org/10.1016/j.dib)
30. El-Khair, Ibrahim Abu. 2016. [1.5 billion words Arabic corpus](https://md.research.elamur.ai/free/en/papers/el-khair-2016-1-5-billion-words-arabic/). ArXiv, abs/1611.04033. [↗](https://arxiv.org/abs/1611.04033)
31. Falcon, William et al. 2019. PyTorch lightning. GitHub. Note. [https://github.com/PyTorchLightning/pytorch](https://github.com/PyTorchLightning/pytorch) -lightning. 3(6):11. [↗](https://github.com/PyTorchLightning/pytorch)
32. Gal, Yarin and Zoubin Ghahramani. 2016. [A theoretically grounded application of dropout in recurrent neural networks](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/). Advances in Neural Information Processing Systems, 29:1019–1027.
33. Guellil, Imane, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. [Arabic natural language processing: An overview](https://md.research.elamur.ai/free/en/papers/guellil-2019-arabic-natural-language-processing-overview/). Journal of King Saud University-Computer and Information Sciences, 33(5):497–507. https://doi.org/10.1016/j.jksuci.2019.02.006 [↗](https://doi.org/10.1016/j.jksuci.2019.02.006)
34. Heafield, Kenneth. 2011. [KenLM: Faster and smaller language model queries](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/). In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197.
35. Heafield, Kenneth. 2013. Efficient Language Modeling Algorithms with Applications to Statistical Machine Translation. Ph. D. thesis, Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=Heafield%2C+Kenneth%2E+2013%2E+Efficient+Language+Modeling+Algorithms+with+Applications+to+Statistical+Machine+Translation%2E+Ph%2E+D%2E+thesis%2C+Carnegie+Mellon+University%2E)
36. Heafield, Kenneth, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. [Scalable modified Kneser-Ney language model estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696.
37. Heaps, Harold Stanley. 1978. Information Retrieval, Computational and Theoretical Aspects. Academic Press. [find](https://scholar.google.com/scholar?q=Heaps%2C+Harold+Stanley%2E+1978%2E+Information+Retrieval%2C+Computational+and+Theoretical+Aspects%2E+Academic+Press%2E)
38. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735, PubMed: 9377276 [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
39. Inan, Hakan, Khashayar Khosravi, and Richard Socher. 2016. [Tying word vectors and word classifiers: A loss framework for language modeling](https://md.research.elamur.ai/free/en/papers/inan-2016-tying-word-vectors-word-classifiers/). arXiv preprint arXiv:1611.01462. [↗](https://arxiv.org/abs/1611.01462)
40. Katz, Slava. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3):400–401. https://doi.org/10.1109/TASSP.1987.1165125 [↗](https://doi.org/10.1109/tassp.1987.1165125)
41. Khandelwal, Urvashi, He He, Peng Qi, and Dan Jurafsky. 2018. [Sharp nearby, fuzzy far away: How neural language models use context](https://md.research.elamur.ai/free/en/papers/khandelwal-2018-sharp-nearby-fuzzy-far-away/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284–294. https://doi.org/10 .18653/v1/P18-1027 [↗](https://aclanthology.org/P18-1027/)
42. Khreisat, Laila. 2009. A machine learning approach for Arabic text classification using n-gram frequency statistics. Journal of Informetrics, 3(1):72–77. https://doi.org/10.1016/j.joi.2008.11.005 [↗](https://doi.org/10.1016/j.joi.2008.11.005)
43. Kingma, Diederik P. and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
44. Kneser, Reinhard and Hermann Ney. 1995. Improved backing-off for M-gram language modeling. In 1995 International Conference on Acoustics, Speech, and Signal Processing, 1:181–184. https://doi.org/10.1109/ICASSP.1995.479394 [↗](https://doi.org/10.1109/icassp.1995.479394)
45. Kudo, Taku and John Richardson. 2018. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71. https://doi.org/10 .18653/v1/D18-2012 [↗](https://aclanthology.org/D18-2012/)
46. Li, Liam, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. 2020. A system for massively parallel hyperparameter tuning. In Proceedings of Machine Learning and Systems, 2:230–246. [find](https://scholar.google.com/scholar?q=Li%2C+Liam%2C+Kevin+Jamieson%2C+Afshin+Rostamizadeh%2C+Ekaterina+Gonina%2C+Jonathan+Ben%2Dtzur%2C+Moritz+Hardt%2C+Benjamin+Recht%2C+and+Ameet+Talwalkar%2E+2020%2E+A+system+for+massively+parallel+hyperparameter+tuning%2E+In+P)
47. Li, Wentian. 1992. Random texts exhibit Zipf’s-law-like word frequency distribution. IEEE Transactions on Information Theory, 38(6):1842–1845. https://doi.org/10.1109/18.165464 [↗](https://doi.org/10.1109/18.165464)
48. Liaw, Richard, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118. [↗](https://arxiv.org/abs/1807.05118)
49. Liu, Liyuan, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1908.03265)
50. Magueresse, Alexandre, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264. [↗](https://arxiv.org/abs/2006.07264)
51. Mghari, Mohammed, Omar Bouras, and Abdelaaziz El Hibaoui. 2022. Sanadset 650K: Data on Hadith narrators. Data in Brief, 44. https://doi.org/10.1016/j.dib.2022.108540, PubMed: 36065202 [↗](https://doi.org/10.60692/hz9yp-cah90)
52. Moreno-Sánchez, Isabel, Francesc Font-Clos, and Álvaro Corral. 2016. Large-scale analysis of Zipf’s law in English texts. PloS One, 11(1):e0147073. https://doi.org/10.1371/journal.pone.0147073, PubMed: 26800025 [↗](https://doi.org/10.1371/journal.pone.0147073)
53. Ney, Hermann, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1):1–38. https://doi.org/10.1006/csla.1994.1001 [↗](https://doi.org/10.1006/csla.1994.1001)
54. Nivre, Joakim, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. [Universal Dependencies v2: An evergrowing multilingual treebank collection](https://md.research.elamur.ai/free/en/papers/nivre-2020-universal-dependencies-v2/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043. [↗](https://doi.org/10.48550/arxiv.2004.10643)
55. Nowlan, Steven and Geoffrey E Hinton. 1991. Adaptive soft weight tying using gaussian mixtures. Advances in Neural Information Processing Systems, 4:993–1000. [find](https://scholar.google.com/scholar?q=Nowlan%2C+Steven+and+Geoffrey+E+Hinton%2E+1991%2E+Adaptive+soft+weight+tying+using+gaussian+mixtures%2E+Advances+in+Neural+Information+Processing+Systems%2C+4%3A993%E2%80%931000%2E)
56. Pappas, Nikolaos, Lesly Miculicich, and James Henderson. 2018. Beyond weight tying: Learning joint input-output embeddings for neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 73–83. https://doi.org/10.18653/v1/W18-6308 Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. [PyTorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in Neural Information Processing Systems, 32:8026–8037. [↗](https://doi.org/10.18653/v1/w18-6308)
57. Post, Matt. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191. https://doi.org/10.18653/v1/W18-6319 [↗](https://doi.org/10.18653/v1/w18-6319)
58. Press, Ofir and Lior Wolf. 2017. [Using the output embedding to improve language models](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. https://doi.org/10.18653/v1/E17-2025 [↗](https://doi.org/10.18653/v1/e17-2025)
59. Rom, Aviad and Kfir Bar. 2021. [Supporting undotted Arabic with pre-trained language models](https://md.research.elamur.ai/free/en/papers/rom-2021-supporting-undotted-arabic-pre-trained/). In Proceedings of the 4th International Conference on Natural Language and Speech Processing (ICNLSP 2021), pages 89–94. [↗](https://doi.org/10.48550/arxiv.2111.09791)
60. Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. 1985. Learning internal representations by error propagation. University of California San Diego La Jolla Institute for Cognitive Science. https://doi.org/10.21236/ADA164453 [↗](https://doi.org/10.21236/ada164453)
61. Selab, Essma and Ahmed Guessoum. 2015. Building TALAA, a free general and categorized Arabic corpus. In ICAART (1), pages 284–291. https://doi.org/10.5220/0005352102840291 [↗](https://doi.org/10.5220/0005352102840291)
62. Sicilia-Garcia, Elvira I., Ji Ming, Francis J. Smith, et al. 2003. Extension of Zipf’s law to word and character n-grams for English and Chinese. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Processing, pages 77–102. [find](https://scholar.google.com/scholar?q=Sicilia%2DGarcia%2C+Elvira+I%2E%2C+Ji+Ming%2C+Francis+J%2E+Smith%2C+et+al%2E+2003%2E+Extension+of+Zipf%E2%80%99s+law+to+word+and+character+n%2Dgrams+for+English+and+Chinese%2E+In+International+Journal+of+Computational+Linguistics+)
63. Smith, Ray. 2011. Limits on the application of frequency-based language models to OCR. In 2011 International Conference on Document Analysis and Recognition, pages 538–542. https://doi.org/10.1109/ICDAR.2011.114 [↗](https://doi.org/10.1109/icdar.2011.114)
64. Stolcke, Andreas. 2002. SRILM - an extensible language modeling toolkit. In Seventh International Conference on Spoken Language Processing. https://doi.org/10.21437/ICSLP.2002-303 [↗](https://doi.org/10.21437/ICSLP.2002-303)
65. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 30:5998–6008.
66. Welsh, Dominic. 1988. Codes and Cryptography. Oxford University Press. [find](https://scholar.google.com/scholar?q=Welsh%2C+Dominic%2E+1988%2E+Codes+and+Cryptography%2E+Oxford+University+Press%2E)
67. Werbos, Paul J. 1990. Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560. https://doi.org/10 .1109/5.58337 Wikimedia Foundation. Wikimedia downloads. [↗](https://doi.org/10.1109/5.58337)
68. Zhang, Hui and David Chiang. 2014. Kneser-Ney smoothing on expected counts. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 765–774. https://doi.org/10 .3115/v1/P14-1072 [↗](https://aclanthology.org/P14-1072/)
69. Ziemski, Michał, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. [The United Nations Parallel Corpus v1.0](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534.
70. Zipf, George Kingsley 2016. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Ravenio Books. [find](https://scholar.google.com/scholar?q=Zipf%2C+George+Kingsley+2016%2E+Human+Behavior+and+the+Principle+of+Least+Effort%3A+An+Introduction+to+Human+Ecology%2E+Ravenio+Books%2E)

*Updated 2 November 2026*