---
title: "Qalam — paper by Bhatia et al. (2024) — summary and ideas"
description: "The authors present Qalam, a vision encoder-decoder transformer that… 3.95% WER on OnlineKHATT. 82 references, cited by 8 papers here."
authors: ["Gagan Bhatia", "El Moatez Billah Nagoudi", "Fakhraddin Alwajih", "Muhammad Abdul-Mageed"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.19/"
doi: "10.18653/v1/2024.arabicnlp-1.19"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bhatia-2024-qalam-multimodal-llm-arabic-optical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bhatia-2024-qalam-multimodal-llm-arabic-optical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

# Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition

**Authors:** [Gagan Bhatia](https://md.research.elamur.ai/free/en/authors/gagan-bhatia/), [El Moatez Billah Nagoudi](https://md.research.elamur.ai/free/en/authors/el-moatez-billah-nagoudi/), [Fakhraddin Alwajih](https://md.research.elamur.ai/free/en/authors/fakhraddin-alwajih/) and [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.19/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.19.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.19](https://doi.org/10.18653/v1/2024.arabicnlp-1.19) · [OpenAlex](https://openalex.org/W4402671279)

**Citations:** 12 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/bhatia-2024-qalam-multimodal-llm-arabic-optical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic text is hard for OCR and handwriting recognition (HWR). The script is cursive, letters change shape with position, and diacritics, dots, ligatures and overlaps all have to be read. Fonts and personal hands vary widely, and annotated data is limited. The authors present Qalam, a vision encoder-decoder transformer that pairs a SwinV2 image encoder with a RoBERTa text decoder. They also assemble MIDAD, a benchmark that gathers existing Arabic, Persian and Urdu collections at character, word and line level. Training draws on over 4.5 million manuscript images plus 60k synthetic image-text pairs. Qalam is compared with Tesseract, TrOCR, CTC-based CNN/BiLSTM systems and other encoder-decoder combinations, and it reaches the lowest error on nearly every MIDAD set. Its average WER is 0.80% on HWR and 1.18% on OCR. On unseen real-world data it beats prior work on KHATT but trails on historical manuscripts. The work matters because it shows that script-specific pretraining on large data beats English-pretrained recognizers for Arabic.

## Key points
- Architecture: SwinV2 encoder with a RoBERTa decoder. Variants tested include DeIT+ARBERT and SwinV2(fp)+ARBERT, with WER of 11.06% and 4.56% respectively on the overall MIDAD Score.
- MIDAD covers the following collections:
  - MADBase (digits) and AHCD (characters)
  - ADAB and Alexuw (handwritten words)
  - OnlineKHATT (handwritten lines; InkML strokes rendered as images)
  - PATS-A01 (printed lines), Shotor and IDPL-PFOD (Persian)
  - UPTI (Urdu Nastaliq)
- Data splits: original Train/Dev/Test splits are kept where they exist; otherwise data is shuffled into 80/10/10.
- Results: Qalam is error-free on MADBase and AHCD, has 3.95% WER on OnlineKHATT and 1.90% on PATS01, and scores 0.94% on the MIDAD Score (average WER over all sets). Tesseract and TrOCR Base perform far worse.
- Zero-shot CER on in-the-wild data:
  - KHATT: 10.43, versus 18.45 for the previous SOTA.
  - British Library/Qatar Foundation Historical Manuscripts: 30.88, versus 21.9 for the previous SOTA.

## Interesting ideas
- Results are grouped into clusters by unit size (character, word, line) and by task, then averaged into one benchmark-wide number, so a model cannot look good on only one granularity.
- After in-domain testing, the model is run zero-shot on completely unseen real-world sets, and the loss on historical manuscripts is reported openly.
- Encoder and decoder are swapped systematically (DeIT vs SwinV2, ARBERT vs RoBERTa) to separate each component's contribution.
- Online pen-stroke data (InkML) is converted into the same image form as the other sources, so very different datasets can feed one model.
- Datasets from closely related languages (Persian, Urdu) are added to the Arabic data to broaden coverage of the script.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Mova could report one composite score that averages separate token, tree and frame clusters, so gains at one stage cannot hide losses at another.
- **I2** (learn): Mova could keep a fixed out-of-domain shelf (e.g. old literary prose) that training never touches, as a standing check against overfitting to silver data.
- **I3** (tree): gparse could ablate its encoder and its scoring head one at a time to see which part drives LAS gains.
- **I4** (tree): treebanks in other UD dialects or formats could be normalized into one representation before training, as with the UD dialect conversion.
- **I5** (tokens): text in English varieties such as historical or regional English could enrich the tagger and morphology with rarer spellings and forms.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 98 works · 34,805 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [⚙️ Engineering](https://md.research.elamur.ai/free/en/shelves/engineering/) · 133 works · [⚙️ Media Technology](https://md.research.elamur.ai/free/en/shelves/engineering/media-technology/) · 34 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih and Muhammad Abdul-Mageed. 2024. Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bhatia2024qalam,
  title = {{Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition}},
  author = {Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.19},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.19}
}
```

- [← previous MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding](https://md.research.elamur.ai/free/en/papers/li-2024-mmsci-dataset-graduate-level-multi/)
- [next → Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://md.research.elamur.ai/free/en/papers/miletic-2024-semantics-multiword-expressions-transformer-based/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

82 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, Abdelrahim Elmadany, Alcides Inciarte, Md Tawkat Islam Khondaker, et al. 2023. [Jasmine: Arabic gpt models for few-shot learning](https://md.research.elamur.ai/free/en/papers/nagoudi-2023-jasmine-arabic-gpt-models-few/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16721–16744.
2. Riaz Ahmad, Saeeda Naz, M Zeshan Afzal, S Faisal Rashid, Marcus Liwicki, and Andreas Dengel. 2017. [Khatt: A deep learning benchmark on arabic script](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 7, pages 10–14. IEEE.
3. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774)
4. Oscar E Agazzi and Shyh-shiaw Kuo. 1993. Hidden markov model based optical character recognition in the presence of deterministic transformations. Pattern recognition, 26(12):1813–1826. [↗](https://doi.org/10.1016/0031-3203(93)90178-y)
5. Husni Al-Muhtaseb, Sabri Mahmoud, and Rami Qah-waji. 2009. Automatic arabic text image optical character recognition method. [find](https://scholar.google.com/scholar?q=Husni+Al%2DMuhtaseb%2C+Sabri+Mahmoud%2C+and+Rami+Qah%2Dwaji%2E+2009%2E+Automatic+arabic+text+image+optical+character+recognition+method%2E)
6. Salah Alghyaline. 2023. Arabic optical character recognition: A review. CMES-Computer Modeling in Engineering & Sciences, 135(3). [↗](https://doi.org/10.32604/cmes.2022.024555)
7. Somaya Alma’adeed, Colin Higgens, and Dave Elli-man. 2002. Recognition of off-line handwritten arabic words using hidden markov model approach. In [https://alliancecan.ca](https://alliancecan.ca) 2002 International Conference on Pattern Recognition, volume 3, pages 481–484. IEEE. [↗](https://alliancecan.ca)
8. Naseem Alrobah and Saleh Albahli. 2022. Arabic handwritten recognition using deep learning: A survey. Arabian Journal for Science and Engineering, 47(8):9943–9963. [↗](https://doi.org/10.1007/s13369-021-06363-3)
9. Fakhraddin Alwajih, Eman Badr, Sherif Abdou, and Aly Fahmy. 2021. Deeponkhatt: An end-to-end arabic online handwriting recognition system. International Journal of Pattern Recognition and Artificial Intelligence, 35. [↗](https://doi.org/10.1142/s0218001421530062)
10. Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. [Peacock: A family of arabic multimodal large language models and benchmarks](https://md.research.elamur.ai/free/en/papers/alwajih-2024-peacock-family-arabic-multimodal-large/). arXiv preprint arXiv:2403.01031. [↗](https://arxiv.org/abs/2403.01031)
11. Amir Abbas Asadi. 2020. Shotor dataset. [find](https://scholar.google.com/scholar?q=Amir+Abbas+Asadi%2E+2020%2E+Shotor+dataset%2E)
12. Christian Clausner, Apostolos Antonacopoulos, Nora Mcgregor, and Daniel Wilson-Nunn. 2018. Icfhr 2018 competition on recognition of historical arabic scientific manuscripts–rasm2018. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 471–476. IEEE. [find](https://scholar.google.com/scholar?q=Christian+Clausner%2C+Apostolos+Antonacopoulos%2C+Nora+Mcgregor%2C+and+Daniel+Wilson%2DNunn%2E+2018%2E+Icfhr+2018+competition+on+recognition+of+historical+arabic+scientific+manuscripts%E2%80%93rasm2018%2E+In+2018+16th+Inte)
13. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. [↗](https://arxiv.org/abs/1911.02116)
14. Denis Coquenet, Clément Chatelain, and Thierry Paquet. 2020. Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 19–24. IEEE. [find](https://scholar.google.com/scholar?q=Denis+Coquenet%2C+Cl%C3%A9ment+Chatelain%2C+and+Thierry+Paquet%2E+2020%2E+Recurrence%2Dfree+unconstrained+handwritten+text+recognition+using+gated+fully+convolutional+network%2E+In+2020+17th+International+Conference+o)
15. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473. [↗](https://arxiv.org/abs/1409.0473)
16. Denis Coquenet, Clément Chatelain, and Thierry Paquet. 2022. End-to-end handwritten paragraph text recognition using a vertical attention network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):508–524. [↗](https://doi.org/10.1109/tpami.2022.3144899)
17. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254. [↗](https://arxiv.org/abs/2106.08254)
18. Iam Palatnik de Sousa. 2018. Convolutional ensembles for arabic handwritten character and digit recognition. PeerJ Computer Science, 4. [↗](https://doi.org/10.7717/peerj-cs.167)
19. Killian Barrere, Yann Soullard, Aurélie Lemaitre, and Bertrand Coüasnon. 2022. A light transformer-based architecture for handwritten text recognition. In International Workshop on Document Analysis Systems, pages 275–290. Springer. [find](https://scholar.google.com/scholar?q=Killian+Barrere%2C+Yann+Soullard%2C+Aur%C3%A9lie+Lemaitre%2C+and+Bertrand+Co%C3%BCasnon%2E+2022%2E+A+light+transformer%2Dbased+architecture+for+handwritten+text+recognition%2E+In+International+Workshop+on+Document+Analysis+S)
20. Arthur Flor de Sousa Neto, Byron Leite Dantas Bezerra, Alejandro Héctor Toselli, and Estanislau Baptista Lima. 2020. Htr-flor: A deep learning system for offline handwritten text recognition. In 2020 33rd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 54–61. IEEE. [find](https://scholar.google.com/scholar?q=Arthur+Flor+de+Sousa+Neto%2C+Byron+Leite+Dantas+Bezerra%2C+Alejandro+H%C3%A9ctor+Toselli%2C+and+Estanislau+Baptista+Lima%2E+2020%2E+Htr%2Dflor%3A+A+deep+learning+system+for+offline+handwritten+text+recognition%2E+In+2020+)
21. Théodore Bluche, Jérôome Louradour, and Ronaldo Messina. 2017. Scan, attend and read: End-to-end handwritten paragraph recognition with mdlstm attention. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1050–1055. IEEE. [find](https://scholar.google.com/scholar?q=Th%C3%A9odore+Bluche%2C+J%C3%A9r%C3%B4ome+Louradour%2C+and+Ronaldo+Messina%2E+2017%2E+Scan%2C+attend+and+read%3A+End%2Dto%2Dend+handwritten+paragraph+recognition+with+mdlstm+attention%2E+In+2017+14th+IAPR+international+conference+on+)
22. Théodore Bluche and Ronaldo Messina. 2017. Gated convolutional recurrent neural networks for multilingual handwriting recognition. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 646–651. IEEE. [find](https://scholar.google.com/scholar?q=Th%C3%A9odore+Bluche+and+Ronaldo+Messina%2E+2017%2E+Gated+convolutional+recurrent+neural+networks+for+multilingual+handwriting+recognition%2E+In+2017+14th+IAPR+international+conference+on+document+analysis+and+r)
23. Houcine Boubaker, Abdelkarim Elbaati, Najiba Tagougui, Haikal El Abed, Monji Kherallah, Volker Märgner, and Adel M. Alimi. 2021. Adab database. [find](https://scholar.google.com/scholar?q=Houcine+Boubaker%2C+Abdelkarim+Elbaati%2C+Najiba+Tagougui%2C+Haikal+El+Abed%2C+Monji+Kherallah%2C+Volker+M%C3%A4rgner%2C+and+Adel+M%2E+Alimi%2E+2021%2E+Adab+database%2E)
24. Thomas M Breuel. 2017. High performance text recognition using a hybrid convolutional-lstm implementation. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 11–16. IEEE. [find](https://scholar.google.com/scholar?q=Thomas+M+Breuel%2E+2017%2E+High+performance+text+recognition+using+a+hybrid+convolutional%2Dlstm+implementation%2E+In+2017+14th+IAPR+international+conference+on+document+analysis+and+recognition+%28ICDAR%29%2C+volu)
25. Horst Bunke, Markus Roth, and Ernst Günter Schukat-Talamazzini. 1995. Off-line cursive handwriting recognition using hidden markov models. Pattern recognition, 28(9):1399–1413. [find](https://scholar.google.com/scholar?q=Horst+Bunke%2C+Markus+Roth%2C+and+Ernst+G%C3%BCnter+Schukat%2DTalamazzini%2E+1995%2E+Off%2Dline+cursive+handwriting+recognition+using+hidden+markov+models%2E+Pattern+recognition%2C+28%289%29%3A1399%E2%80%931413%2E)
26. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
27. Patrick Doetsch, Albert Zeyer, and Hermann Ney. 2016. Bidirectional decoder networks for attention-based end-to-end offline handwriting recognition. In 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 361–366. IEEE. [find](https://scholar.google.com/scholar?q=Patrick+Doetsch%2C+Albert+Zeyer%2C+and+Hermann+Ney%2E+2016%2E+Bidirectional+decoder+networks+for+attention%2Dbased+end%2Dto%2Dend+offline+handwriting+recognition%2E+In+2016+15th+International+Conference+on+Frontiers+)
28. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). arXiv preprint arXiv:2010.11929. [↗](https://arxiv.org/abs/2010.11929)
29. Haikal El Abed, Volker Märgner, Monji Kherallah, and Adel M Alimi. 2009. Icdar 2009 online arabic handwriting recognition competition. In 2009 10th International Conference on Document Analysis and Recognition, pages 1388–1392. IEEE. [find](https://scholar.google.com/scholar?q=Haikal+El+Abed%2C+Volker+M%C3%A4rgner%2C+Monji+Kherallah%2C+and+Adel+M+Alimi%2E+2009%2E+Icdar+2009+online+arabic+handwriting+recognition+competition%2E+In+2009+10th+International+Conference+on+Document+Analysis+and+Re)
30. Ahmed El-Sawy, Hazem EL-Bakry, and Mohamed Loey. 2017a. Cnn for handwritten arabic digits recognition based on lenet-5. pages 566–575. [find](https://scholar.google.com/scholar?q=Ahmed+El%2DSawy%2C+Hazem+EL%2DBakry%2C+and+Mohamed+Loey%2E+2017a%2E+Cnn+for+handwritten+arabic+digits+recognition+based+on+lenet%2D5%2E+pages+566%E2%80%93575%2E)
31. Ahmed El-Sawy, Mohamed Loey, and Hazem El-Bakry. 2017b. Arabic handwritten characters recognition using convolutional neural network. WSEAS Transactions on Computer Research, 5(1):11–19. [find](https://scholar.google.com/scholar?q=Ahmed+El%2DSawy%2C+Mohamed+Loey%2C+and+Hazem+El%2DBakry%2E+2017b%2E+Arabic+handwritten+characters+recognition+using+convolutional+neural+network%2E+WSEAS+Transactions+on+Computer+Research%2C+5%281%29%3A11%E2%80%9319%2E)
32. Ahmed El-Sawy, Mohamed Loey, and Hazem El-Bakry. 2017c. Arabic handwritten characters recognition using convolutional neural network. WSEAS Transactions on Computer Research, 5(1):11–19. [find](https://scholar.google.com/scholar?q=Ahmed+El%2DSawy%2C+Mohamed+Loey%2C+and+Hazem+El%2DBakry%2E+2017c%2E+Arabic+handwritten+characters+recognition+using+convolutional+neural+network%2E+WSEAS+Transactions+on+Computer+Research%2C+5%281%29%3A11%E2%80%9319%2E)
33. Safiullah Faizullah, Muhammad Sohaib Ayub, Sajid Hussain, and Muhammad Asad Khan. 2023. A survey of ocr in arabic language: Applications, techniques, and challenges. Applied Sciences, 13(7):4584. [↗](https://doi.org/10.3390/app13074584)
34. Chen-Yu Lee and Simon Osindero. 2016. Recursive recurrent nets with attention modeling for ocr in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2231– 2239. [↗](https://doi.org/10.48550/arxiv.1603.03101)
35. Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2021. Trocr: Transformer-based optical character recognition with pre-trained models. arXiv preprint arXiv:2109.10282. [↗](https://arxiv.org/abs/2109.10282)
36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/).
37. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369–376. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Santiago+Fern%C3%A1ndez%2C+Faustino+Gomez%2C+and+J%C3%BCrgen+Schmidhuber%2E+2006%2E+Connectionist+temporal+classification%3A+labelling+unsegmented+sequence+data+with+recurrent+neural+networks%2E+In+Proceedings)
38. Alex Graves and Jürgen Schmidhuber. 2008. Offline handwriting recognition with multidimensional recurrent neural networks. Advances in neural information processing systems, 21. Hindawi. [link]. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+J%C3%BCrgen+Schmidhuber%2E+2008%2E+Offline+handwriting+recognition+with+multidimensional+recurrent+neural+networks%2E+Advances+in+neural+information+processing+systems%2C+21%2E+Hindawi%2E+%5Blink%5D%2E)
39. Fatemeh sadat Hosseini, Shima Kashef, Elham Shaban-inia, and Hossein Nezamabadi-pour. 2021. Idpl-pfod: An image dataset of printed farsi text for ocr research. pages 22–31. [find](https://scholar.google.com/scholar?q=Fatemeh+sadat+Hosseini%2C+Shima+Kashef%2C+Elham+Shaban%2Dinia%2C+and+Hossein+Nezamabadi%2Dpour%2E+2021%2E+Idpl%2Dpfod%3A+An+image+dataset+of+printed+farsi+text+for+ocr+research%2E+pages+22%E2%80%9331%2E)
40. Mohamed E. Hussein, Marwan Torki, Ahmed Elsallamy, and Mahmoud Fayyaz. 2014a. Alexu-word: A new dataset for isolated-word closed-vocabulary offline arabic handwriting recognition. [↗](https://doi.org/10.48550/arxiv.1411.4670)
41. Mohamed E Hussein, Marwan Torki, Ahmed Elsallamy, and Mahmoud Fayyaz. 2014b. Alexu-word: a new dataset for isolated-word closed-vocabulary offline arabic handwriting recognition. arXiv preprint arXiv:1411.4670. [↗](https://arxiv.org/abs/1411.4670)
42. Mohamed E. Hussein, Marwan Torki, Ahmed Elsallamy, and Mahmoud Fayyaz. 2014c. Alexu-word: A new dataset for isolated-word closed-vocabulary offline arabic handwriting recognition. [↗](https://doi.org/10.48550/arxiv.1411.4670)
43. Mohit Jain, Minesh Mathew, and C.V. Jawahar. 2017. Unconstrained ocr for urdu using deep cnn-rnn hybrid networks. pages 747–752. [find](https://scholar.google.com/scholar?q=Mohit+Jain%2C+Minesh+Mathew%2C+and+C%2EV%2E+Jawahar%2E+2017%2E+Unconstrained+ocr+for+urdu+using+deep+cnn%2Drnn+hybrid+networks%2E+pages+747%E2%80%93752%2E)
44. Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, and Mauricio Villegas. 2022. Pay attention to what you read: non-recurrent handwritten text-line recognition. Pattern Recognition, 129:108766. [↗](https://doi.org/10.1016/j.patcog.2022.108766)
45. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. [Ocr-free document understanding transformer](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/). In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVIII, pages 498–517. Springer.
46. Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023. [On the hidden mystery of ocr in large multimodal models](https://md.research.elamur.ai/free/en/papers/liu-2023-ocrbench-hidden-mystery-ocr-large/). arXiv preprint arXiv:2305.07895. [↗](https://arxiv.org/abs/2305.07895)
47. Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. 2024. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473. [↗](https://arxiv.org/abs/2403.04473)
48. Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. 2022. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009–12019. [find](https://scholar.google.com/scholar?q=Ze+Liu%2C+Han+Hu%2C+Yutong+Lin%2C+Zhuliang+Yao%2C+Zhenda+Xie%2C+Yixuan+Wei%2C+Jia+Ning%2C+Yue+Cao%2C+Zheng+Zhang%2C+Li+Dong%2C+et+al%2E+2022%2E+Swin+transformer+v2%3A+Scaling+up+capacity+and+resolution%2E+In+Proceedings+of+the+I)
49. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. [Swin transformer: Hierarchical vision transformer using shifted windows](https://md.research.elamur.ai/free/en/papers/liu-2021-swin-transformer-hierarchical-vision-transformer/). In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
50. Pengyuan Lyu, Chengquan Zhang, Shanshan Liu, Meina Qiao, Yangliu Xu, Liang Wu, Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. 2022. Maskocr: Text recognition with masked encoder-decoder pre-training. arXiv preprint arXiv:2206.00311. [↗](https://arxiv.org/abs/2206.00311)
51. Rania Maalej, Najiba Tagougui, and Monji Kherallah. 2016. Online arabic handwriting recognition with dropout applied in deep recurrent neural networks. In 2016 12th IAPR Workshop on Document Analysis Systems (DAS), pages 417–421. [find](https://scholar.google.com/scholar?q=Rania+Maalej%2C+Najiba+Tagougui%2C+and+Monji+Kherallah%2E+2016%2E+Online+arabic+handwriting+recognition+with+dropout+applied+in+deep+recurrent+neural+networks%2E+In+2016+12th+IAPR+Workshop+on+Document+Analysis+)
52. Sabri Mahmoud, Irfan Ahmad, Wasfi Al-Khatib, Mohammad Alshayeb, Mohammad Parvez, Volker Märgner, and Gernot Fink. 2014. Khatt: An open arabic offline handwritten text database. Pattern Recognition, 47:1096–1112. [↗](https://doi.org/10.1016/j.patcog.2013.08.009)
53. Sabri A Mahmoud, Hamzah Luqman, Baligh M Al-Helali, Galal BinMakhashen, and Mohammad Tanvir Parvez. 2018. Online-khatt: an open-vocabulary database for arabic online-text processing. The Open Cybernetics & Systemics Journal, 12(1). [↗](https://doi.org/10.2174/1874110x01812010042)
54. Johannes Michael, Roger Labahn, Tobias Grüning, and Jochen Zöllner. 2019. Evaluating sequence-to-sequence models for handwritten text recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1286–1293. IEEE. [find](https://scholar.google.com/scholar?q=Johannes+Michael%2C+Roger+Labahn%2C+Tobias+Gr%C3%BCning%2C+and+Jochen+Z%C3%B6llner%2E+2019%2E+Evaluating+sequence%2Dto%2Dsequence+models+for+handwritten+text+recognition%2E+In+2019+International+Conference+on+Document+Analysis)
55. Saleh Momeni and Bagher BabaAli. 2023. A transformer-based approach for arabic offline handwritten text recognition. arXiv preprint arXiv:2307.15045. [↗](https://arxiv.org/abs/2307.15045)
56. Aly Mostafa, Omar Mohamed, Ali Ashraf, Ahmed El-behery, Salma Jamal, Ghada Khoriba, and Amr S Ghoneim. 2021. Ocformer: A transformer-based model for arabic handwritten text recognition. In 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), pages 182– 186. IEEE. [find](https://scholar.google.com/scholar?q=Aly+Mostafa%2C+Omar+Mohamed%2C+Ali+Ashraf%2C+Ahmed+El%2Dbehery%2C+Salma+Jamal%2C+Ghada+Khoriba%2C+and+Amr+S+Ghoneim%2E+2021%2E+Ocformer%3A+A+transformer%2Dbased+model+for+arabic+handwritten+text+recognition%2E+In+2021+Intern)
57. Hee-Seon Park and Seong-Whan Lee. 1996. Off-line recognition of large-set handwritten characters with multiple hidden markov models. Pattern Recognition, 29(2):231–244. [↗](https://doi.org/10.1016/0031-3203(95)00081-x)
58. Chirag Patel, Atul Patel, and Dharmendra Patel. 2012. Optical character recognition by open source ocr tool tesseract: A case study. International Journal of Computer Applications, 55(10):50–56. [↗](https://doi.org/10.5120/8794-2784)
59. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In 2014 14th international conference on frontiers in handwriting recognition, pages 285–290. IEEE. [find](https://scholar.google.com/scholar?q=Vu+Pham%2C+Th%C3%A9odore+Bluche%2C+Christopher+Kermorvant%2C+and+J%C3%A9r%C3%B4me+Louradour%2E+2014%2E+Dropout+improves+recurrent+neural+networks+for+handwriting+recognition%2E+In+2014+14th+international+conference+on+frontiers)
60. Rohit Prasad, Shirin Saleem, Matin Kamali, Ralf Meermeier, and Prem Natarajan. 2008. Improvements in hidden markov model based arabic ocr. In 2008 19th International Conference on Pattern Recognition, pages 1–4. IEEE. [find](https://scholar.google.com/scholar?q=Rohit+Prasad%2C+Shirin+Saleem%2C+Matin+Kamali%2C+Ralf+Meermeier%2C+and+Prem+Natarajan%2E+2008%2E+Improvements+in+hidden+markov+model+based+arabic+ocr%2E+In+2008+19th+International+Conference+on+Pattern+Recognition%2C)
61. Joan Puigcerver. 2017. Are multidimensional recurrent layers really necessary for handwritten text recognition? 01:67–72. [find](https://scholar.google.com/scholar?q=Joan+Puigcerver%2E+2017%2E+Are+multidimensional+recurrent+layers+really+necessary+for+handwritten+text+recognition%3F+01%3A67%E2%80%9372%2E)
62. Nazly Sabbour and Faisal Shafait. 2013. A segmentation-free approach to arabic and urdu ocr. [find](https://scholar.google.com/scholar?q=Nazly+Sabbour+and+Faisal+Shafait%2E+2013%2E+A+segmentation%2Dfree+approach+to+arabic+and+urdu+ocr%2E)
63. Fatemeh sadat Hosseini, Shima Kashef, Elham Shaban-inia, and Hossein Nezamabadi-pour. 2021. Idpl-pfod: An image dataset of printed farsi text for ocr research. In Proceedings of the Second International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2021) co-located with ICNLSP 2021, pages 22–31. [find](https://scholar.google.com/scholar?q=Fatemeh+sadat+Hosseini%2C+Shima+Kashef%2C+Elham+Shaban%2Dinia%2C+and+Hossein+Nezamabadi%2Dpour%2E+2021%2E+Idpl%2Dpfod%3A+An+image+dataset+of+printed+farsi+text+for+ocr+research%2E+In+Proceedings+of+the+Second+Internation)
64. Baoguang Shi, Xiang Bai, and Cong Yao. 2016. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304. [↗](https://doi.org/10.1109/tpami.2016.2646371)
65. Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. 2018. [find](https://scholar.google.com/scholar?q=Baoguang+Shi%2C+Mingkun+Yang%2C+Xinggang+Wang%2C+Pengyuan+Lyu%2C+Cong+Yao%2C+and+Xiang+Bai%2E+2018%2E)
66. Aster: An attentional scene text recognizer with flexible rectification. IEEE transactions on pattern analysis and machine intelligence, 41(9):2035–2048. [find](https://scholar.google.com/scholar?q=Aster%3A+An+attentional+scene+text+recognizer+with+flexible+rectification%2E+IEEE+transactions+on+pattern+analysis+and+machine+intelligence%2C+41%289%29%3A2035%E2%80%932048%2E)
67. Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. 2023. Exploring ocr capabilities of gpt-4v (ision): A quantitative and in-depth evaluation. arXiv preprint arXiv:2310.16809. [↗](https://arxiv.org/abs/2310.16809)
68. Amarjot Singh, Ketan Bacchuwar, and Akshay Bhasin. 2012. A survey of ocr applications. International Journal of Machine Learning and Computing, 2(3):314. [↗](https://doi.org/10.7763/ijmlc.2012.v2.137)
69. Mohamed Sobhi, Yasser Hifny, and Saleh Mesbah Elkaffas. 2020. Arabic optical character recognition using attention based encoder-decoder architecture. In 2020 2nd International Conference on Artificial Intelligence, Robotics and Control, pages 1–5. [find](https://scholar.google.com/scholar?q=Mohamed+Sobhi%2C+Yasser+Hifny%2C+and+Saleh+Mesbah+Elkaffas%2E+2020%2E+Arabic+optical+character+recognition+using+attention+based+encoder%2Ddecoder+architecture%2E+In+2020+2nd+International+Conference+on+Artificia)
70. Bolan Su and Shijian Lu. 2015. Accurate scene text recognition based on recurrent neural network. In Computer Vision–ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part I 12, pages 35–48. Springer. [find](https://scholar.google.com/scholar?q=Bolan+Su+and+Shijian+Lu%2E+2015%2E+Accurate+scene+text+recognition+based+on+recurrent+neural+network%2E+In+Computer+Vision%E2%80%93ACCV+2014%3A+12th+Asian+Conference+on+Computer+Vision%2C+Singapore%2C+Singapore%2C+November)
71. Jorge Sueiras, Victoria Ruiz, Angel Sanchez, and Jose F Velez. 2018. Offline continuous handwriting recognition using sequence to sequence neural networks. Neurocomputing, 289:119–128. [↗](https://doi.org/10.1016/j.neucom.2018.02.008)
72. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). Advances in neural information processing systems, 27. [↗](https://doi.org/10.48550/arxiv.1409.3215)
73. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805. [↗](https://arxiv.org/abs/2312.11805)
74. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2020. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). [↗](https://doi.org/10.48550/arxiv.2012.12877)
75. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
76. Paul Voigtlaender, Patrick Doetsch, and Hermann Ney. 2016. Handwriting recognition with large multidimensional long short-term memory recurrent neural networks. In 2016 15th international conference on frontiers in handwriting recognition (ICFHR), pages 228–233. IEEE. [find](https://scholar.google.com/scholar?q=Paul+Voigtlaender%2C+Patrick+Doetsch%2C+and+Hermann+Ney%2E+2016%2E+Handwriting+recognition+with+large+multidimensional+long+short%2Dterm+memory+recurrent+neural+networks%2E+In+2016+15th+international+conference+o)
77. Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. 2024. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models. arXiv preprint arXiv:2401.13311. [↗](https://arxiv.org/abs/2401.13311)
78. Christoph Wick, Jochen Zöllner, and Tobias Grüning. 2022. Rescoring sequence-to-sequence models for text line recognition with ctc-prefixes. In Document Analysis Systems: 15th IAPR International Workshop, DAS 2022, La Rochelle, France, May 22–25, 2022, Proceedings, pages 260–274. Springer. [find](https://scholar.google.com/scholar?q=Christoph+Wick%2C+Jochen+Z%C3%B6llner%2C+and+Tobias+Gr%C3%BCning%2E+2022%2E+Rescoring+sequence%2Dto%2Dsequence+models+for+text+line+recognition+with+ctc%2Dprefixes%2E+In+Document+Analysis+Systems%3A+15th+IAPR+International+Works)
79. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663. [find](https://scholar.google.com/scholar?q=Zhenda+Xie%2C+Zheng+Zhang%2C+Yue+Cao%2C+Yutong+Lin%2C+Jianmin+Bao%2C+Zhuliang+Yao%2C+Qi+Dai%2C+and+Han+Hu%2E+2022%2E+Simmim%3A+A+simple+framework+for+masked+image+modeling%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Co)
80. Mohamed Yousef, Khaled F Hussain, and Usama S Mohammed. 2020. Accurate, data-efficient, unconstrained text recognition with convolutional neural networks. Pattern Recognition, 108:107482. [↗](https://doi.org/10.1016/j.patcog.2020.107482)
81. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. [↗](https://doi.org/10.1109/tpami.2024.3369699)
82. Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. 2023. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107. [↗](https://arxiv.org/abs/2306.17107)

*Updated 1 November 2026*