---
title: "TafsirExtractor — paper by Kruse and Ahmed (2024)"
description: "The authors present Tafsir Extractor, a modular pipeline with four stages. 19 references."
authors: ["Carl Kruse", "Sajawel Ahmed"]
year: 2024
venue: "Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024"
source: "https://aclanthology.org/2024.osact-1.8/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kruse-2024-tafsirextractor-text-preprocessing-pipeline-preparing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kruse-2024-tafsirextractor-text-preprocessing-pipeline-preparing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / TafsirExtractor: Text Preprocessing Pipeline preparing Classical Arabic Literature for Machine Learning Applications

# TafsirExtractor: Text Preprocessing Pipeline preparing Classical Arabic Literature for Machine Learning Applications

**Authors:** Carl Kruse and Sajawel Ahmed · **Year:** 2024 · **Venue:** [Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024](https://md.research.elamur.ai/free/en/venues/osact/) ([Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024](https://md.research.elamur.ai/free/en/venues/osact/2024/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.osact-1.8/) · [PDF](https://aclanthology.org/2024.osact-1.8.pdf) · [OpenAlex](https://openalex.org/W7131792163)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Classical Arabic exegetical literature, such as the huge Quran commentary of Al-Tabari, lacks tooling that turns expert-annotated texts into the formats modern ML models need. Existing Arabic toolkits (CAMeL tools, Stanza, MADAMIRA, Farasa) mostly target Modern Standard Arabic and cannot convert XML/TEI annotations into CoNLL. The authors present Tafsir Extractor, a modular pipeline with four stages. First, the Tabari text is taken from the OCR-based Gawami' al-Kalim platform. GK2TEI then converts that platform's markup into TEI XML, where experts annotate named entities, topics and subtopics by hand. TEI2CoNLL turns these annotations into an extended CoNLL matrix, and CoNLL-Reader produces filtered variants of the data. A separate routine crawls OpenITI and builds a one-sentence-per-line corpus for training embeddings from scratch. The output supports NER and topic modeling, either with large models through MaChAmp or with lightweight embedding models. The authors report that this enables state-of-the-art results on Classical Arabic. The tool, code and data are released openly, filling a gap for this low-resource historical domain.

## Key points
- The pipeline has four stages: OCR source text, then GK2TEI, then TEI2CoNLL, then CoNLL-Reader. It was applied to the entire Tabari corpus, and annotation was done in the Oxygen XML Editor.
- TEI2CoNLL reads specific TEI elements:
  - `<persName>` for the Isnad (chain of transmitters);
  - `<name>` for named entities;
  - `<p ana=…>` for topics and `<seg ana=…>` for subtopics;
  - `<said>` for the Matn (the transmitted text).
- Output format:
  - Entities are converted to BIO tags with the types PER, ORG, LOC, TME and OTH.
  - Topics are marked per sentence as 1/0, with 'nil' for undefined topics.
  - Subtopics are span-based BIO tags.
- User flags control the output: whether to include NEs or Isnad, which topics and subtopics to keep, and the order of the functions. Overlong sentences are split with heuristics from Schweter and Ahmed (2019).
- CoNLL-Reader can strip or alter punctuation, diacritics (tashkeel) and individual letters, which supports the authors' "script-compression" work. The OpenITI text is segmented with CAMeL tools tokenization. Everything is released at github.com/sa-j/ArabicNLP.

## Interesting ideas
- The authors keep rich gold annotation in one structured source format (TEI) and generate several task-specific training views from it with flags, instead of maintaining separate datasets per task.
- A single CoNLL-style row can carry several annotation layers side by side: sentence-level binary topic flags, span-level BIO subtopics and token-level entity tags.
- Training variants are made by systematically removing script features such as diacritics, punctuation or certain letters, to see how much the models depend on them.
- Structural elements of the text, such as the transmitter chain versus the transmitted content, can be included or excluded as a switch. This lets the same corpus test whether framing material helps or hurts a task.
- The authors explicitly flag context-dependent ambiguity, where one word may be a common noun or a personal name (e.g. "madina", "mansur").

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver and gold data could be stored once as full annotated trees and frames, with exporters deriving parser-, tagger- or role-specific training files on demand, which keeps versions consistent.
- **I2** (world): Entity mentions, state annotations and discourse-level labels could share one token-aligned file with UD, so world-building errors can be traced back to exact tokens.
- **I3** (tokens): Ablated copies of the input (lowercased, depunctuated, contractions expanded) can test how robust the English tagger and lemmatizer are and expose hidden reliance on surface cues.
- **I4** (frames): Reported-speech wrappers ("X said that …") could be toggled in evaluation, to check whether speech-act and role extraction holds up with and without the attribution layer.
- **I5** (world): Entity resolution should defer the choice between name and common noun until syntactic and discourse context is available, rather than fixing it from a lexicon.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [📚 Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) · 340 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 1,484 works

## Cite

Cite as: Carl Kruse and Sajawel Ahmed. 2024. TafsirExtractor: Text Preprocessing Pipeline preparing Classical Arabic Literature for Machine Learning Applications. Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kruse2024tafsirextractor,
  title = {{TafsirExtractor: Text Preprocessing Pipeline preparing Classical Arabic Literature for Machine Learning Applications}},
  author = {Carl Kruse and Sajawel Ahmed},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/4nwr7n98cr7a},
  url = {https://doi.org/10.63317/4nwr7n98cr7a}
}
```

- [← previous Syntagmatic and Paradigmatic Analysis of Lexemes for ‘Rain’ in Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/pratesi-2024-syntagmatic-paradigmatic-analysis-lexemes-rain/)
- [next → Teaching an Arabic Dialect to Non-Native Learner in the Mid-Fifteenth Century Case Study: Al-Balāṭī Manuscript](https://md.research.elamur.ai/free/en/papers/al-hamad-2024-teaching-arabic-dialect-non-native/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2016-farasa-fast-furious-segmenter-arabic/)
2. Sajawel Ahmed, Misbahur Rehman, Joshua Tisch-lik, Carl Kruse, Edin Mahmutovic, and Ömer Özsoy. 2022a. Linked Open Tafsir—Rekonstruktion der Entstehungsdynamik (en) des Korans mithilfe der Netzwerkmodellierung früher islamischer Überlieferungen. In 8. Jahrestagung des Verbandes Digital Humanities im deutschsprachigen Raum (DHd). [find](https://scholar.google.com/scholar?q=Sajawel+Ahmed%2C+Misbahur+Rehman%2C+Joshua+Tisch%2Dlik%2C+Carl+Kruse%2C+Edin+Mahmutovic%2C+and+%C3%96mer+%C3%96zsoy%2E+2022a%2E+Linked+Open+Tafsir%E2%80%94Rekonstruktion+der+Entstehungsdynamik+%28en%29+des+Korans+mithilfe+der+Netzwerkmode)
3. Sajawel Ahmed, Rob van der Goot, Misbahur Rehman, Carl Kruse, Ömer Özsoy, Alexander Mehler, and Gemma Roig. 2022b. Tafsir dataset: A novel multi-task benchmark for named entity recognition and topic modeling in classical Arabic literature. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3753–3768, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sajawel+Ahmed%2C+Rob+van+der+Goot%2C+Misbahur+Rehman%2C+Carl+Kruse%2C+%C3%96mer+%C3%96zsoy%2C+Alexander+Mehler%2C+and+Gemma+Roig%2E+2022b%2E+Tafsir+dataset%3A+A+novel+multi%2Dtask+benchmark+for+named+entity+recognition+and+topic+m)
4. Jalal Al-Din Al-Suyuti. 1505. Al-itqan Fi ’ulum Al-Qur’an (The Perfect Guide to the Sciences of the Qu’ran). Garnet Publishing; Bilingual edition (May 1, 2012). [https://tafsirtabari.com/about](https://tafsirtabari.com/about) [↗](https://tafsirtabari.com/about)
5. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. [↗](https://doi.org/10.1162/tacl_a_00041)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, NA. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
8. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural Architectures for Named Entity Recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1603.01360) [our card](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/)
9. Wang Ling, Chris Dyer, Alan Black, and Isabel Trancoso. 2015. Two/Too Simple Adaptations of word2vec for Syntax Problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-two-too-simple-adaptations-word2vec/)
10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [↗](https://doi.org/10.48550/arxiv.1310.4546) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
11. Matthew Thomas Miller, Maxim G. Romanov, and Sarah Bowen Savant. 2018. Digitizing the textual heritage of the premodern islamicate world: Principles and plans. International Journal of Middle East Studies, 50(1):103–109. [↗](https://doi.org/10.1017/s0020743817000964)
12. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. [our card](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/)
13. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland. European Language Resources Association (ELRA). [our card](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/)
14. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. [↗](https://doi.org/10.18653/v1/2020.acl-demos.14) [our card](https://md.research.elamur.ai/free/en/papers/qi-2020-stanza-python-natural-language-processing/)
15. Stefan Schweter and Sajawel Ahmed. 2019. Deep-EOS: General-Purpose Neural Networks for Sentence Boundary Detection. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS). [find](https://scholar.google.com/scholar?q=Stefan+Schweter+and+Sajawel+Ahmed%2E+2019%2E+Deep%2DEOS%3A+General%2DPurpose+Neural+Networks+for+Sentence+Boundary+Detection%2E+In+Proceedings+of+the+15th+Conference+on+Natural+Language+Processing+%28KONVENS%29%2E)
16. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Abdulhadi+Shoufan+and+Sumaya+Alameri%2E+2015%2E+Natural+language+processing+for+dialectical+Arabic%3A+A+survey%2E+In+Proceedings+of+the+Second+Workshop+on+Arabic+Natural+Language+Processing%2C+pages+36%E2%80%9348%2C+Beij)
17. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. [↗](https://doi.org/10.48550/arxiv.cs/0306050) [our card](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/)
18. Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176–197, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.eacl-demos.22) [our card](https://md.research.elamur.ai/free/en/papers/goot-2021-massive-choice-ample-tasks-machamp/)
19. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pre-training for language understanding. In Advances in Neural Information Processing Systems, volume 32, NA. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pre%2Dtraining+for+language+understanding%2E+In+Advances+in+Neural+Infor)

*Updated 24 October 2026*