--- title: "AMCrawl — paper by Aboukozzana et al. (2025)" description: "Most vision-language pretraining relies on image-text pairs scraped from alt-text. 1 reference." authors: ["Shahad Aboukozzana", "Muhammad Kamran J Khan", "Ahmed Ali"] year: 2025 venue: "Proceedings of The Third Arabic Natural Language Processing Conference" source: "https://aclanthology.org/2025.arabicnlp-main.37/" doi: "10.18653/v1/2025.arabicnlp-main.37" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/aboukozzana-2025-amcrawl-arabic-web-scale-dataset/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aboukozzana-2025-amcrawl-arabic-web-scale-dataset/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/16/) / AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs # AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs **Authors:** Shahad Aboukozzana, Muhammad Kamran J Khan and [Ahmed Ali](https://md.research.elamur.ai/free/en/authors/ahmed-m-abdelatty-ali/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-main.37/) · [PDF](https://aclanthology.org/2025.arabicnlp-main.37.pdf) · [DOI 10.18653/v1/2025.arabicnlp-main.37](https://doi.org/10.18653/v1/2025.arabicnlp-main.37) · [OpenAlex](https://openalex.org/W4416037662) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Most vision-language pretraining relies on image-text pairs scraped from alt-text. These captions are often empty, are just file names, or are short and unrelated to the image. Interleaved documents, where images sit inside running text, give much richer context, but public resources of this kind (MMC4, OBELICS, MINT-1T) are essentially English-only. The authors adapt the OBELICS pipeline to Arabic and run it over a CommonCrawl snapshot. They filter pages for quality and safety and keep each image in its place within the document layout. They then prompt GPT to turn the documents into question-answer pairs tied to images, and they also produce Arabic translations of several standard multimodal datasets. To test the resource, they train a 13B-parameter Arabic VLM in two stages and score its answers with GPT-4o as a judge. Fine-tuning on AMCrawl-QA clearly beats fine-tuning only on translated English data. The released data and code give Arabic multimodal research a native, culturally grounded foundation instead of relying solely on translation. ## Key points - Scale: 8.6M Arabic web pages were processed, giving 5.8M images and 1.3B text tokens. The final release has 2.8M interleaved documents and 5M QA pairs (AMCrawl-QA). - Training recipe: - Stage 1 aligns only the projection layer on Arabic-translated CC3M-595K and MSCOCO (about 1.5M samples). Loss fell from 7.54 to 1.74. - Stage 2 tunes the projection layer and the LLM on AMCrawl QA, with the vision encoder frozen. Loss fell from 1.77 to 0.08. - Evaluation follows the LLaVA protocol on 45k sampled test QA pairs, with GPT-4o as judge. The mean score rose from 32.3 (MIMIC+LLaVA data) to 43.6 after AMCrawl fine-tuning. - Safety filtering had two steps: - URLs were screened for explicit keywords before download. - Downloaded images were checked by a MobileNet-based NSFW classifier, and any document with a flagged image was dropped. This removed 44,701 documents in total. - Stated limitations: - The data comes from a single snapshot (June 2024). - Perceptual hashing suggests only about 45% of images are unique. - No human evaluation and no standard benchmarks yet; Henna and CVQA testing is ongoing. ## Interesting ideas - Instead of trusting a short attached label, the pipeline uses the page's DOM structure to keep each item embedded in its real surrounding context, which carries far more meaning than a caption. - Grounded QA pairs are generated automatically by an LLM from real documents, turning unlabeled web pages into instruction data at scale. - The unit of rejection is the whole document: one flagged component discards everything around it, trading some yield for cleaner data. - Training proceeds in stages: first only a small bridging layer is fitted while the big components stay frozen, then more of the model is unfrozen. - The authors openly measured near-duplicates with perceptual hashing and reported low uniqueness rather than hiding it. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): when collecting raw text for coreference and role training, keep whole documents with their paragraph structure rather than isolated sentences, so pronouns and event chains have their true antecedents. - **I2** (answer): have an LLM teacher draft questions over fables and tales, then keep only those whose answers Mova's world-of-the-text can actually derive, yielding explainable training and test items. - **I3** (learn): if any sentence in a document fails witness-parser agreement badly, drop the full document from silver data so context-dependent errors do not leak in. - **I4** (tree): when attaching a new head or a new UD dialect to gparse, first train only the head on a frozen encoder, then fine-tune jointly, which stabilizes early training. - **I5** (learn): hash normalized sentences across treebanks and silver corpora to quantify duplication and to guarantee that no evaluation sentence leaks into training. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works ## Cite Cite as: Shahad Aboukozzana, Muhammad Kamran J Khan and Ahmed Ali. 2025. AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs. Proceedings of The Third Arabic Natural Language Processing Conference. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{aboukozzana2025amcrawl, title = {{AMCrawl: An Arabic Web-Scale Dataset of Interleaved Image-Text Documents and Image-Text Pairs}}, author = {Shahad Aboukozzana, Muhammad Kamran J Khan and Ahmed Ali}, booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference}, year = {2025}, doi = {10.18653/v1/2025.arabicnlp-main.37}, url = {https://doi.org/10.18653/v1/2025.arabicnlp-main.37} } ``` - [← previous Amado at SemEval-2025 Task 11: Multi-label Emotion Detection in Amharic and English Data](https://md.research.elamur.ai/free/en/papers/bade-2025-amado-semeval-2025-task-11/) - [next → Amharic News Topic Classification: Dataset and Transformer-Based Model Benchmarks](https://md.research.elamur.ai/free/en/papers/marilign-2025-amharic-news-topic-classification-dataset/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/16/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work. 1. Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas Guibas. 2021. Artemis: Affective language for visual art. CoRR, abs/2101.07396. [↗](https://arxiv.org/abs/2101.07396) *Updated 2 November 2026*