---
title: "Taqyim: Evaluating Arabic NLP Tasks Using — Alyafeai 2023"
description: "The paper investigates how well general-purpose chat LLMs handle Arabic, a language that makes up a small share of their training data."
authors: ["Zaid Alyafeai", "Maged S. Alshaibani", "Badr AlKhamissi", "Hamzah Luqman", "Ebrahim Alareqi", "Ali Fadel"]
year: 2023
venue: "arXiv:2306.16322"
source: "https://arxiv.org/abs/2306.16322"
doi: "10.48550/arxiv.2306.16322"
arxiv: "2306.16322"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alyafeai-2023-taqyim-evaluating-arabic-nlp-tasks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alyafeai-2023-taqyim-evaluating-arabic-nlp-tasks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

# Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

**Authors:** [Zaid Alyafeai](https://md.research.elamur.ai/free/en/authors/zaid-alyafeai/), [Maged S. Alshaibani](https://md.research.elamur.ai/free/en/authors/maged-s-alshaibani/), [Badr AlKhamissi](https://md.research.elamur.ai/free/en/authors/badr-alkhamissi/), [Hamzah Luqman](https://md.research.elamur.ai/free/en/authors/hamzah-luqman/), Ebrahim Alareqi, [Ali Fadel](https://md.research.elamur.ai/free/en/authors/ali-fadel/) · **Year:** 2023 · **Venue:** arXiv:2306.16322 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.16322) · [PDF](https://arxiv.org/pdf/2306.16322) · [DOI 10.48550/arxiv.2306.16322](https://doi.org/10.48550/arxiv.2306.16322) · [arXiv 2306.16322](https://arxiv.org/abs/2306.16322) · [OpenAlex](https://openalex.org/W4382619177)

**Citations:** 9 citations (openalex-snapshot · 2026-10-14) · [16](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper investigates how well general-purpose chat LLMs handle Arabic, a language that makes up a small share of their training data. The authors ran GPT-3.5 and GPT-4, mostly zero-shot, on seven Arabic tasks: sentiment analysis, translation, transliteration (Arabizi to Egyptian Arabic), paraphrasing, part-of-speech tagging with UD tags, summarization, and diacritization. They compared the results with task-specific state-of-the-art systems.

Specialized Arabic models still win on most tasks. Summarization is the exception, where both chat models beat the prior best. GPT-4 is stronger than GPT-3.5 on five tasks, while GPT-3.5 leads on summarization and diacritization.

A deeper study of dialectal sentiment (AJGT) varies temperature, prompt wording and the number of demonstrations, and it inspects the raw model replies. The authors also release Taqyim, an open-source Python wrapper around OpenAI evals that makes such evaluations easier to run, resume and debug. The work gives a transparent, reproducible snapshot of where foundation models stand on Arabic and where gaps remain.

## Key points
- Seven tasks were tested on two models (GPT-3.5, GPT-4). GPT-4 is better on 5 of 7, and GPT-3.5 is better on summarization and diacritization.
- Only in summarization did the chat models surpass existing SoTA systems. Diacritization results are additionally broken down across seven domains.
- On the 360-sample AJGT test set, GPT-3.5 gave a usable label for only 342, 304, 317 and 312 samples at 0, 3, 5 and 10 shots.
  - Its failures included asking for an English version, explaining in Arabic, or refusing over offensive content.
  - Its zero-shot replies took more than 20 distinct forms.
- GPT-4 almost always answered with the two allowed labels. The one exception was an invented "Neutral" label in the 3-shot setting.
- Taqyim extends OpenAI evals with:
  - a Python interface;
  - Hugging Face datasets loading;
  - tiktoken-based context budgeting;
  - a resume_from_record option;
  - sequential requests so sample ids match dataset rows.

## Interesting ideas
- For POS tagging, the model receives an already tokenized sentence and is told not to merge, split or alter tokens, so its output can be aligned row by row with the gold data. — For Mova (learn): LLM teacher tiers could get our own en tokens fixed in the prompt, letting their tags or arcs be compared directly with witness parsers when building silver data.
- Instead of mapping every reply onto the label set, the authors build a confusion matrix over the raw responses, which exposes off-template answers, refusals and invented classes. — For Mova (learn): when grading LLM teachers or our own readers, counting the unexpected outputs separately would act as a cheap check for format drift and hidden failure modes.
- Requests are sent in order and runs can resume from the last saved record, which keeps results traceable to specific inputs and avoids paying twice after a crash. — For Mova (learn): long teacher-labelling jobs could log by stable sentence id and resume, making silver-data provenance auditable and errors easy to locate.
- Each prompt is tried in several phrasings and with varying temperature and number of demonstrations, so a score reflects robustness rather than one lucky prompt. — For Mova (learn): reporting teacher agreement across several prompt variants would show which silver labels are stable and which depend on wording.
- Diacritization scores are reported per domain rather than as one average, which reveals where performance actually breaks down. — For Mova (tree): evaluating parsers and dialect conversion per genre (news, fables, school texts) would show which text types need more treebank data or repair rules.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

16 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 13 not yet.

- [Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-domain-adaptation-arabic-machine-translation/)
- [Investigating Cultural Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-investigating-cultural-alignment-large-language/)
- [TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alyafeai2023taqyim,
  title = {{Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models}},
  author = {Alyafeai, Zaid and Alshaibani, Maged S. and AlKhamissi, Badr and Luqman, Hamzah and Alareqi, Ebrahim and Fadel, Ali},
  year = {2023},
  doi = {10.48550/arxiv.2306.16322},
  eprint = {2306.16322},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.16322}
}
```

- [← previous Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text](https://md.research.elamur.ai/free/en/papers/bahar-2023-take-hint-improving-arabic-diacritization/)
- [next → TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties](https://md.research.elamur.ai/free/en/papers/kadaoui-2023-tarjamat-evaluation-bard-chatgpt-machine/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+%282020%29%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+i)
2. Cascella, M., Montomoli, J., Bellini, V., and Elena, B. (2023). Evaluating the feasibility of chatgpt in healthcare: An analysis of multiple clinical and research scenarios. Journal of Medical Systems, 47. [find](https://scholar.google.com/scholar?q=Cascella%2C+M%2E%2C+Montomoli%2C+J%2E%2C+Bellini%2C+V%2E%2C+and+Elena%2C+B%2E+%282023%29%2E+Evaluating+the+feasibility+of+chatgpt+in+healthcare%3A+An+analysis+of+multiple+clinical+and+research+scenarios%2E+Journal+of+Medical+Systems)
3. Abdelali, A., Mubarak, H., Chowdhury, S. A., Hasanain, M., Mousi, B., Boughorbel, S., Kheir, Y. E., Izham, D., Dalvi, F., Hawasly, M., et al. (2023). Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982. [↗](https://arxiv.org/abs/2305.14982)
4. Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., and Song, Y. (2023). Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. arXiv preprint arXiv:2304.14827. [↗](https://arxiv.org/abs/2304.14827) [our card](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
5. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020). Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
6. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [↗](https://arxiv.org/abs/2204.02311)
7. Alghamdi, A., Duan, X., Jiang, W., Wang, Z., Wu, Y., Xia, Q., Wang, Z., Zheng, Y., Rezagholizadeh, M., Huai, B., et al. (2023). Aramus: Pushing the limits of data and model scale for arabic natural language processing. arXiv preprint arXiv:2306.06800. [↗](https://arxiv.org/abs/2306.06800)
8. Alian, M., Awajan, A., Al-Hasan, A., and Akuzhia, R. (2021). Building arabic paraphrasing benchmark based on transformation rules. pages 1–17. [find](https://scholar.google.com/scholar?q=Alian%2C+M%2E%2C+Awajan%2C+A%2E%2C+Al%2DHasan%2C+A%2E%2C+and+Akuzhia%2C+R%2E+%282021%29%2E+Building+arabic+paraphrasing+benchmark+based+on+transformation+rules%2E+pages+1%E2%80%9317%2E)
9. AlKhamissi, B., ElNokrashy, M., and Gabr, M. (2020). Deep diacritization: Efficient hierarchical recurrence for improved Arabic diacritization. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 38–48, Barcelona, Spain (Online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=AlKhamissi%2C+B%2E%2C+ElNokrashy%2C+M%2E%2C+and+Gabr%2C+M%2E+%282020%29%2E+Deep+diacritization%3A+Efficient+hierarchical+recurrence+for+improved+Arabic+diacritization%2E+In+Proceedings+of+the+Fifth+Arabic+Natural+Language+Proc)
10. Alomari, K. M., ElSherif, H. M., and Shaalan, K. (2017). Arabic tweets sentimental analysis using machine learning. In Advances in Artificial Intelligence: From Theory to Practice: 30th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2017, Arras, France, June 27-30, 2017, Proceedings, Part I 30, pages 602–610. Springer. [find](https://scholar.google.com/scholar?q=Alomari%2C+K%2E+M%2E%2C+ElSherif%2C+H%2E+M%2E%2C+and+Shaalan%2C+K%2E+%282017%29%2E+Arabic+tweets+sentimental+analysis+using+machine+learning%2E+In+Advances+in+Artificial+Intelligence%3A+From+Theory+to+Practice%3A+30th+International+)
11. Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023). A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023. [↗](https://arxiv.org/abs/2302.04023) [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
12. Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., Strassel, S., Habash, N., Eskander, R., and Rambow, O. (2014). Transliteration of Arabizi into Arabic orthography: Developing a parallel annotated Arabizi-Arabic script SMS/chat corpus. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 93–103, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bies%2C+A%2E%2C+Song%2C+Z%2E%2C+Maamouri%2C+M%2E%2C+Grimes%2C+S%2E%2C+Lee%2C+H%2E%2C+Wright%2C+J%2E%2C+Strassel%2C+S%2E%2C+Habash%2C+N%2E%2C+Eskander%2C+R%2E%2C+and+Rambow%2C+O%2E+%282014%29%2E+Transliteration+of+Arabizi+into+Arabic+orthography%3A+Developing+a+paral)
13. Bommasani et al., R. (2021). On the opportunities and risks of foundation models. ArXiv, abs/2108.07258. [↗](https://arxiv.org/abs/2108.07258) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
14. Darwish, K., Mubarak, H., and Abdelali, A. (2017). Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17, Valencia, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Darwish%2C+K%2E%2C+Mubarak%2C+H%2E%2C+and+Abdelali%2C+A%2E+%282017%29%2E+Arabic+diacritization%3A+Stats%2C+rules%2C+and+hacks%2E+In+Proceedings+of+the+Third+Arabic+Natural+Language+Processing+Workshop%2C+pages+9%E2%80%9317%2C+Valencia%2C+Spain%2E) [our card](https://md.research.elamur.ai/free/en/papers/aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks/)
15. Davis, E. (2023). Mathematics, word problems, common sense, and artificial intelligence. arXiv preprint arXiv:2301.09723. [↗](https://arxiv.org/abs/2301.09723)
16. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
17. El-Haj, M., Kruschwitz, U., and Fox, C. (2010). Using mechanical turk to create a corpus of arabic summaries. [find](https://scholar.google.com/scholar?q=El%2DHaj%2C+M%2E%2C+Kruschwitz%2C+U%2E%2C+and+Fox%2C+C%2E+%282010%29%2E+Using+mechanical+turk+to+create+a+corpus+of+arabic+summaries%2E)
18. Elmadany, A., Abdul-Mageed, M., et al. (2022a). Arat5: Text-to-text transformers for arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647. [find](https://scholar.google.com/scholar?q=Elmadany%2C+A%2E%2C+Abdul%2DMageed%2C+M%2E%2C+et+al%2E+%282022a%29%2E+Arat5%3A+Text%2Dto%2Dtext+transformers+for+arabic+language+generation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/)
19. Elmadany, A., Nagoudi, E. M. B., and Abdul-Mageed, M. (2022b). Orca: A challenging benchmark for arabic language understanding. ArXiv, abs/2212.10758. [↗](https://arxiv.org/abs/2212.10758)
20. Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. (2023). Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867. [↗](https://arxiv.org/abs/2301.13867) [our card](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/)
21. Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022). Revisiting pre-trained language models and their evaluation for arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151. [find](https://scholar.google.com/scholar?q=Ghaddar%2C+A%2E%2C+Wu%2C+Y%2E%2C+Bagga%2C+S%2E%2C+Rashid%2C+A%2E%2C+Bibi%2C+K%2E%2C+Rezagholizadeh%2C+M%2E%2C+Xing%2C+C%2E%2C+Wang%2C+Y%2E%2C+Duan%2C+X%2E%2C+Wang%2C+Z%2E%2C+et+al%2E+%282022%29%2E+Revisiting+pre%2Dtrained+language+models+and+their+evaluation+for+arabic+)
22. Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., and Chartash, D. (2022). How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment. medRxiv, pages 2022–12. [find](https://scholar.google.com/scholar?q=Gilson%2C+A%2E%2C+Safranek%2C+C%2E%2C+Huang%2C+T%2E%2C+Socrates%2C+V%2E%2C+Chi%2C+L%2E%2C+Taylor%2C+R%2E+A%2E%2C+and+Chartash%2C+D%2E+%282022%29%2E+How+well+does+chatgpt+do+when+taking+the+medical+licensing+exams%3F+the+implications+of+large+language)
23. Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., and Wu, Y. (2023). How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. [↗](https://arxiv.org/abs/2301.07597) [our card](https://md.research.elamur.ai/free/en/papers/guo-2023-close-chatgpt-human-experts-comparison/)
24. Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., and Awadalla, H. H. (2023). How good are gpt models at machine translation? a comprehensive evaluation. Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hendy%2C+A%2E%2C+Abdelrehim%2C+M%2E%2C+Sharaf%2C+A%2E%2C+Raunak%2C+V%2E%2C+Gabr%2C+M%2E%2C+Matsushita%2C+H%2E%2C+Kim%2C+Y%2E+J%2E%2C+Afify%2C+M%2E%2C+and+Awadalla%2C+H%2E+H%2E+%282023%29%2E+How+good+are+gpt+models+at+machine+translation%3F+a+comprehensive+evaluati)
25. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022). Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786. [↗](https://arxiv.org/abs/2211.01786) [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
26. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+the+North+America)
27. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. [↗](https://arxiv.org/abs/2203.15556)
28. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. (2023). Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476. [↗](https://arxiv.org/abs/2302.06476) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/)
29. Jiao, W., Wang, W., Huang, J., Wang, X., and Tu, Z. (2023). Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.08745. [↗](https://arxiv.org/abs/2301.08745) [our card](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
30. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%2C+et+al%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
31. Kondratyuk, D. and Straka, M. (2019). 75 languages, 1 model: Parsing universal dependencies universally. arXiv preprint arXiv:1904.02099. [↗](https://arxiv.org/abs/1904.02099) [our card](https://md.research.elamur.ai/free/en/papers/ud-2019-kondratyuk-75-languages-1-model-parsing/)
32. Ladhak, F., Durmus, E., Cardie, C., and McKeown, K. (2020). Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093. [↗](https://arxiv.org/abs/2010.03093) [our card](https://md.research.elamur.ai/free/en/papers/ladhak-2020-wikilingua-new-benchmark-dataset-cross/)
33. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [↗](https://arxiv.org/abs/1910.13461) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
34. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., and Ge, B. (2023). Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Han%2C+T%2E%2C+Ma%2C+S%2E%2C+Zhang%2C+J%2E%2C+Yang%2C+Y%2E%2C+Tian%2C+J%2E%2C+He%2C+H%2E%2C+Li%2C+A%2E%2C+He%2C+M%2E%2C+Liu%2C+Z%2E%2C+Wu%2C+Z%2E%2C+Zhu%2C+D%2E%2C+Li%2C+X%2E%2C+Qiang%2C+N%2E%2C+Shen%2C+D%2E%2C+Liu%2C+T%2E%2C+and+Ge%2C+B%2E+%282023%29%2E+Summary+of+chatgpt%2Fgpt%2D4+research+an)
35. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pre-training approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
36. Lu, H., Huang, H., Ma, S., Zhang, D., Lam, W., and Wei, F. (2022). Trip: Triangular document-level pre-training for multilingual language models. arXiv preprint arXiv:2212.07752. [↗](https://arxiv.org/abs/2212.07752)
37. Mubarak, H., Abdelali, A., Sajjad, H., Samih, Y., and Darwish, K. (2019). Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395, Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. [↗](https://arxiv.org/abs/2110.08207) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
38. Shazal, A., Usman, A., and Habash, N. (2020). A unified model for arabizi detection and transliteration using sequence-to-sequence models. In Proceedings of the fifth arabic natural language processing workshop, pages 167–177. [find](https://scholar.google.com/scholar?q=Shazal%2C+A%2E%2C+Usman%2C+A%2E%2C+and+Habash%2C+N%2E+%282020%29%2E+A+unified+model+for+arabizi+detection+and+transliteration+using+sequence%2Dto%2Dsequence+models%2E+In+Proceedings+of+the+fifth+arabic+natural+language+processin)
39. Smrz, O., Bielickỳ, V., Kourilová, I., Krácmar, J., Hajic, J., and Zemánek, P. (2008). Prague arabic dependency treebank: A word on the million words. In Proceedings of the workshop on Arabic and local languages (LREC 2008), pages 16–23. [find](https://scholar.google.com/scholar?q=Smrz%2C+O%2E%2C+Bielicky%CC%80%2C+V%2E%2C+Kourilov%C3%A1%2C+I%2E%2C+Kr%C3%A1cmar%2C+J%2E%2C+Hajic%2C+J%2E%2C+and+Zem%C3%A1nek%2C+P%2E+%282008%29%2E+Prague+arabic+dependency+treebank%3A+A+word+on+the+million+words%2E+In+Proceedings+of+the+workshop+on+Arabic+and+loc)
40. Tawkat Islam Khondaker, M., Waheed, A., Moatez Billah Nagoudi, E., and Abdul-Mageed, M. (2023). Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp. arXiv e-prints, pages arXiv–2305. [find](https://scholar.google.com/scholar?q=Tawkat+Islam+Khondaker%2C+M%2E%2C+Waheed%2C+A%2E%2C+Moatez+Billah+Nagoudi%2C+E%2E%2C+and+Abdul%2DMageed%2C+M%2E+%282023%29%2E+Gptaraeval%3A+A+comprehensive+evaluation+of+chatgpt+on+arabic+nlp%2E+arXiv+e%2Dprints%2C+pages+arXiv%E2%80%932305%2E)
41. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [↗](https://arxiv.org/abs/2302.13971)
42. Wang, T., Roberts, A., Hesslow, D., Le Scao, T., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. (2022). What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pages 22964–22984. PMLR. [find](https://scholar.google.com/scholar?q=Wang%2C+T%2E%2C+Roberts%2C+A%2E%2C+Hesslow%2C+D%2E%2C+Le+Scao%2C+T%2E%2C+Chung%2C+H%2E+W%2E%2C+Beltagy%2C+I%2E%2C+Launay%2C+J%2E%2C+and+Raffel%2C+C%2E+%282022%29%2E+What+language+model+architecture+and+pretraining+objective+works+best+for+zero%2Dshot+gener)
43. Zeman et al., D. (2020). Universal dependencies 2.7. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. [find](https://scholar.google.com/scholar?q=Zeman+et+al%2E%2C+D%2E+%282020%29%2E+Universal+dependencies+2%2E7%2E+LINDAT%2FCLARIAH%2DCZ+digital+library+at+the+Institute+of+Formal+and+Applied+Linguistics+%28%C3%9AFAL%29%2C+Faculty+of+Mathematics+and+Physics%2C+Charles+University)
44. Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223. [↗](https://arxiv.org/abs/2303.18223) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
45. Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016). The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534. [find](https://scholar.google.com/scholar?q=Ziemski%2C+M%2E%2C+Junczys%2DDowmunt%2C+M%2E%2C+and+Pouliquen%2C+B%2E+%282016%29%2E+The+united+nations+parallel+corpus+v1%2E+0%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9916) [our card](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/)
46. Zitouni, I. and Sarikaya, R. (2009). Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23:257–276. [find](https://scholar.google.com/scholar?q=Zitouni%2C+I%2E+and+Sarikaya%2C+R%2E+%282009%29%2E+Arabic+diacritic+restoration+approach+based+on+maximum+entropy+models%2E+Computer+Speech+%26+Language%2C+23%3A257%E2%80%93276%2E)
47. Šlapeta, J. (2023). Are chatgpt and other pretrained language models good parasitologists? Trends in Parasitology, 39(5):314–316. *Figure 8: GPT-3.5 responses to AJGT dataset confusion matrix with 3 few-shot examples. Numbers are removed as most of them, except the highlighted areas, are zeros. Also, responses are removed for clarity purposes* [find](https://scholar.google.com/scholar?q=%C5%A0lapeta%2C+J%2E+%282023%29%2E+Are+chatgpt+and+other+pretrained+language+models+good+parasitologists%3F+Trends+in+Parasitology%2C+39%285%29%3A314%E2%80%93316%2E+%2AFigure+8%3A+GPT%2D3%2E5+responses+to+AJGT+dataset+confusion+matrix+with+3+f)