---
title: "ParsiNLU — paper by Khashabi et al. (2021)"
description: "Persian is spoken by tens of millions of people, yet it has had very few benchmarks for high-level… 71 references, cited by 46 papers here."
authors: ["Daniel Khashabi", "Arman Cohan", "Siamak Shakeri", "Pedram Hosseini", "Pouya Pezeshkpour", "Malihe Alikhani", "Moin Aminnaseri", "Marzieh Bitaab", "Faeze Brahman", "Sarik Ghazarian", "Mozhdeh Gheini", "Arman Kabiri", "Rabeeh Karimi Mahabadi", "Omid Memarrast", "Ahmadreza Mosallanezhad", "Erfan Noury", "Shahab Raji", "Mohammad Sadegh Rasooli", "Sepideh Sadeghi", "Erfan Sadeqi Azer", "Niloofar Safi Samghabadi", "Mahsa Shafaei", "Saber Sheybani", "Ali Tazarv", "Yadollah Yaghoobzadeh"]
year: 2021
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2021.tacl-1.68/"
doi: "10.1162/tacl_a_00419"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khashabi-2021-parsinlu-suite-language-understanding-challenges/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khashabi-2021-parsinlu-suite-language-understanding-challenges/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / ParsiNLU: A Suite of Language Understanding Challenges for Persian

# ParsiNLU: A Suite of Language Understanding Challenges for Persian

**Authors:** [Daniel Khashabi](https://md.research.elamur.ai/free/en/authors/daniel-khashabi/), [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/), [Siamak Shakeri](https://md.research.elamur.ai/free/en/authors/siamak-shakeri/), Pedram Hosseini, [Pouya Pezeshkpour](https://md.research.elamur.ai/free/en/authors/pouya-pezeshkpour/), [Malihe Alikhani](https://md.research.elamur.ai/free/en/authors/malihe-alikhani/), Moin Aminnaseri, Marzieh Bitaab, [Faeze Brahman](https://md.research.elamur.ai/free/en/authors/faeze-brahman/), Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, [Rabeeh Karimi Mahabadi](https://md.research.elamur.ai/free/en/authors/rabeeh-karimi-mahabadi/), Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv and [Yadollah Yaghoobzadeh](https://md.research.elamur.ai/free/en/authors/yadollah-yaghoobzadeh/) · **Year:** 2021 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2021](https://md.research.elamur.ai/free/en/venues/tacl/2021/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.tacl-1.68/) · [PDF](https://aclanthology.org/2021.tacl-1.68.pdf) · [DOI 10.1162/tacl\_a\_00419](https://doi.org/10.1162/tacl_a_00419) · [OpenAlex](https://openalex.org/W3213458975)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/khashabi-2021-parsinlu-suite-language-understanding-challenges/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Persian is spoken by tens of millions of people, yet it has had very few benchmarks for high-level language understanding. Most earlier Persian evaluation resources cover low-level tasks such as POS tagging, NER and parsing, and major multilingual suites largely ignore the language. ParsiNLU fills this gap with a benchmark of six tasks:
- reading comprehension
- multiple-choice question answering
- textual entailment
- sentiment analysis
- question paraphrasing
- machine translation

The authors avoided relying mainly on automatic translation of English data. Instead, they built most instances through native-speaker annotation and new collection strategies, for example search-engine auto-complete queries and questions from past university entrance exams. The resulting benchmark has over 14.5k new instances.

They then tested multilingual and monolingual pre-trained models, including the mT5 family. Models were trained on Persian data, English data, or a balanced mix of both, and compared against estimated human scores. Humans score high, while most mid-sized models trail well behind. Only the very large mT5-XL comes near human level, and only on some tasks. The benchmark gives Persian NLU a shared yardstick and reveals clear weaknesses, especially in reasoning.

## Key points
- ParsiNLU contains 14.5k+ new instances across 6 tasks and targets standard Iranian Persian; Dari and Tajiki are not covered.
- To the authors' knowledge, no public Persian datasets existed before this for reading comprehension, question answering or paraphrasing.
- Sentiment data adds aspect-level labels on a five-point intensity scale, covering food & beverages and movie reviews.
- On multiple-choice QA, the gap between models and humans exceeds 40%. Scaling up the models barely helps, especially on literature and math & logic questions.
- English-trained mT5 transfers well to Persian, particularly for subsets derived from English data (qqp, mnli). Joint English+Persian training usually helps, but not always.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations
- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 162 works · 35,836 citations
- [University of Pittsburgh](https://md.research.elamur.ai/free/en/universities/university-of-pittsburgh/) — University, Pittsburgh, United States · 59 works · 22,324 citations
- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 61 works · 20,107 citations
- [University of California, Irvine](https://md.research.elamur.ai/free/en/universities/university-of-california-irvine/) — University, Irvine, United States · 37 works · 36,319 citations
- [École Polytechnique Fédérale de Lausanne](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique-federale-de-lausanne/) — University, Lausanne, Switzerland · 44 works · 7,050 citations
- [George Washington University](https://md.research.elamur.ai/free/en/universities/george-washington-university/) — University, Washington, United States · 51 works · 8,148 citations
- [Rutgers, The State University of New Jersey](https://md.research.elamur.ai/free/en/universities/rutgers-the-state-university-of-new-jersey/) — University, New Brunswick, United States · 36 works · 12,026 citations
- [University of California, Santa Cruz](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-cruz/) — University, Santa Cruz, United States · 36 works · 3,515 citations
- [Indiana University Bloomington](https://md.research.elamur.ai/free/en/universities/indiana-university-bloomington/) — University, Bloomington, United States · 37 works · 3,685 citations
- [Arizona State University](https://md.research.elamur.ai/free/en/universities/arizona-state-university/) — University, Tempe, United States · 50 works · 2,032 citations
- [Indiana University](https://md.research.elamur.ai/free/en/universities/indiana-university/) — University, Bloomington, United States · 41 works · 3,663 citations
- [University of Maryland, Baltimore County](https://md.research.elamur.ai/free/en/universities/university-of-maryland-baltimore-county/) — University, Baltimore, United States · 14 works · 915 citations
- [Expedia Group](https://md.research.elamur.ai/free/en/companies/expedia-group/) — Company, Bellevue, United States · 2 works · 66 citations

Also on the byline: [Microsoft (Canada)](https://ror.org/04xhxg104).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv and Yadollah Yaghoobzadeh. 2021. ParsiNLU: A Suite of Language Understanding Challenges for Persian. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{khashabi2021parsinlu,
  title = {{ParsiNLU: A Suite of Language Understanding Challenges for Persian}},
  author = {Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv and Yadollah Yaghoobzadeh},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00419},
  url = {https://doi.org/10.1162/tacl_a_00419}
}
```

- [← previous Evaluating Various Tokenizers for Arabic Text Classification](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/)
- [next → Benchmarking Transformer-based Language Models for Arabic Sentiment and Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/farha-2021-benchmarking-transformer-based-language-models/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/)

## References

71 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hossein Amirkhani, Mohammad Azari Jafari, Azadeh Amirak, Zohreh Pourjafari, Soroush Faridan Jahromi, and Zeinab Kouhkan. 2020. Farstail: A Persian natural language inference dataset. arXiv preprint arXiv:2009.08820. — [arXiv:2009.08820](https://arxiv.org/abs/2009.08820) [find](https://scholar.google.com/scholar?q=Hossein+Amirkhani%2C+Mohammad+Azari+Jafari%2C+Azadeh+Amirak%2C+Zohreh+Pourjafari%2C+Soroush+Faridan+Jahromi%2C+and+Zeinab+Kouhkan%2E+2020%2E+Farstail%3A+A+Persian+natural+language+inference+dataset%2E+arXiv+preprint+ar)
2. Stefanos Angelidis and Mirella Lapata. 2018. [Multiple instance learning networks for fine-grained sentiment analysis](https://md.research.elamur.ai/free/en/papers/angelidis-2018-multiple-instance-learning-networks-fine/). Transactions of the Association for Computational Linguistics, 6:17–31.
3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. [Translation artifacts in cross-lingual transfer learning](https://md.research.elamur.ai/free/en/papers/artetxe-2020-translation-artifacts-cross-lingual-transfer/). In Proceedings of EMNLP, pages 7674–7684.
4. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of ACL, pages 4623–4637.
5. Seyed Arad Ashrafi Asli, Behnam Sabeti, Zahra Majdabadi, Preni Golazizian, Reza Fahmi, and Omid Momenzadeh. 2020. Optimizing annotation effort using active learning strategies: A sentiment analysis case study in Persian. In Proceedings of LREC, pages 2855–2861. translation using paraphrases. In Proc… [find](https://scholar.google.com/scholar?q=Seyed+Arad+Ashrafi+Asli%2C+Behnam+Sabeti%2C+Zahra+Majdabadi%2C+Preni+Golazizian%2C+Reza+Fahmi%2C+and+Omid+Momenzadeh%2E+2020%2E+Optimizing+annotation+effort+using+active+learning+strategies%3A+A+sentiment+analysis+ca)
6. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020a. TyDi QA: a benchmark for information-seeking question answering in ty pologically di verse languages. Transactions of the Association for Computational Linguistics, 8:454–4… [find](https://scholar.google.com/scholar?q=Jonathan+H+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+2020a%2E+TyDi+QA%3A+a+benchmark+for+information%2Dseeking+question+answering+in+ty+po)
7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
8. Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020b. From ‘F’ to ‘A’ on the NY Regents Science Exams: An Overview of the Aristo Project. AI Magazine, 41(4):39–53. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Daniel+Khashabi%2C+Bhavana+Mishra%2C+Kyle+Richardson%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+Oyvind+Tafjord%2C+Niket+Tandon%2C+et+al%2E+2020b%2E+From+%E2%80%98F%E2%80%99+to+%E2%80%98A%E2%80%99+on+the+NY+Reg)
9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
10. Taha Shangipour Ataei, Kamyar Darvishi, Soroush Javdan, Behrouz Minaei-Bidgoli, and Sauleh Eetemadi. 2019. Pars-absa: an aspect-based sentiment analysis dataset for Persian. arXiv preprint arXiv:1908.01815. — [arXiv:1908.01815](https://arxiv.org/abs/1908.01815) [find](https://scholar.google.com/scholar?q=Taha+Shangipour+Ataei%2C+Kamyar+Darvishi%2C+Soroush+Javdan%2C+Behrouz+Minaei%2DBidgoli%2C+and+Sauleh+Eetemadi%2E+2019%2E+Pars%2Dabsa%3A+an+aspect%2Dbased+sentiment+analysis+dataset+for+Persian%2E+arXiv+preprint+arXiv%3A1908%2E)
11. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475– 2485. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+EMN)
12. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of EMNLP, pages 1533–1544.
13. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Morgan & Claypool Publishers. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Dan+Roth%2C+Mark+Sammons%2C+and+Fabio+Massimo+Zanzotto%2E+2013%2E+Recognizing+Textual+Entailment%3A+Models+and+Applications%2E+Morgan+%26+Claypool+Publishers%2E)
14. Mahmood Bijankhan. 2004. The role of the corpus in writing a grammar: An introduction to a software. Iranian Journal of Linguistics, 19(2):48–67. [find](https://scholar.google.com/scholar?q=Mahmood+Bijankhan%2E+2004%2E+The+role+of+the+corpus+in+writing+a+grammar%3A+An+introduction+to+a+software%2E+Iranian+Journal+of+Linguistics%2C+19%282%29%3A48%E2%80%9367%2E)
15. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of EMNLP.
16. Chris Callison-Burch, Philipp Koehn, and Miles Osborne. 2006. Improved statistical machine Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL, pages 4171–4186.
17. Pablo Duboue and Jennifer Chu-Carroll. 2006. Answering the question you wish they had asked: The impact of paraphrasing for question answering. In Proceedings of NAACL, pages 33–36. [find](https://scholar.google.com/scholar?q=Pablo+Duboue+and+Jennifer+Chu%2DCarroll%2E+2006%2E+Answering+the+question+you+wish+they+had+asked%3A+The+impact+of+paraphrasing+for+question+answering%2E+In+Proceedings+of+NAACL%2C+pages+33%E2%80%9336%2E)
18. Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2020. Parsbert: Transformer-based model for Persian language understanding. arXiv preprint arXiv:2005.12515. [find](https://scholar.google.com/scholar?q=Mehrdad+Farahani%2C+Mohammad+Gharachorloo%2C+Marzieh+Farahani%2C+and+Mohammad+Manthouri%2E+2020%2E+Parsbert%3A+Transformer%2Dbased+model+for+Persian+language+understanding%2E+arXiv+preprint+arXiv%3A2005%2E12515%2E)
19. Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. [GLUECoS: An evaluation benchmark for code-switched NLP](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/). In Proceedings of ACL.
20. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. [find](https://scholar.google.com/scholar?q=Joseph+L+Fleiss%2E+1971%2E+Measuring+nominal+scale+agreement+among+many+raters%2E+Psychological+bulletin%2C+76%285%29%3A378%2E)
21. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. [Unifiedqa: Crossing format boundaries with a single qa system](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). In Proceedings of EMNLP (Findings), pages 1896–1907.
22. Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. [Injecting numerical reasoning skills into language models](https://md.research.elamur.ai/free/en/papers/geva-2020-injecting-numerical-reasoning-skills-into/). In Proceedings of ACL, pages 946–958.
23. Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Mmqa: A multi-domain multi-lingual question-answering framework for english and hindi. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Deepak+Gupta%2C+Surabhi+Kumari%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+Mmqa%3A+A+multi%2Ddomain+multi%2Dlingual+question%2Danswering+framework+for+english+and+hindi%2E+In+Proceedings+of+LREC%2E)
24. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of SIGDIAL, pages 379–391. [find](https://scholar.google.com/scholar?q=Prakhar+Gupta%2C+Shikib+Mehri%2C+Tiancheng+Zhao%2C+Amy+Pavel%2C+Maxine+Eskenazi%2C+and+Jeffrey+P+Bigham%2E+2019%2E+Investigating+evaluation+of+open%2Ddomain+dialogue+systems+with+human+generated+multiple+references%2E+)
25. Pedram Hosseini, Ali Ahmadian Ramaki, Hassan Maleki, Mansoureh Anvari, and Seyed Abolghasem Mirroshandel. 2018. Sentipers: A sentiment analysis corpus for Persian. arXiv preprint arXiv:1801.07737. — [arXiv:1801.07737](https://arxiv.org/abs/1801.07737) [find](https://scholar.google.com/scholar?q=Pedram+Hosseini%2C+Ali+Ahmadian+Ramaki%2C+Hassan+Maleki%2C+Mansoureh+Anvari%2C+and+Seyed+Abolghasem+Mirroshandel%2E+2018%2E+Sentipers%3A+A+sentiment+analysis+corpus+for+Persian%2E+arXiv+preprint+arXiv%3A1801%2E07737%2E+%E2%80%94+%5B)
26. Fatemeh HosseinzadehBendarkheili, Rezvan Mo-hammadiBaghmolaei, and Ali Ahmadi. 2019. Product quality assessment using opinion mining in persian online shopping. In Proceedings of ICEE, pages 1917–1921. IEEE. [find](https://scholar.google.com/scholar?q=Fatemeh+HosseinzadehBendarkheili%2C+Rezvan+Mo%2DhammadiBaghmolaei%2C+and+Ali+Ahmadi%2E+2019%2E+Product+quality+assessment+using+opinion+mining+in+persian+online+shopping%2E+In+Proceedings+of+ICEE%2C+pages+1917%E2%80%931921)
27. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of ICML, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
28. Akbar Karimi, Ebrahim Ansari, and Bahram Sadeghi Bigham. 2018. Extracting an english-persian parallel corpus from comparable corpora. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Akbar+Karimi%2C+Ebrahim+Ansari%2C+and+Bahram+Sadeghi+Bigham%2E+2018%2E+Extracting+an+english%2Dpersian+parallel+corpus+from+comparable+corpora%2E+In+Proceedings+of+LREC%2E)
29. Omid Kashefi. 2018. Mizan: a large Persian-english parallel corpus. arXiv preprint arXiv:1801.02107. — [arXiv:1801.02107](https://arxiv.org/abs/1801.02107) [find](https://scholar.google.com/scholar?q=Omid+Kashefi%2E+2018%2E+Mizan%3A+a+large+Persian%2Denglish+parallel+corpus%2E+arXiv+preprint+arXiv%3A1801%2E02107%2E+%E2%80%94+%5BarXiv%3A1801%2E02107%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1801%2E02107%29)
30. Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021. Gooaq: Open question answering with diverse answer types. arXiv preprint arXiv:2104.08727. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Amos+Ng%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Hannaneh+Hajishirzi%2C+and+Chris+Callison%2DBurch%2E+2021%2E+Gooaq%3A+Open+question+answering+with+diverse+answer+types%2E+arXiv+preprint+arXiv%3A2104%2E08727%2E)
31. Hadi Abdi Khojasteh, Ebrahim Ansari, and Mahdi Bohlouli. 2020. Lscp: Enhanced large scale colloquial persian language understanding. In Proceedings of LREC, pages 6323–6327. [find](https://scholar.google.com/scholar?q=Hadi+Abdi+Khojasteh%2C+Ebrahim+Ansari%2C+and+Mahdi+Bohlouli%2E+2020%2E+Lscp%3A+Enhanced+large+scale+colloquial+persian+language+understanding%2E+In+Proceedings+of+LREC%2C+pages+6323%E2%80%936327%2E)
32. Maria Khvalchik and Mikhail Malkin. 2020. Departamento de nosotros: How machine translated corpora affects language models in mrc tasks. In Proceedings of the Workshop on Hybrid Intelligence for Natural Language Processing Tasks (HI4NLP 2020) co-located with 24th European Conference on Artificial In… [find](https://scholar.google.com/scholar?q=Maria+Khvalchik+and+Mikhail+Malkin%2E+2020%2E+Departamento+de+nosotros%3A+How+machine+translated+corpora+affects+language+models+in+mrc+tasks%2E+In+Proceedings+of+the+Workshop+on+Hybrid+Intelligence+for+Natur)
33. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistic…
34. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. [find](https://scholar.google.com/scholar?q=J+Richard+Landis+and+Gary+G+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+biometrics%2C+pages+159%E2%80%93174%2E)
35. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. [MLQA: Evaluating cross-lingual extractive question answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). In Proceedings of ACL, pages 7315–7330.
36. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of EMNLP, pages 6008– 6018. [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+et+al%2E+2020%2E+Xglue%3A+A+new+benchmark+datasetfor+cross%2Dlingual+pre%2Dtraining%2C+underst)
37. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. [Xcopa: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In Proceedings of EMNLP, pages 2362–2376.
38. Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. Korquad1. 0: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv:1909.07005. — [arXiv:1909.07005](https://arxiv.org/abs/1909.07005) [find](https://scholar.google.com/scholar?q=Seungyoung+Lim%2C+Myungji+Kim%2C+and+Jooyoul+Lee%2E+2019%2E+Korquad1%2E+0%3A+Korean+qa+dataset+for+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1909%2E07005%2E+%E2%80%94+%5BarXiv%3A1909%2E07005%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1909%2E0)
39. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. [SemEval-2014 task 4: Aspect based sentiment analysis](https://md.research.elamur.ai/free/en/papers/pontiki-2014-semeval-2014-task-4-aspect/). In Proceedings of the International Workshop on Semantic Evaluation (SemEval 2014), pages 27–35.
40. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. [find](https://scholar.google.com/scholar?q=Bing+Liu%2E+2012%2E+Sentiment+analysis+and+opinion+mining%2E+Synthesis+lectures+on+human+language+technologies%2C+5%281%29%3A1%E2%80%93167%2E)
41. Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/). arXiv preprint arXiv:2007.15207.
42. Colin P Masica. 1993. The indo-aryan languages. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Colin+P+Masica%2E+1993%2E+The+indo%2Daryan+languages%2E+Cambridge+University+Press%2E)
43. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Todor+Mihaylov%2C+Peter+Clark%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2018%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+EMNLP%2E)
44. Mahsa Mohaghegh, Abdolhossein Sarrafzadeh, and Tom Moir. 2010. Improved language modeling for English-Persian statistical machine translation. In Proceedings of the Workshop on Syntax and Structure in Statistical Translation, pages 75–82. [find](https://scholar.google.com/scholar?q=Mahsa+Mohaghegh%2C+Abdolhossein+Sarrafzadeh%2C+and+Tom+Moir%2E+2010%2E+Improved+language+modeling+for+English%2DPersian+statistical+machine+translation%2E+In+Proceedings+of+the+Workshop+on+Syntax+and+Structure+in)
45. Mahsa Mohaghegh, Abdolhossein Sarrafzadeh, and Tom Moir. 2011. Improving Persian-English statistical machine translation:experiments in domain adaptation. In Proceedings of the Workshop on South Southeast Asian Natural Language Processing (WSSANLP), pages 9–15. [find](https://scholar.google.com/scholar?q=Mahsa+Mohaghegh%2C+Abdolhossein+Sarrafzadeh%2C+and+Tom+Moir%2E+2011%2E+Improving+Persian%2DEnglish+statistical+machine+translation%3Aexperiments+in+domain+adaptation%2E+In+Proceedings+of+the+Workshop+on+South+South)
46. Ellie Pavlick, Matt Post, Ann Irvine, Dmitry Kachaev, and Chris Callison-Burch. 2014. The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics, 2:79–92. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick%2C+Matt+Post%2C+Ann+Irvine%2C+Dmitry+Kachaev%2C+and+Chris+Callison%2DBurch%2E+2014%2E+The+language+demographics+of+amazon+mechanical+turk%2E+Transactions+of+the+Association+for+Computational+Linguistics)
47. Mohammad Taher Pilevar, Heshaam Faili, and Abdol Hamid Pilevar. 2011. Tep: Tehran english-Persian parallel corpus. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 68– 79. Springer. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilevar%2C+Heshaam+Faili%2C+and+Abdol+Hamid+Pilevar%2E+2011%2E+Tep%3A+Tehran+english%2DPersian+parallel+corpus%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+Linguist)
48. Matt Post. 2018. [A call for clarity in reporting BLEU scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/). In Proceedings of WMT, pages 186–191.
49. Prokopis Prokopidis, Vassilis Papavassiliou, and Stelios Piperidis. 2016. Parallel global voices: a collection of multilingual corpora with citizen media stories. In Proceedings of LREC, pages 900–905. [find](https://scholar.google.com/scholar?q=Prokopis+Prokopidis%2C+Vassilis+Papavassiliou%2C+and+Stelios+Piperidis%2E+2016%2E+Parallel+global+voices%3A+a+collection+of+multilingual+corpora+with+citizen+media+stories%2E+In+Proceedings+of+LREC%2C+pages+900%E2%80%93905)
50. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
51. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of EMNLP.
52. Mohammad Sadegh Rasooli, Ahmed El Kholy, and Nizar Habash. 2013. Orthographic and morphological processing for Persian-to-English statistical machine translation. In Proceedings of IJCNLP, pages 1047–1051. [find](https://scholar.google.com/scholar?q=Mohammad+Sadegh+Rasooli%2C+Ahmed+El+Kholy%2C+and+Nizar+Habash%2E+2013%2E+Orthographic+and+morphological+processing+for+Persian%2Dto%2DEnglish+statistical+machine+translation%2E+In+Proceedings+of+IJCNLP%2C+pages+1047%E2%80%93)
53. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. [Mctest: A challenge dataset for the open-domain machine comprehension of text](https://md.research.elamur.ai/free/en/papers/richardson-2013-mctest-challenge-dataset-open-domain/). In Proceedings of EMNLP, pages 193–203.
54. Mojgan Seraji, Carina Jahani, Beáta Megyesi, and Joakim Nivre. 2013. Uppsala Persian dependency treebank annotation guidelines. Technical report, Uppsala University. [find](https://scholar.google.com/scholar?q=Mojgan+Seraji%2C+Carina+Jahani%2C+Be%C3%A1ta+Megyesi%2C+and+Joakim+Nivre%2E+2013%2E+Uppsala+Persian+dependency+treebank+annotation+guidelines%2E+Technical+report%2C+Uppsala+University%2E)
55. Mahsa Sadat Shahshahani, Mahdi Mohseni, Azadeh Shakery, and Heshaam Faili. 2019. [find](https://scholar.google.com/scholar?q=Mahsa+Sadat+Shahshahani%2C+Mahdi+Mohseni%2C+Azadeh+Shakery%2C+and+Heshaam+Faili%2E+2019%2E)
56. Payma: A tagged corpus of Persian named entities. Signal and Data Processing, 16(1):91– 110. [find](https://scholar.google.com/scholar?q=Payma%3A+A+tagged+corpus+of+Persian+named+entities%2E+Signal+and+Data+Processing%2C+16%281%29%3A91%E2%80%93+110%2E)
57. Javad PourMostafa Roshan Sharami, Parsa Abbasi Sarabestani, and Seyed Abolghasem Mirroshandel. 2020. Deepsentipers: Novel deep learning models trained over proposed augmented persian sentiment corpus. arXiv preprint arXiv:2004.05328. — [arXiv:2004.05328](https://arxiv.org/abs/2004.05328) [find](https://scholar.google.com/scholar?q=Javad+PourMostafa+Roshan+Sharami%2C+Parsa+Abbasi+Sarabestani%2C+and+Seyed+Abolghasem+Mirroshandel%2E+2020%2E+Deepsentipers%3A+Novel+deep+learning+models+trained+over+proposed+augmented+persian+sentiment+corpus%2E)
58. Tatiana Shavrina, Alena Fenogenova, Emelyanov Anton, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. In Proceedings of EMNLP, pages 4717–472… [find](https://scholar.google.com/scholar?q=Tatiana+Shavrina%2C+Alena+Fenogenova%2C+Emelyanov+Anton%2C+Denis+Shevelev%2C+Ekaterina+Artemova%2C+Valentin+Malykh%2C+Vladislav+Mikhailov%2C+Maria+Tikhonova%2C+Andrey+Chertok%2C+and+Andrey+Evlampiev%2E+2020%2E+Russiansuper)
59. Gary F Simons and Charles D Fennig. 2017. Ethnologue: languages of Asia. sil International Dallas. [find](https://scholar.google.com/scholar?q=Gary+F+Simons+and+Charles+D+Fennig%2E+2017%2E+Ethnologue%3A+languages+of+Asia%2E+sil+International+Dallas%2E)
60. Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. [Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence](https://md.research.elamur.ai/free/en/papers/sun-2019-utilizing-bert-aspect-based-sentiment/). In Proceedings of NAACL, pages 380–385.
61. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of NAACL, pages 4149–4158.
62. Tun Thura Thet, Jin-Cheon Na, and Christopher SG Khoo. 2010. Aspect-based sentiment analysis of movie reviews on discussion boards. Journal of information science, 36(6):823–848. [find](https://scholar.google.com/scholar?q=Tun+Thura+Thet%2C+Jin%2DCheon+Na%2C+and+Christopher+SG+Khoo%2E+2010%2E+Aspect%2Dbased+sentiment+analysis+of+movie+reviews+on+discussion+boards%2E+Journal+of+information+science%2C+36%286%29%3A823%E2%80%93848%2E)
63. Jörg Tiedemann and Lars Nygaard. 2004. The OPUS corpus - parallel and free: [http://logos.uio.no/opus](http://logos.uio.no/opus). In Proceedings of LREC. [↗](http://logos.uio.no/opus)
64. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Proceedings of NourIPS, pages 3266–3280.
65. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of NAACL, pages 1112–1122.
66. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. [Clue: A chinese language understanding evaluation benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/). In Proceedings of COLING, pages 4762–4772.
67. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
68. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [Paws-x: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of EMNLP-IJCNLP, pages 3678–3683.
69. Poorya Zaremoodi, Wray Buntine, and Gholamreza Haffari. 2018. Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 656–661. [find](https://scholar.google.com/scholar?q=Poorya+Zaremoodi%2C+Wray+Buntine%2C+and+Gholamreza+Haffari%2E+2018%2E+Adaptive+knowledge+sharing+in+multi%2Dtask+learning%3A+Improving+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+56th+Annual+Me)
70. Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. [Improving massively multilingual neural machine translation and zero-shot translation](https://md.research.elamur.ai/free/en/papers/zhang-2020-improving-massively-multilingual-neural-machine/). In Proceedings of ACL, pages 1628–1639.
71. Ingrid Zukerman and Bhavani Raskutti. 2002. Lexical query paraphrasing for document retrieval. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Ingrid+Zukerman+and+Bhavani+Raskutti%2E+2002%2E+Lexical+query+paraphrasing+for+document+retrieval%2E+In+Proceedings+of+COLING%2E)

*Updated 31 October 2026*