--- title: "ArPanEmo: An Open-Source Dataset for — Althobaiti 2023" description: "Arabic emotion recognition has few annotated resources." authors: ["Maha Jarallah Althobaiti"] year: 2023 venue: "arXiv:2305.17580" source: "https://arxiv.org/abs/2305.17580" doi: "10.48550/arxiv.2305.17580" arxiv: "2305.17580" licence: "CC BY-SA 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-sa/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/althobaiti-2023-arpanemo-open-source-dataset-fine/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/althobaiti-2023-arpanemo-open-source-dataset-fine/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / ArPanEmo: An Open-Source Dataset for Fine-Grained Emotion Recognition in Arabic Online Content during COVID-19 Pandemic # ArPanEmo: An Open-Source Dataset for Fine-Grained Emotion Recognition in Arabic Online Content during COVID-19 Pandemic **Authors:** Maha Jarallah Althobaiti · **Year:** 2023 · **Venue:** arXiv:2305.17580 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2305.17580) · [PDF](https://arxiv.org/pdf/2305.17580) · [DOI 10.48550/arxiv.2305.17580](https://doi.org/10.48550/arxiv.2305.17580) · [arXiv 2305.17580](https://arxiv.org/abs/2305.17580) · [OpenAlex](https://openalex.org/W4391271503) **Citations:** 1 citations (openalex-snapshot · 2026-10-14) ## Summary Arabic emotion recognition has few annotated resources. Most of those that exist mix dialects with MSA and cover general topics. ArPanEmo addresses this with a single-dialect, single-topic corpus of Saudi Arabic online posts about COVID-19. The posts were gathered over two years after the WHO pandemic declaration from three sources: Twitter (geolocated to Saudi Arabia), YouTube comments, and reader comments on the newspaper Al-marsd. Keyword lists selected pandemic-related content and assigned each post to a domain, either healthcare or impacted life aspects (ILA). Annotation ran in two steps. A lexicon of emotion terms first sorted posts into emotional and neutral candidates, with manual checking. Annotators then gave the emotional posts fine-grained labels. Each post received three annotations, and agreement was substantial. The authors validated the resource by training an SVM, a BiGRU and a fine-tuned AraBERT. AraBERT scored best. The dataset supports dialect-specific and topic-specific emotion research, and the collection pipeline can be reused for other varieties or languages. ## Key points - The corpus has 11,128 posts and 223,008 tokens, labeled with ten emotions plus neutral. The emotions are anger, anticipation, confusion, disgust, fear, joy, optimism, pessimism, sadness and surprise. Posts average about 19 tokens (roughly 112 characters). - By domain, 8,751 posts are healthcare and 2,377 are ILA. The fixed 80/20 split (8,901 train, 2,227 test) is stratified by both domain and emotion class. - Fleiss' kappa over three annotators is 0.7123, with a 95% CI of 0.625–0.8, computed with AgreeStat360. Percent agreement is about 0.75. - Baseline results: - AraBERTv0.2-Twitter-large reaches 0.67 accuracy and 0.67 weighted F1. - SVM on word 1–3-gram TF-IDF plus char 2–5-grams reaches 0.58 accuracy. - BiGRU reaches 0.40 accuracy. - The data is released as CSV on Mendeley Data (DOI 10.17632/d9yy8w52ns.2). Twitter content is shared only as tweet IDs with labels, to comply with platform terms. ## Interesting ideas - The annotation is split into a cheap yes/no pass (does the post carry emotion at all?) followed by a fine-grained pass, which lowers annotator load and keeps the fine label space clean. — For Mova (frames): Irony or speech-act labelling could first ask a binary question ("is this non-literal?") and only then assign the specific subtype, so ordinary cases never reach the harder classifier. - A lexicon pre-sorts items into likely-positive and likely-negative pools, and humans then verify both pools rather than trusting the lexicon. — For Mova (learn): When building silver data, a cheap rule-based filter could route sentences to the right LLM teacher tier, while sampled human or teacher checks on the rejected pool catch what the filter misses. - The test split is stratified jointly on two axes (domain and label) and published as a fixed file, so every later model is compared on the same distribution. — For Mova (learn): Evaluation sets for fables, school tests or word problems could be frozen and stratified by both text genre and target phenomenon, making regressions comparable across runs. - Similar macro-averaged and weighted-averaged scores are read as evidence that no class dominates the results. — For Mova (learn): Reporting both averages for tagger, parser-relation or frame-role metrics would quickly reveal whether gains come only from frequent labels. - Reliability is reported as a chance-corrected multi-rater coefficient with a confidence interval, not just raw agreement. — For Mova (learn): Agreement among several witness parsers could be measured with Fleiss' kappa per relation type, giving a principled threshold for accepting silver trees. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{althobaiti2023arpanemo, title = {{ArPanEmo: An Open-Source Dataset for Fine-Grained Emotion Recognition in Arabic Online Content during COVID-19 Pandemic}}, author = {Althobaiti, Maha Jarallah}, year = {2023}, doi = {10.48550/arxiv.2305.17580}, eprint = {2305.17580}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.17580} } ``` - [← previous ARCOQ: Arabic Closest Opposite Questions Dataset](https://md.research.elamur.ai/free/en/papers/rizkallah-2023-arcoq-arabic-closest-opposite-questions/) - [next → Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models](https://md.research.elamur.ai/free/en/papers/zacharopoulos-2023-assessing-influence-attractor-verb-distance/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) ## References 16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. J. Roesslein, "Tweepy: An easy-to-use Python library for accessing the Twitter API," [Online]. Available: [https://www.tweepy.org/](https://www.tweepy.org/). [Accessed 6 March 2023]. [↗](https://www.tweepy.org/) 2. S. Richter, "lxml - XML and HTML with Python," [Online]. Available: [https://lxml.de/](https://lxml.de/). [Accessed 6 March 2023]. [↗](https://lxml.de/) 3. Urllib, "URL handling modules," [Online]. Available: [https://docs.python.org/3/library/urllib.html#module-urllib](https://docs.python.org/3/library/urllib.html#module-urllib). [Accessed 6 March 2023]. [↗](https://docs.python.org/3/library/urllib.html#module-urllib) 4. L. Richardson, "Beautiful Soup Documentation," [Online]. Available: [https://beautiful-soup-4.readthedocs.io/en/latest/](https://beautiful-soup-4.readthedocs.io/en/latest/). [Accessed 6 March 2023]. [↗](https://beautiful-soup-4.readthedocs.io/en/latest/) 5. S. M. Mohammad, "Word Affect Intensities," in Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan, 2018. [find](https://scholar.google.com/scholar?q=S%2E+M%2E+Mohammad%2C+%22Word+Affect+Intensities%2C%22+in+Proceedings+of+the+11th+Edition+of+the+Language+Resources+and+Evaluation+Conference+%28LREC%2D2018%29%2C+Miyazaki%2C+Japan%2C+2018%2E) 6. S. K. Saif Mohammad, "Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories," in Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. [find](https://scholar.google.com/scholar?q=S%2E+K%2E+Saif+Mohammad%2C+%22Understanding+Emotions%3A+A+Dataset+of+Tweets+to+Study+Interactions+between+Affect+Categories%2C%22+in+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Ev) 7. S. Mohammad, F. Bravo-Marquez, M. Salameh and S. Kiritchenko, "SemEval-2018 Task 1: Affect in Tweets," New Orleans, Louisiana, 2018. [find](https://scholar.google.com/scholar?q=S%2E+Mohammad%2C+F%2E+Bravo%2DMarquez%2C+M%2E+Salameh+and+S%2E+Kiritchenko%2C+%22SemEval%2D2018+Task+1%3A+Affect+in+Tweets%2C%22+New+Orleans%2C+Louisiana%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/) 8. R. Plutchik, "A general psychoevolutionary theory of emotion," in Theories of emotion, Elsevier, 1980, pp. 3-33. [find](https://scholar.google.com/scholar?q=R%2E+Plutchik%2C+%22A+general+psychoevolutionary+theory+of+emotion%2C%22+in+Theories+of+emotion%2C+Elsevier%2C+1980%2C+pp%2E+3%2D33%2E) 9. J. L. Fleiss, "Measuring nominal scale agreement among many raters," Psychological bulletin, vol. 76, no. 5, pp. 378-382, 1971. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Fleiss%2C+%22Measuring+nominal+scale+agreement+among+many+raters%2C%22+Psychological+bulletin%2C+vol%2E+76%2C+no%2E+5%2C+pp%2E+378%2D382%2C+1971%2E) 10. J. L. Fleiss, B. Levin and M. C. Paik, Statistical methods for rates and proportions, john wiley & sons, 2013. [find](https://scholar.google.com/scholar?q=J%2E+L%2E+Fleiss%2C+B%2E+Levin+and+M%2E+C%2E+Paik%2C+Statistical+methods+for+rates+and+proportions%2C+john+wiley+%26+sons%2C+2013%2E) 11. K. Gwet, AgreeStat360 [Computer software]. AgreeStat Analytics. [https://www.agreestat.com/software/default.html](https://www.agreestat.com/software/default.html), 2020. [↗](https://www.agreestat.com/software/default.html) 12. J. R. Landis and G. G. Koch, "The measurement of observer agreement for categorical data," biometrics, vol. 33, no. 1, pp. 159 - 174, 1977. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Landis+and+G%2E+G%2E+Koch%2C+%22The+measurement+of+observer+agreement+for+categorical+data%2C%22+biometrics%2C+vol%2E+33%2C+no%2E+1%2C+pp%2E+159+%2D+174%2C+1977%2E) 13. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg and others, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, no. 1, pp. 2825-2830, 2011. [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg+and+others%2C+%22Scikit%2Dlearn%3A+Machine+learning+in+Python%2C%22+Journal+of+Machine+) 14. J. Devlin, M. W. Chang, K. Lee and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies., Minneapolis, Minnesota, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Devlin%2C+M%2E+W%2E+Chang%2C+K%2E+Lee+and+K%2E+Toutanova%2C+%22BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2C%22+in+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter) 15. W. Antoun, F. Baly and H. Hajj, "AraBERT: Transformer-based Model for Arabic Language Understanding," in Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, Marseille, France, 2020. [find](https://scholar.google.com/scholar?q=W%2E+Antoun%2C+F%2E+Baly+and+H%2E+Hajj%2C+%22AraBERT%3A+Transformer%2Dbased+Model+for+Arabic+Language+Understanding%2C%22+in+Proceedings+of+the+4th+Workshop+on+Open%2DSource+Arabic+Corpora+and+Processing+Tools%2C+with+a+Shar) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/) 16. Twitter, "Developer Agreement and Policy," [Online]. Available: [https://developer.twitter.com/en/developer-terms/agreement-and-policy](https://developer.twitter.com/en/developer-terms/agreement-and-policy). [Accessed 6 March 2023]. [↗](https://developer.twitter.com/en/developer-terms/agreement-and-policy)